Site Crawler
Map pages, links, and errors
On it. Running Site Crawler now and I'll bring the results straight back here.
Try asking
About
Site Crawler walks any website and collects per-page metadata — titles, descriptions, heading structure, word counts, response times, and link counts. Give it a URL and get back a structured inventory of everything it finds.
It's the fastest way to audit a site's content landscape without manually clicking through pages. Use it to find thin content, spot broken links, map internal structure, or build a content inventory before an SEO project.
What you can do
- Crawl up to 100 pages from any starting URL
- Collect title, description, heading counts, word count, link count, and response time per page
- Detect 404s, timeouts, and other errors
- Control crawl scope — same-origin only, or extend to subdomains and external links
- Adjust timeout per page for slow-loading sites
Who it's for
SEO professionals running content audits. Developers mapping site structure before a migration. Content teams identifying thin or duplicate pages. Anyone who needs a fast overview of what's actually on a site.
How to use it
- Use crawl_site with your starting URL — set max_pages to 10 for a quick overview or up to 100 for full coverage
- Check the errors array for 404s and timeouts
- Sort by word_count to find thin content pages
- Set include_subdomains to true for multi-subdomain sites like docs.example.com
Getting started
Save your primary domain as a credential so it's used as the default — or just pass any URL directly to crawl_site.
Crawl pages from a starting URL and collect metadata including title, description, heading counts, link counts, response time, and crawl diagnostics.
- Added subtitle, expanded description, and agent instructions
- Initial release
Related
Questions
Can it crawl a whole site or just one page?
It walks a website from a starting URL and gathers data page by page, so it works for both single-page and full-site audits.
What page data does it collect?
It collects titles, descriptions, heading counts, word counts, response times, and link counts.
Is it useful for broken-link or content inventory work?
Yes. Those are two of the most practical uses for the crawler.
Do I need to start from the homepage?
No. You can start from any URL and let the crawl expand from there.