Site Crawler walks any website and collects per-page metadata — titles, descriptions, heading structure, word counts, response times, and link counts. Give it a URL and get back a structured inventory of everything it finds.
It's the fastest way to audit a site's content landscape without manually clicking through pages. Use it to find thin content, spot broken links, map internal structure, or build a content inventory before an SEO project.
What you can do
- Crawl up to 100 pages from any starting URL
- Collect title, description, heading counts, word count, link count, and response time per page
- Detect 404s, timeouts, and other errors
- Control crawl scope — same-origin only, or extend to subdomains and external links
- Adjust timeout per page for slow-loading sites
Who it's for
SEO professionals running content audits. Developers mapping site structure before a migration. Content teams identifying thin or duplicate pages. Anyone who needs a fast overview of what's actually on a site.
How to use it
- Use crawl_site with your starting URL — set max_pages to 10 for a quick overview or up to 100 for full coverage
- Check the errors array for 404s and timeouts
- Sort by word_count to find thin content pages
- Set include_subdomains to true for multi-subdomain sites like docs.example.com
Getting started
Save your primary domain as a credential so it's used as the default — or just pass any URL directly to crawl_site.
