Build a Full Site Inventory
Crawl your entire website to build a complete inventory of every page, its status, and its metadata.
Site Crawler
Map pages, links, and errors
Walk a website from any URL, collecting per-page titles, descriptions, heading counts, word counts, response times, and link counts. Ideal for site audits, content inventories, broken-link detection, and SEO analysis.
Crawl Site
Crawl pages from a starting URL and collect metadata including title, description, heading counts, link counts, response time, and crawl diagnostics.
Returns: Bounded crawl results with per-page SEO metadata, link graph hints, response timing, and failure diagnostics
Parameters
url *stringStarting URL or hostname to crawl
max_pagesnumberMaximum number of pages to crawl (1-100)
timeout_msnumberRequest timeout per page in milliseconds (3000-60000)
same_origin_onlybooleanOnly crawl pages on the same origin as the start URL
include_subdomainsbooleanTreat subdomains as internal links when same-origin mode is enabled
include_externalbooleanQueue external links for crawling when same_origin_only is false
Example
Crawl a small sample of example.com pages
curl -H "Authorization: Bearer $TOOLROUTER_API_KEY" \
-d '{
"tool": "site-crawler",
"skill": "crawl_site",
"input": {
"url": "https://example.com",
"max_pages": 5
}
}' \
https://api.toolrouter.com/v1/tools/callQuick Start
MCP (Claude Code)
claude mcp add --transport stdio \
--env TOOLROUTER_API_KEY=YOUR_API_KEY \
toolrouter -- npx -y toolrouter-mcpREST API
curl -H "Authorization: Bearer $TOOLROUTER_API_KEY" \
-d '{"tool":"site-crawler","skill":"crawl_site","input":{}}' \
https://api.toolrouter.com/v1/tools/callUse Cases
Find 404 Errors Across Your Site
Crawl your website to discover all pages returning 404 errors and the internal links pointing to them.
Search Papers by Topic
Find relevant academic papers on any research topic across millions of scholarly publications.
Analyze App Review Sentiment
Understand how users feel about your app by analyzing sentiment patterns across hundreds or thousands of reviews.
Workflows
Comprehensive Website Audit
Run a full-spectrum website audit combining SEO analysis, performance testing, site crawling, and visual documentation in one workflow.
Website Migration Audit
Prepare for website migration by crawling the current site, documenting SEO baselines, benchmarking performance, and recording DNS configuration.
Web Application Security Test
Crawl a web application, analyze HTTP security, test for injections, and document findings with screenshots.
Frequently Asked Questions
Can it crawl a whole site or just one page?
It walks a website from a starting URL and gathers data page by page, so it works for both single-page and full-site audits.
What page data does it collect?
It collects titles, descriptions, heading counts, word counts, response times, and link counts.
Is it useful for broken-link or content inventory work?
Yes. Those are two of the most practical uses for the crawler.
Do I need to start from the homepage?
No. You can start from any URL and let the crawl expand from there.