How to Crawl Multi-Page Sites with Claude
Crawl Multi-Page Sites with Claude and ToolRouter. Follow links across a full site and extract structured data from every matching page in a single run.
Tool
Use Claude with Stealth Scraper to crawl a multi-page site and analyze the collected dataset. Claude can guide the crawl scope, identify patterns in the extracted data across pages, and surface insights that require looking at the full site rather than individual pages.
Connect ToolRouter to Claude
1Open connector settings Open Settings
2Add a custom connector with these details
Name
ToolRouterURL
https://api.toolrouter.com/mcp3Let Claude set you up Open Claude
Steps
Once connected (see setup above), use the Stealth Scraper tool:
- Provide the starting URL and describe the scope — which pages you want, how deep to crawl, and what data to extract from each.
- Ask Claude to use `stealth-scraper` with `stealth_crawl` starting from your URL.
- Ask Claude to summarize patterns across all crawled pages once the dataset is collected.
- Follow up with specific questions about the dataset — pricing trends, content gaps, or structural anomalies.
Example Prompt
Try this with Claude using the Stealth Scraper tool
Use stealth-scraper to crawl the documentation site at https://docs.example.com starting from the index page. Follow all internal links up to 2 levels deep. Extract the title, section, and main body text from each page. Once collected, tell me which sections have the most pages and whether any pages appear to have incomplete content.
Tips
- Define the crawl scope before starting — specify whether you want all pages or only those matching a pattern like /docs/ or /products/.
- Ask Claude to look for patterns across the full dataset rather than summarizing individual pages.
- For large sites, start with a shallow crawl (depth 1) to verify the extraction logic before going deeper.