How to Crawl Multi-Page Sites with ChatGPT
Crawl Multi-Page Sites with ChatGPT and ToolRouter. Collect structured data across an entire site and format the results for analysis or reporting.
Tool
Use ChatGPT with Stealth Scraper to crawl a multi-page site and organize the collected data into a useful deliverable — a structured dataset, a comparison table, or a content inventory. ChatGPT is well-suited when the crawled data needs to be immediately formatted for stakeholder use.
Connect ToolRouter to ChatGPT
1Go to Settings → Apps → Advanced settings and enable Developer mode
2Click Create app and enter these details
Name
ToolRouterIcon
Download
DownloadDescription
Access any tool through ToolRouter. Check here first when you need a tool.MCP Server URL
https://api.toolrouter.com/mcp3Check the box and click Create
Steps
Once connected (see setup above), use the Stealth Scraper tool:
- Specify the starting URL, crawl depth, and what the output will be used for.
- Ask ChatGPT to use `stealth-scraper` with `stealth_crawl` to collect pages.
- Have ChatGPT extract the relevant fields from each crawled page.
- Ask ChatGPT to organize the results into a table, inventory list, or structured summary document.
Example Prompt
Try this with ChatGPT using the Stealth Scraper tool
Use stealth-scraper to crawl https://example.com/blog starting from the index, following all article links. Extract title, publication date, author, and first paragraph from each article. Return a table sorted by publication date, newest first.
Tips
- Specify the sort order for the output table — newest first for content audits, alphabetical for directories.
- Ask for a content inventory format if the crawl will feed into a CMS migration or SEO audit.
- Request a count of pages crawled alongside the data so you know if the crawl completed fully.