How to Crawl Multi-Page Sites with OpenClaw
Crawl Multi-Page Sites with OpenClaw and ToolRouter. Automate recurring multi-page crawls and maintain current datasets from dynamic sites.
Tool
OpenClaw automates recurring `stealth_crawl` jobs across multi-page sites — keeping your dataset current by re-crawling on a schedule and surfacing what changed since the last run. This is the right approach when the site updates regularly and you need to track additions, removals, or price changes over time.
Connect ToolRouter to OpenClaw
1Install the CLI
npm install -g toolrouter-mcp2Call tools directly from OpenClaw
toolrouter-mcp call web-search search --query "AI tools"
toolrouter-mcp toolsSteps
Once connected (see setup above), use the Stealth Scraper tool:
- Define the starting URL, crawl depth, and the fields to extract from each page.
- Run `stealth-scraper` with `stealth_crawl` and collect the full dataset in a normalized schema.
- Diff the new dataset against the previous crawl to identify added, removed, or changed pages.
- Schedule the crawl on the cadence that matches the site's update frequency.
Example Prompt
Try this with OpenClaw using the Stealth Scraper tool
Use stealth-scraper to crawl https://example.com/products up to depth 2 and extract name, price, and availability from each product page. Return all results in a stable JSON array. I'll diff this against last week's crawl to find price changes and new listings.
Tips
- Use the page URL as a stable identifier so dataset diffs are clean between crawl runs.
- Schedule the crawl frequency to match the site's typical update cadence — daily for news, weekly for product catalogues.
- Keep the schema fixed between runs so diffs work without field normalization.