Get started
Tools / Site Crawler / Use Cases / Build a Full Site Inventory / OpenClaw

How to Build a Full Site Inventory with OpenClaw

Build a site page inventory with OpenClaw and ToolRouter. Complete website discovery and mapping.

Tool
Site Crawler icon
Site Crawler

OpenClaw crawls your site and outputs a structured page inventory in JSON or CSV format, ready for import into spreadsheets, databases, or migration planning tools. The automated approach handles sites with thousands of pages and provides consistent, machine-readable output for downstream analysis pipelines.

Connect ToolRouter to OpenClaw

1Install the CLI
npm install -g toolrouter-mcp
2Call tools directly from OpenClaw
toolrouter-mcp call web-search search --query "AI tools"
toolrouter-mcp tools
No API key needed — account created automatically.All platforms →

Steps

Once connected (see setup above), use the Site Crawler tool:

  1. Ask: "Crawl mysite.com and build a full page inventory"
  2. OpenClaw returns every discovered page with metadata
  3. Request: "Organize the inventory by URL path and highlight any issues"
  4. Use the inventory as a baseline for ongoing site management

Example Prompt

Try this with OpenClaw using the Site Crawler tool
Create a complete page inventory for mysite.com. List every page with its status and title.

Tips

  • Save the inventory output and re-crawl periodically to track changes
  • Ask for a summary of page counts by directory or subdomain
  • Use as the foundation for content audits and migration planning

More OpenClaw Guides

How to Find 404 Errors with OpenClawHow to Identify Orphaned Pages with OpenClawHow to Check Internal Link Distribution with OpenClawHow to Discover Duplicate Content with OpenClawHow to Audit URL Structure with OpenClaw

Related Workflows

Comprehensive Website AuditRun a full-spectrum website audit combining SEO analysis, performance testing, site crawling, and visual documentation in one workflow.Website Migration AuditPrepare for website migration by crawling the current site, documenting SEO baselines, benchmarking performance, and recording DNS configuration.Web Application Security TestCrawl a web application, analyze HTTP security, test for injections, and document findings with screenshots.