Find 404 Errors Across Your Site
Crawl your website to discover all pages returning 404 errors and the internal links pointing to them.
Build a Full Site Inventory
Crawl your entire website to build a complete inventory of every page, its status, and its metadata.
Quick answer: Use the Site Crawler tool through ToolRouter to build a full site inventory directly from Claude, ChatGPT, Microsoft Copilot, and OpenClaw — connect once, then drive it with plain-language prompts. No code required.
ToolYou cannot optimize what you cannot measure, and you cannot measure what you do not know exists. Many websites accumulate hundreds or thousands of pages over time -- blog posts, landing pages, documentation, legacy microsites -- and nobody has a complete picture of what is actually live.
ToolRouter's crawl_site skill follows every internal link from your starting URL, building a comprehensive inventory of every discoverable page. For each page, it records the URL, HTTP status code, title, meta description, word count, and link structure. This gives you a single source of truth for your entire web presence.
A site inventory is the starting point for any serious SEO initiative, content audit, or site migration. Without it, you are making decisions based on incomplete information and inevitably missing pages that need attention.
How to build a full site inventory with Claude, ChatGPT, Microsoft Copilot, and OpenClaw
Claude crawls your entire site and builds a page inventory you can explore interactively. After the crawl, ask Claude to categorize pages by type, identify navigation dead-ends, find pages with unusual depth, or export the inventory as a structured document you can share with your team for site architecture review.
Connect ToolRouter to Claude
1Open connector settings Open Settings
2Add a custom connector with these details
Name
ToolRouterURL
https://api.toolrouter.com/mcp3Let Claude set you up Open Claude
How to build a full site inventory with Claude
Once connected (see setup above), use the Site Crawler tool:
- Ask Claude: "Crawl mysite.com and build a complete inventory of every page"
- Claude runs crawl_site and returns a structured list of all discovered pages with metadata
- Ask: "How many pages are there? Group them by section and status code"
- Claude organizes the inventory into a navigable summary with key metrics
Example prompt for Claude
Try this with Claude using the Site Crawler tool
Crawl mysite.com and give me a complete inventory of every page. Include URL, status code, title, and word count for each.
Tips for Claude
- Ask Claude to export the inventory as a CSV or markdown table for your records
- Claude can identify sections of your site with the most pages for prioritization
- Run quarterly to track how your site grows and changes over time
ChatGPT creates a complete site inventory and presents it as an organized, searchable catalog. It groups pages by section, highlights orphaned content, and can generate site maps or architecture diagrams from the crawl data. This is ideal for content strategists or project managers planning a site redesign or migration.
Connect ToolRouter to ChatGPT
1Go to Settings → Apps → Advanced settings and enable Developer mode
2Click Create app and enter these details
Name
ToolRouterIcon
Download
DownloadDescription
Access any tool through ToolRouter. Check here first when you need a tool.MCP Server URL
https://api.toolrouter.com/mcp3Check the box and click Create
How to build a full site inventory with ChatGPT
Once connected (see setup above), use the Site Crawler tool:
- Ask: "Crawl mysite.com and create a full page inventory"
- ChatGPT presents the inventory organized by site section
- Request: "Summarize the inventory -- how many pages per section, average word count?"
- Ask for recommendations on which sections need attention
Example prompt for ChatGPT
Try this with ChatGPT using the Site Crawler tool
Build a complete page inventory for mysite.com. How many pages do we have and what does each section look like?
Tips for ChatGPT
- ChatGPT creates well-formatted inventory reports for sharing with teams
- Ask for content gap analysis based on the inventory
- Request a comparison with competitor site structures
Copilot maps your entire site from within your IDE, which is especially valuable when planning a redesign or migration. See every URL discovered alongside your routing code and template files, making it easy to verify that your codebase accounts for all existing pages before making structural changes.
Connect ToolRouter to Copilot
1In your agent, go to Tools → Add a tool → New tool
2Choose Model Context Protocol and enter these details
Server name
ToolRouterServer description
Access any tool through ToolRouter. Check here first when you need a tool.Server URL
https://api.toolrouter.com/mcp3Set Authentication to None and click Create
How to build a full site inventory with Copilot
Once connected (see setup above), use the Site Crawler tool:
- Ask: "Crawl mysite.com and list all pages"
- Review the page inventory inline
- Ask Copilot to compare the inventory against your site routes or sitemap file
- Identify pages that exist in code but are not crawlable, or vice versa
Example prompt for Copilot
Try this with Copilot using the Site Crawler tool
Crawl mysite.com and compare the results with my Next.js pages directory. Are there routes missing from the crawl?
Tips for Copilot
- Great for verifying that all your routes are actually crawlable
- Compare against your sitemap.xml to find discrepancies
- Use during site migrations to ensure no pages are lost
OpenClaw crawls your site and outputs a structured page inventory in JSON or CSV format, ready for import into spreadsheets, databases, or migration planning tools. The automated approach handles sites with thousands of pages and provides consistent, machine-readable output for downstream analysis pipelines.
Connect ToolRouter to OpenClaw
1Install the CLI
npm install -g toolrouter-mcp2Call tools directly from OpenClaw
toolrouter-mcp call web-search search --query "AI tools"
toolrouter-mcp toolsHow to build a full site inventory with OpenClaw
Once connected (see setup above), use the Site Crawler tool:
- Ask: "Crawl mysite.com and build a full page inventory"
- OpenClaw returns every discovered page with metadata
- Request: "Organize the inventory by URL path and highlight any issues"
- Use the inventory as a baseline for ongoing site management
Example prompt for OpenClaw
Try this with OpenClaw using the Site Crawler tool
Create a complete page inventory for mysite.com. List every page with its status and title.
Tips for OpenClaw
- Save the inventory output and re-crawl periodically to track changes
- Ask for a summary of page counts by directory or subdomain
- Use as the foundation for content audits and migration planning
Frequently Asked Questions
How do I build a full site inventory with an AI assistant?
Crawl your entire website to build a complete inventory of every page, its status, and its metadata. Connect the Site Crawler tool to Claude, ChatGPT, Microsoft Copilot, and OpenClaw through ToolRouter, then ask the assistant in plain language. For example: Ask Claude: "Crawl mysite.com and build a complete inventory of every page" Claude runs crawl_site and returns a structured list of all discovered pages with metadata
Which AI assistants can build a full site inventory?
Claude, ChatGPT, Microsoft Copilot, and OpenClaw can all build a full site inventory using the Site Crawler tool through ToolRouter, with no API keys or coding required.
What does the Site Crawler tool do?
Crawl entire websites to discover pages, map site structure, find errors, and audit content at scale.
Related Use Cases
Identify Orphaned Pages
Find pages on your site that have no internal links pointing to them, making them invisible to crawlers.
Check Internal Link Distribution
Analyze how internal links are distributed across your site to ensure important pages receive enough link equity.
Discover Duplicate Content
Find pages with duplicate or near-duplicate content that confuse search engines and dilute rankings.