Find 404 Errors Across Your Site
Crawl your website to discover all pages returning 404 errors and the internal links pointing to them.
Build a Full Site Inventory
Crawl your entire website to build a complete inventory of every page, its status, and its metadata.
ToolYou cannot optimize what you cannot measure, and you cannot measure what you do not know exists. Many websites accumulate hundreds or thousands of pages over time -- blog posts, landing pages, documentation, legacy microsites -- and nobody has a complete picture of what is actually live.
ToolRouter's crawl_site skill follows every internal link from your starting URL, building a comprehensive inventory of every discoverable page. For each page, it records the URL, HTTP status code, title, meta description, word count, and link structure. This gives you a single source of truth for your entire web presence.
A site inventory is the starting point for any serious SEO initiative, content audit, or site migration. Without it, you are making decisions based on incomplete information and inevitably missing pages that need attention.
Agent Guides
Claude
- Connect ToolRouter: claude mcp add toolrouter -- npx -y toolrouter-mcp
- Ask Claude: "Crawl mysite.com and build a complete inventory of every page"
- Claude runs crawl_site and returns a structured list of all discovered pages with metadata
ChatGPT
- Add ToolRouter to ChatGPT via MCP configuration
- Ask: "Crawl mysite.com and create a full page inventory"
- ChatGPT presents the inventory organized by site section
Copilot
- Add ToolRouter to Copilot MCP config
- Ask: "Crawl mysite.com and list all pages"
- Review the page inventory inline
OpenClaw
- Connect ToolRouter: openclaw mcp add toolrouter -- npx -y toolrouter-mcp
- Ask: "Crawl mysite.com and build a full page inventory"
- OpenClaw returns every discovered page with metadata
Related Use Cases
Identify Orphaned Pages
Find pages on your site that have no internal links pointing to them, making them invisible to crawlers.
Check Internal Link Distribution
Analyze how internal links are distributed across your site to ensure important pages receive enough link equity.
Discover Duplicate Content
Find pages with duplicate or near-duplicate content that confuse search engines and dilute rankings.