How to Map Website Structure with Claude
Map complete website structure using Claude and ToolRouter. Discover all URLs and site architecture automatically.
Tool
Claude maps website structure by scraping page content and navigation patterns, then helps you analyze the architecture interactively. After the crawl, discuss information architecture improvements, ask Claude to identify navigation bottlenecks, or generate a sitemap document from the discovered URL hierarchy.
Connect ToolRouter to Claude
1Open connector settings Open Settings
2Add a custom connector with these details
Name
ToolRouterURL
https://api.toolrouter.com/mcp3Let Claude set you up Open Claude
Steps
Once connected (see setup above), use the Web Scraper tool:
- Ask Claude: "Map the complete site structure of example.com using the web-scraper tool"
- Claude runs map_site to discover all URLs and builds a hierarchical site map
- Ask: "Which sections have the most pages? Are there any orphan pages?"
- Claude analyzes the structure and identifies architectural issues
Example Prompt
Try this with Claude using the Web Scraper tool
Map the full site structure of example.com. Show me the URL hierarchy, page counts per section, and any orphan pages.
Tips
- Claude can visualize site structure as a tree for easy understanding
- Ask for a comparison between the sitemap.xml and actual discoverable pages
- Use the map to plan site migrations without losing any pages