Get started
Tools / Web Scraper / Use Cases / Map Complete Website Structure and URLs / Claude

How to Map Website Structure with Claude

Map complete website structure using Claude and ToolRouter. Discover all URLs and site architecture automatically.

Tool
Web Scraper icon
Web Scraper

Claude maps website structure by scraping page content and navigation patterns, then helps you analyze the architecture interactively. After the crawl, discuss information architecture improvements, ask Claude to identify navigation bottlenecks, or generate a sitemap document from the discovered URL hierarchy.

Connect ToolRouter to Claude

1Go to Settings → Connectors → Add custom connector
2Enter the details below and click Add
Name
ToolRouter
URL
https://api.toolrouter.com/mcp
3Done — works on Claude chat, desktop, and mobile
No API key needed — account created automatically.All platforms →

Steps

Once connected (see setup above), use the Web Scraper tool:

  1. Ask Claude: "Map the complete site structure of example.com using the web-scraper tool"
  2. Claude runs map_site to discover all URLs and builds a hierarchical site map
  3. Ask: "Which sections have the most pages? Are there any orphan pages?"
  4. Claude analyzes the structure and identifies architectural issues

Example Prompt

Try this with Claude using the Web Scraper tool
Map the full site structure of example.com. Show me the URL hierarchy, page counts per section, and any orphan pages.

Tips

  • Claude can visualize site structure as a tree for easy understanding
  • Ask for a comparison between the sitemap.xml and actual discoverable pages
  • Use the map to plan site migrations without losing any pages

More Claude Guides

How to Extract Product Data with ClaudeHow to Monitor Competitor Pricing with ClaudeHow to Scrape Job Listings with ClaudeHow to Extract Contact Info with ClaudeHow to Scrape Reviews for Sentiment with Claude

Related Workflows

Ecommerce Competitor IntelligenceGather ecommerce competitive intelligence by scraping pricing data, researching businesses, monitoring social commerce, and analyzing ad campaigns.Real Estate Market ResearchResearch real estate markets by scraping listings, analyzing locations, visualizing pricing data, and tracking market news.Pricing IntelligenceMonitor and analyze competitor pricing by extracting price data, analyzing strategies, normalizing currencies, and visualizing comparisons.Talent Market ResearchResearch the talent market by scraping job data, benchmarking salaries, analyzing company hiring signals, and comparing labor markets.Web Content ArchiverArchive web pages completely with archive checks, content extraction, and visual screenshots.