Tools / Web Scraper / Use Cases / Map Complete Website Structure and URLs

Map Complete Website Structure and URLs

Map the complete structure of a website including all pages, URL hierarchy, and internal linking patterns.

Tool
Web Scraper icon
Web Scraper

Understanding the full structure of a website is essential for SEO audits, migration planning, competitive analysis, and content inventory management. A site map reveals how pages are organized, which sections are the deepest, where orphan pages hide, and how the internal linking structure distributes page authority.

ToolRouter's map_site skill discovers all accessible URLs on a domain and builds a hierarchical map of the site structure, while crawl_site follows links to verify page status and connectivity. Together, they produce a complete picture of a website's architecture — from the homepage down to the deepest pages.

SEO professionals use site maps to identify structural issues that hurt crawlability. Developers use them before migrations to ensure no pages are lost. Content strategists use them to inventory existing content and identify gaps. Competitive analysts use them to understand how competitors organize their sites and where they invest in content.

Agent Guides

Claude

  1. Connect ToolRouter to Claude: claude mcp add toolrouter -- npx -y toolrouter-mcp
  2. Ask Claude: "Map the complete site structure of example.com using the web-scraper tool"
  3. Claude runs map_site to discover all URLs and builds a hierarchical site map
Read full guide →

ChatGPT

  1. Configure ToolRouter in ChatGPT via MCP settings
  2. Ask: "Map the complete structure of example.com and show me how the site is organized"
  3. ChatGPT crawls the site and presents the URL hierarchy
Read full guide →

Copilot

  1. Add ToolRouter to your Copilot MCP configuration
  2. Ask Copilot: "Map the site structure of example.com"
  3. Review the URL hierarchy and page structure in your editor
Read full guide →

OpenClaw

  1. Connect ToolRouter to OpenClaw: openclaw mcp add toolrouter -- npx -y toolrouter-mcp
  2. Ask: "Map the complete site structure of example.com"
  3. OpenClaw discovers all URLs and builds the site hierarchy
Read full guide →

Related Use Cases