Build a Full Site Inventory
Crawl your entire website to build a complete inventory of every page, its status, and its metadata.
Discover Duplicate Content
Find pages with duplicate or near-duplicate content that confuse search engines and dilute rankings.
ToolDuplicate content is one of the most misunderstood SEO problems. It does not cause penalties, but it does cause confusion -- when multiple pages have the same content, search engines must choose which one to index and rank. Often they choose wrong, ranking a less important version while ignoring your preferred page.
Duplicate content commonly arises from URL parameters creating multiple versions of the same page, HTTP/HTTPS and www/non-www variations, print-friendly page versions, paginated content, and CMS-generated category and tag pages that overlap with main content.
ToolRouter's crawl_site skill detects pages with matching or near-matching titles, descriptions, and content hashes. It identifies URL patterns likely to create duplicates and checks for proper canonical tag implementation across all variations.
Agent Guides
Claude
- Connect ToolRouter: claude mcp add toolrouter -- npx -y toolrouter-mcp
- Ask Claude: "Crawl mysite.com and find pages with duplicate or near-duplicate content"
- Claude identifies pages sharing titles, descriptions, or similar content
ChatGPT
- Configure ToolRouter in ChatGPT
- Ask: "Crawl mysite.com and find duplicate content issues"
- ChatGPT lists duplicate page groups with the type of duplication
Copilot
- Add ToolRouter to Copilot MCP config
- Ask: "Crawl mysite.com and detect duplicate content"
- Review duplicate page groups and their URLs
OpenClaw
- Connect ToolRouter: openclaw mcp add toolrouter -- npx -y toolrouter-mcp
- Ask: "Crawl mysite.com and find duplicate content"
- OpenClaw identifies pages with matching or similar content
Related Use Cases
Find 404 Errors Across Your Site
Crawl your website to discover all pages returning 404 errors and the internal links pointing to them.
Identify Orphaned Pages
Find pages on your site that have no internal links pointing to them, making them invisible to crawlers.
Check Internal Link Distribution
Analyze how internal links are distributed across your site to ensure important pages receive enough link equity.