---
title: ToolRouter — Tool Detail
description: Detailed documentation for a ToolRouter tool including skills, parameters, and examples.
url: https://toolrouter.com/tools/site-crawler
---
# Site Crawler

Walk a website from any URL, collecting per-page titles, descriptions, heading counts, word counts, response times, and link counts. Ideal for site audits, content inventories, broken-link detection, and SEO analysis.

- **Version:** 0.02
- **Categories:** search, analytics, development

## Skills (1)

### Crawl Site (`crawl_site`)

Crawl pages from a starting URL and collect metadata including title, description, heading counts, link counts, response time, and crawl diagnostics.

**Parameters:**

| Name | Type | Required | Description |
|------|------|----------|-------------|
| `url` | string | Yes | Starting URL or hostname to crawl |
| `max_pages` | number | No | Maximum number of pages to crawl (1-100) [default: 20] |
| `timeout_ms` | number | No | Request timeout per page in milliseconds (3000-60000) [default: 10000] |
| `same_origin_only` | boolean | No | Only crawl pages on the same origin as the start URL [default: true] |
| `include_subdomains` | boolean | No | Treat subdomains as internal links when same-origin mode is enabled [default: false] |
| `include_external` | boolean | No | Queue external links for crawling when same_origin_only is false [default: false] |

**Returns:** Bounded crawl results with per-page SEO metadata, link graph hints, response timing, and failure diagnostics

**Examples:**

*Crawl a small sample of example.com pages:*

```json
{
  "url": "https://example.com",
  "max_pages": 5
}
```

## Quick Start

```bash
# Via CLI
TOOLROUTER_API_KEY=tr_live_xxx npx -y toolrouter-mcp call site-crawler crawl_site --url https://example.com --max_pages 5

# Via REST API
curl -H "Authorization: Bearer tr_live_xxx" \
     -d '{"tool":"site-crawler","skill":"crawl_site","input":{"url":"https://example.com","max_pages":5}}' \
     https://api.toolrouter.com/v1/tools/call
```

## Frequently Asked Questions

### Can it crawl a whole site or just one page?

It walks a website from a starting URL and gathers data page by page, so it works for both single-page and full-site audits.

### What page data does it collect?

It collects titles, descriptions, heading counts, word counts, response times, and link counts.

### Is it useful for broken-link or content inventory work?

Yes. Those are two of the most practical uses for the crawler.

### Do I need to start from the homepage?

No. You can start from any URL and let the crawl expand from there.

---

## Get Started with ToolRouter

**Already connected?** Call any tool with `use_tool({ tool, skill, input })`. Browse tools with `discover` or visit https://toolrouter.com/llms-full.txt

**New to ToolRouter?** Connect with one command (no API key needed — auto-provisions on first use):

```bash
# MCP (Claude Code, Codex CLI)
claude mcp add toolrouter -- npx -y toolrouter-mcp

# MCP (Cursor, Windsurf, Cline, VS Code, Gemini CLI, Claude Desktop)
{"mcpServers":{"toolrouter":{"command":"npx","args":["-y","toolrouter-mcp"]}}}

# Claude Chat / Claude Cowork / Claude Mobile — add as a connector:
# Settings → Connectors → Add custom connector → URL: https://api.toolrouter.com/mcp

# ChatGPT — add as an app:
# Settings → Apps → Advanced settings → enable Developer mode → Create app → MCP Server URL: https://api.toolrouter.com/mcp

# CLI (works instantly, no restart needed)
npx -y toolrouter-mcp tools                    # browse all tools
npx -y toolrouter-mcp search "seo"             # search by keyword
npx -y toolrouter-mcp call seo analyze_page --url https://example.com
```

Free tools work immediately. For paid tools, visit https://toolrouter.com/claim to add credits.

All platforms: https://toolrouter.com/connect | Full catalog: https://toolrouter.com/llms-full.txt | Docs: https://toolrouter.com/docs