Web Scraper — Scrape, crawl & extract web data

Tools / Web Scraper

Scrape Page

Scrape a single web page with full JavaScript rendering, anti-bot bypass, and configurable output formats. Supports markdown, HTML, and content filtering by CSS tags.

Returns: Scraped page content in the requested formats (markdown, HTML, etc.) with metadata

Parameters

url *stringURL of the page to scrape

formatsarrayOutput formats to return (e.g. "markdown", "html", "rawHtml", "links", "screenshot")

onlyMainContentbooleanExtract only the main content, removing navbars, footers, and sidebars

includeTagsarrayCSS tags to include in extraction (e.g. ["article", "main"])

excludeTagsarrayCSS tags to exclude from extraction (e.g. ["nav", "footer"])

mobilebooleanUse a mobile user agent and viewport for rendering

waitFornumberMilliseconds to wait after page load before capturing content

timeoutnumberMaximum time in milliseconds to wait for the page to load

proxystringProxy type: "basic" (fast, default), "enhanced" (anti-bot bypass, slower), or "auto" (tries basic first, falls back to enhanced)

countrystringISO country code for geo-targeted proxy (e.g. "us", "gb", "de", "jp")

languagesarrayBrowser language headers for geo-targeted requests (e.g. ["en-US", "en"])

Example

Scrape a page as markdown

curl -H "Authorization: Bearer $TOOLROUTER_API_KEY" \
  -d '{
  "tool": "web-scraper",
  "skill": "scrape_page",
  "input": {
    "url": "https://example.com"
  }
}' \
  https://api.toolrouter.com/v1/tools/call

Crawl Site

Recursively crawl a website starting from a URL, following links up to a configurable depth and page limit. Returns scraped content for all discovered pages.

Returns: Array of scraped pages with content, metadata, and crawl status

Parameters

url *stringStarting URL for the crawl

limitnumberMaximum number of pages to crawl (default 50)

maxDepthnumberMaximum link-following depth from the starting URL

includePathsarrayURL path patterns to include (e.g. ["/blog/*", "/docs/*"])

excludePathsarrayURL path patterns to exclude (e.g. ["/admin/*", "/api/*"])

allowSubdomainsbooleanWhether to follow links to subdomains of the starting URL

allowExternalLinksbooleanWhether to follow links to external domains

proxystringProxy type: "basic" (fast, default), "enhanced" (anti-bot bypass, slower), or "auto" (tries basic first, falls back to enhanced)

countrystringISO country code for geo-targeted proxy (e.g. "us", "gb", "de", "jp")

languagesarrayBrowser language headers for geo-targeted requests (e.g. ["en-US", "en"])

Example

Crawl a blog with a 20-page limit

curl -H "Authorization: Bearer $TOOLROUTER_API_KEY" \
  -d '{
  "tool": "web-scraper",
  "skill": "crawl_site",
  "input": {
    "url": "https://example.com/blog",
    "limit": 20
  }
}' \
  https://api.toolrouter.com/v1/tools/call

Map Site

Quickly discover all URLs on a website using sitemaps and link analysis. Returns a flat list of URLs without scraping content. Optionally filter by keyword.

Returns: List of discovered URLs from the website via sitemaps and link analysis

Parameters

url *stringWebsite URL to map

limitnumberMaximum number of URLs to return (default 1000)

searchstringKeyword filter to narrow down discovered URLs

Example

Map all URLs on a website

curl -H "Authorization: Bearer $TOOLROUTER_API_KEY" \
  -d '{
  "tool": "web-scraper",
  "skill": "map_site",
  "input": {
    "url": "https://example.com"
  }
}' \
  https://api.toolrouter.com/v1/tools/call

Extract Structured Data

Use AI to extract structured data from one or more web pages. Provide a JSON Schema for typed output or a natural language prompt for flexible extraction.

Returns: AI-extracted structured data from the provided URLs, matching the schema or prompt

Parameters

urls *arrayURLs to extract data from

schemaobjectJSON Schema defining the structure of data to extract

promptstringNatural language prompt describing what data to extract

Example

Extract product details using a schema

curl -H "Authorization: Bearer $TOOLROUTER_API_KEY" \
  -d '{
  "tool": "web-scraper",
  "skill": "extract_data",
  "input": {
    "urls": [
      "https://example.com/product/123"
    ],
    "schema": {
      "type": "object",
      "properties": {
        "name": {
          "type": "string"
        },
        "price": {
          "type": "number"
        },
        "currency": {
          "type": "string"
        }
      }
    }
  }
}' \
  https://api.toolrouter.com/v1/tools/call

Search & Scrape

Search the web using a query and optionally scrape the content of each result page. Returns search results with titles, URLs, and snippets, plus full page content when scraping is enabled.

Returns: Search results with titles, URLs, snippets, and optionally full scraped page content in markdown

Parameters

query *stringSearch query string

limitnumberMaximum number of search results to return (default 5)

scrapeResultsbooleanWhether to scrape the full content of each search result page (default false)

countrystringCountry code for localized results (e.g. "us", "gb", "de")

Example

Search for a topic

curl -H "Authorization: Bearer $TOOLROUTER_API_KEY" \
  -d '{
  "tool": "web-scraper",
  "skill": "search_web",
  "input": {
    "query": "best practices for web scraping 2024"
  }
}' \
  https://api.toolrouter.com/v1/tools/call

Stealth Scrape Page

Scrape a single bot-protected web page using enhanced residential proxies, geo-targeted IPs, and extended rendering wait times. Bypasses Cloudflare, Akamai, DataDome, and similar anti-bot systems.

Returns: Scraped page content from bot-protected sites in the requested formats with metadata

Parameters

url *stringURL of the page to scrape

formatsarrayOutput formats to return (e.g. "markdown", "html", "rawHtml", "links", "screenshot")

onlyMainContentbooleanExtract only the main content, removing navbars, footers, and sidebars

includeTagsarrayCSS tags to include in extraction (e.g. ["article", "main"])

excludeTagsarrayCSS tags to exclude from extraction (e.g. ["nav", "footer"])

mobilebooleanUse a mobile user agent and viewport for rendering

waitFornumberMilliseconds to wait after page load before capturing content (default 3000)

timeoutnumberMaximum time in milliseconds to wait for the page to load (default 60000)

countrystringISO country code for geo-targeted proxy (e.g. "us", "gb", "de", "jp"). Default: "us"

languagesarrayBrowser language headers (e.g. ["en-US", "en"])

Example

Scrape a Cloudflare-protected page

curl -H "Authorization: Bearer $TOOLROUTER_API_KEY" \
  -d '{
  "tool": "web-scraper",
  "skill": "stealth_scrape",
  "input": {
    "url": "https://example.com/protected-page"
  }
}' \
  https://api.toolrouter.com/v1/tools/call

Stealth Crawl Site

Recursively crawl a bot-protected website using enhanced proxies on every page. Bypasses anti-bot systems across the entire crawl, with geo-targeted IPs and extended rendering.

Returns: Array of scraped pages from bot-protected sites with content, metadata, and crawl status

Parameters

url *stringStarting URL for the crawl

limitnumberMaximum number of pages to crawl (default 50)

maxDepthnumberMaximum link-following depth from the starting URL

includePathsarrayURL path patterns to include (e.g. ["/blog/*", "/docs/*"])

excludePathsarrayURL path patterns to exclude (e.g. ["/admin/*", "/api/*"])

allowSubdomainsbooleanWhether to follow links to subdomains of the starting URL

allowExternalLinksbooleanWhether to follow links to external domains

countrystringISO country code for geo-targeted proxy (e.g. "us", "gb", "de", "jp"). Default: "us"

languagesarrayBrowser language headers (e.g. ["en-US", "en"])

Example

Crawl a protected blog section

curl -H "Authorization: Bearer $TOOLROUTER_API_KEY" \
  -d '{
  "tool": "web-scraper",
  "skill": "stealth_crawl",
  "input": {
    "url": "https://example.com/blog",
    "limit": 20
  }
}' \
  https://api.toolrouter.com/v1/tools/call

Quick Start

MCP (Claude Code)

claude mcp add --transport stdio \
  --env TOOLROUTER_API_KEY=YOUR_API_KEY \
  toolrouter -- npx -y toolrouter-mcp

REST API

curl -H "Authorization: Bearer $TOOLROUTER_API_KEY" \
  -d '{"tool":"web-scraper","skill":"scrape_page","input":{}}' \
  https://api.toolrouter.com/v1/tools/call