How to Scrape Websites with Claude

Claude can browse the web in supported products, but specialist scraping tools are useful when you need a repeatable crawl, JavaScript rendering, or structured extraction. Connect ToolRouter to use Web Scraper for pages and site crawls, Stealth Scraper for harder-to-render public pages, and other tools for enrichment or platform-specific data. Access only content you are permitted to collect and respect site terms, robots directives, rate limits, privacy, and applicable law.

Coding with Playwright or a scraping library remains the most controllable option for developers. A connected scraper is useful when you want Claude to request the data, work with a declared output shape, and analyze the result in the same conversation.

The main trade-offs are coverage, accuracy, crawl limits, cost, and permission. No scraper can guarantee access to every site or stable extraction from every layout.

What Claude Can and Cannot Do Natively for Web Scraping#

Claude's built-in web features can find and read current sources in supported products, but that is different from a controlled scraping job:

• Claude can search and summarize current web sources in supported modes
• Claude can analyze content you paste into the conversation
• Claude can write web scraping code in Python, JavaScript, or any language you specify
• Claude can explain how scraping works, what tools exist, and how to structure a scraper
• Native browsing is not a guaranteed multi-page crawler or schema-driven extraction API
• Platform-specific posts, profiles, or comments may require a permitted specialist source

How to Connect ToolRouter and Start#

The setup takes two steps.

1. In Claude, go to Customize → Connectors → + → Add custom connector
2. Enter:

• Name: ToolRouter
• URL: https://api.toolrouter.com/mcp

Or visit toolrouter.com/connect for one-click setup.

Once connected, describe what you need rather than selecting tools manually:

• "Scrape [URL] and give me all the product names and prices."
• "Crawl [website] and give me the content of every page."
• "This public page did not render with the standard fetch. Check whether another permitted rendering method is available."
• "Pull the last 50 posts from [subreddit] and tell me what problems people are complaining about."

Claude picks the right tool and the right skill for each request, and handles multi-step workflows without you moving data between applications.

Which ToolRouter Scraping Tool to Use#

Four Scraping Workflows That Replace Manual Research#

1. Scrape Any Page or Crawl a Full Site#

Use this when you want to pull content from one page or gather everything from an entire domain.

Workflow: Web Scraper (scrape_page) for a single URL → Web Scraper (map_site + crawl_site) for a full site

Example:

Scrape the pricing page at [URL]. Give me every plan name, price, feature list, and any pricing footnotes. Format it as a comparison table.

For a full site:

Map [website URL] to see all its pages, then crawl the entire site and pull the content from every page. Summarise the site structure and flag any pages that look like they have product or pricing information.

The scrape_page skill renders JavaScript before returning content — it handles React, Vue, and SPA pages that return blank HTML to standard requests. map_site gives Claude a full URL inventory first, so crawl_site can work efficiently rather than discovering pages as it goes. Claude reads the result and structures it according to what you actually need — not just a raw dump.

2. Retry a Difficult Public Page#

Use this only when you are permitted to access the page and the standard renderer fails. A 403, 429, login, or CAPTCHA can be a signal to stop rather than evade a restriction.

Workflow: Web Scraper (scrape_page) → if blocked, Stealth Scraper (stealth_scrape) or Stealth Scraper (stealth_crawl)

Example:

The public page at [URL] did not render with the standard scraper. Check its access rules, then try an alternative permitted renderer and return the pricing content if it is accessible.

Stealth Scraper offers alternative rendering and network options for difficult public pages. Availability varies by site and provider, and the tool should not be used to defeat authentication, access controls, or an explicit prohibition.

For competitor research, prefer public pages, a low request rate, and the smallest crawl needed for the stated purpose. Keep the source URL with every extracted record so results can be checked.

3. Extract Structured Data at Scale#

Use this when you want typed, structured output from many pages rather than raw content.

Workflow: Web Scraper (extract_data) on target URLs → Lead Finder (enrich_company) to add contact and company data

Example:

Go to [directory URL] and extract every company name, website, location, and description listed on the page. Then for each company, enrich the data with their employee count, tech stack, and a contact email if available.

The extract_data skill takes a natural-language description of what you want rather than CSS selectors or XPath. You describe the schema — "I want: company name, founding year, number of employees, and LinkedIn URL" — and Claude writes the extraction logic internally and returns clean typed data. For enrichment, Lead Finder's enrich_company skill takes a domain and returns headcount, tech stack, revenue estimates, and contact information from multiple data sources.

This workflow replaces the typical scrape → CSV → manual enrichment → CRM import pipeline. The entire thing runs in one conversation.

4. Scrape Social Platforms Without an API#

Use this when you want posts, comments, or profiles from Twitter/X, Reddit, or TikTok.

Workflow: Social Reader (search_posts + read_comments) for Reddit or X, or Social Media Search (search_tiktok_keywords) for TikTok

Example for Reddit:

Search [subreddit] for posts about [topic] from the last month. Pull the top 20 posts and their comment threads. Tell me: what are the most common complaints, what solutions are people recommending, and what products keep coming up?

Example for Twitter/X:

Get the last 50 tweets from [username]. What topics do they post about most often? What gets the most engagement? Are there any patterns in what performs well?

Social Reader returns full comment trees and reads public profiles and posts, so Claude can reason about positioning and messaging rather than just counting likes. Social Media Search covers TikTok — useful for understanding what content formats work in a niche before investing in production.

None of these require you to register for a developer API or manage rate limits. Claude handles the calls and synthesises the results into what you actually want to know.

Best Prompts for Web Scraping in Claude#

Scrape a competitor's pricing page#

Scrape the pricing page at [URL].
Return every plan, its price (monthly and annual if both exist),
every feature listed under each plan, and any footnotes or asterisks.
Format it as a side-by-side comparison table.

Crawl a site and audit its content#

Map [website URL] to get a full list of pages.
Then crawl the site and return the content of every page.
Give me:
1. A list of all pages with a one-line summary of each
2. Any pages that have pricing, product, or contact information highlighted separately
3. Any pages that appear thin or outdated

Extract structured data from a directory#

Scrape [directory or listing page URL].
For each company or item listed, extract:
- Name
- Website URL
- Description or tagline
- Location (if shown)
- Any other fields visible on the page

Return the results as a structured table.

Research what a community is saying about a topic#

Search Reddit for posts about [topic] in [subreddit or across all of Reddit].
Pull the top 20 posts and their top comments.
Tell me:
1. The most common pain points or complaints
2. Products or solutions people recommend most often
3. Any recurring questions that don't have good answers yet

Why One Connector Beats a Stack of Scraping Tools#

The standard web scraping workflow for non-developers: find an online scraping tool, paste a URL, download a CSV, open it in a spreadsheet, manually remove the junk columns, realise the tool doesn't handle JavaScript, find another tool, repeat.

The practical advantage of running the workflow through Claude is continuity: returned data stays available in the conversation when you ask Claude to filter, rank, or summarize it. The request and result are still processed by the connector, ToolRouter, and relevant downstream providers, so do not send sensitive data unless those services and their retention terms are appropriate for the task.

For setup instructions see How to Add Connectors to Claude. If you are scraping sites to build a lead list, the enrichment and outreach steps are covered in How to Find Leads in Claude. For SEO research, browse the SEO tools. For the broader product model, see What Is ToolRouter?.