Get started
Tools / Web Scraper
Web Scraper icon

Web Scraper

Scrape, crawl & extract web data

Try with Claude

Turn any website into structured data with JS rendering, anti-bot bypass, and automatic extraction. Scrape pages, crawl sites, discover URLs, extract typed data with AI, or search and scrape at once. Supports markdown, HTML, CSS filtering, and mobile viewports.

Scrape Page

Scrape a single web page with full JavaScript rendering, anti-bot bypass, and configurable output formats. Supports markdown, HTML, and content filtering by CSS tags.

Returns: Scraped page content in the requested formats (markdown, HTML, etc.) with metadata, or parsed JSON for API endpoints
Crawl Site

Recursively crawl a website starting from a URL, following links up to a configurable depth and page limit. Returns scraped content for all discovered pages.

Returns: Array of scraped pages with content, metadata, and crawl status
Map Site

Quickly discover all URLs on a website using sitemaps and link analysis. Returns a flat list of URLs without scraping content. Optionally filter by keyword.

Returns: List of discovered URLs from the website via sitemaps and link analysis
Extract Structured Data

Use AI to extract structured data from one or more web pages. Provide a JSON Schema for typed output or a natural language prompt for flexible extraction.

Returns: AI-extracted structured data from the provided URLs, matching the schema or prompt
Search & Scrape

Search the web using a query and optionally scrape the content of each result page. Returns search results with titles, URLs, and snippets, plus full page content when scraping is enabled.

Returns: Search results with titles, URLs, snippets, and optionally full scraped page content in markdown
Stealth Scrape Page

Scrape a single bot-protected web page using enhanced residential proxies, geo-targeted IPs, and extended rendering wait times. Bypasses Cloudflare, Akamai, DataDome, and similar anti-bot systems.

Returns: Scraped page content from bot-protected sites in the requested formats with metadata
Stealth Crawl Site

Recursively crawl a bot-protected website using enhanced proxies on every page. Bypasses anti-bot systems across the entire crawl, with geo-targeted IPs and extended rendering.

Returns: Array of scraped pages from bot-protected sites with content, metadata, and crawl status
Loading reviews...

Loading activity...

v0.052026-04-04
  • scrape_page now handles JSON API endpoints — returns parsed JSON and markdown instead of empty content
  • Added headers parameter to scrape_page for custom request headers (e.g. Authorization)
v0.042026-03-23
  • Added stealth_scrape and stealth_crawl skills for bot-protected websites
v0.032026-03-23
  • Added proxy, country, and languages parameters to scrape_page and crawl_site for anti-bot bypass and geo-targeting
v0.022026-03-22
  • Added subtitle, expanded description, and agent instructions
v0.012026-03-20
  • Initial release

Use Cases

Open Search Papers by Topic

Search Papers by Topic

Find relevant academic papers on any research topic across millions of scholarly publications.

Academic Research icon
Academic Research4 agent guides
Open Geocode Addresses to Coordinates

Geocode Addresses to Coordinates

Convert street addresses into precise latitude and longitude coordinates for mapping and spatial analysis.

Address Geocoding icon
Address Geocoding4 agent guides
View all use cases for Web Scraper→

Workflows

Ecommerce Competitor Intelligence
Web Scraper icon
Web Scraper
Competitor Research icon
Competitor Research
Social Shop Products icon
Social Shop Products
Ad Library Search icon
Ad Library Search
Real Estate Market Research
Web Scraper icon
Web Scraper
GEO icon
GEO
Generate Chart icon
Generate Chart
Web Search icon
Web Search
Pricing Intelligence
Web Scraper icon
Web Scraper
Competitor Research icon
Competitor Research
Currency Exchange icon
Currency Exchange
Generate Chart icon
Generate Chart
View all 5 workflows→

Related Tools

Open Web Search
Web Search icon
Web SearchWeb, news, images & maps — one tool
★★★★★2

Related Categories

🗄️ Data💻 Development

Popular with

Amazon SellerAttorneyCategory ManagerConversion Rate OptimizerCopywriterDigital Transformation ConsultantE-Commerce CopywriterE-Commerce Manager

Frequently Asked Questions

Can it handle JavaScript-heavy sites?

Yes. JavaScript rendering is built in, so dynamic pages are part of the normal workflow.

Can it crawl and discover new URLs?

Yes. It can scrape a page, crawl sites, and discover URLs in one tool.

What formats can I extract?

You can pull markdown, HTML, or typed data depending on how structured you want the result to be.

Does it help with anti-bot pages?

Yes. Anti-bot bypass is part of the web access surface for the tool.