Web scraping tools

Quick answer

Route by stack and operating model. Scrapy fits Python teams that want self-hosted crawlers, Crawlee fits Node and TypeScript teams, and Playwright fits jobs where JavaScript rendering is the hard part. Use Firecrawl or another scraping API when fast integration and AI-ready output matter, and use Octoparse or ParseHub when a no-code workflow is the better handoff.

SearchAPI.io

Multi-Engine Real-Time SERP API

SERP APIs

SearchAPI.io delivers real-time SERP data from Google, Bing, Baidu, YouTube, and Amazon with structured JSON output.

Zyte API

All-in-One Web Scraping API with AI Extraction

Scraping FrameworksSERP APIs

Zyte API is a full-stack web scraping API that combines anti-bot bypassing, browser rendering, and AI-powered data extraction in a single endpoint.

Exa AI

Semantic Web Search API Powered by AI

SERP APIs

Exa AI is a modern search engine and web search API that uses semantic embeddings to deliver high-quality, real-time web data for AI applications.

Tavily

Real-Time Search API for AI Agents and RAG

SERP APIs

Tavily is a real-time search engine for AI agents and RAG workflows, delivering fast web search and content extraction through a single secure API.

Brave Search API

Independent Web Search API with 35B+ Pages

SERP APIs

Brave Search API provides enterprise-grade access to an independent index of 35+ billion web pages with real-time results optimized for AI and RAG pipelines.

ScraperAPI

Scale data collection with a simple API

Stars
2
Forks
1
Last commit
2 years ago

ScraperAPI handles proxy rotation, browsers, and CAPTCHAs so developers can scrape any public page with a single API call at scale.

ScrapingBee

Web scraping API with proxy and browser handling

Stars
29
Forks
4
Last commit
5 months ago

ScrapingBee is a web scraping API that handles headless browsers, rotates proxies, and offers AI-powered extraction so you can focus on your data.

DataForSEO

Comprehensive API Stack for SEO and Marketing Data

Stars
39
Forks
12
Last commit
5 months ago

DataForSEO provides SEO and digital marketing data via API, serving as a trusted data partner for 750+ software companies and agencies worldwide.

Oxylabs

Premium proxy service for data at scale

Stars
93
Forks
5
Last commit
6 months ago

Oxylabs offers 175M+ residential and datacenter proxies with web scraping APIs and browser automation tools for large-scale public data extraction.

SerpAPI

Real-Time Google Search Results API

Stars
116
Forks
11
Last commit
7 months ago

SerpAPI is a real-time API that delivers structured Google search results, handling proxies, captchas, and rich data parsing automatically.

Apify

Full-stack web scraping and data extraction platform

Stars
161
Forks
21
Last commit
5 months ago

Apify is a cloud platform for web scraping, browser automation, and data extraction with 19,000+ ready-made tools and code templates.

BeautifulSoup

Python library for parsing HTML and XML

Stars
204
Forks
57
Last commit
3 years ago

Beautiful Soup is a Python library for pulling data out of HTML and XML files, providing Pythonic idioms for navigating and searching parse trees.

Zendriver

Async-first undetected browser automation

Stars
1.1K
Forks
75
Last commit
7 months ago

Zendriver is a fast, async-first undetected browser automation framework based on nodriver with Docker support for scalable scraping.

Patchright

Undetected Playwright fork for bot evasion

Stars
2.2K
Forks
88
Last commit
5 months ago

Patchright is a patched fork of Playwright that evades bot detection while maintaining full API compatibility for seamless migration.

BeeAI Framework

Production-Ready AI Agent Framework

Stars
3.1K
Forks
400
Last commit
5 months ago

BeeAI Framework is an open-source platform for building production-ready multi-agent systems in Python and TypeScript with MCP and workflow support.

Nodriver

Undetected browser automation without Selenium

Stars
3.5K
Forks
341
Last commit
8 months ago

Nodriver is the successor to undetected-chromedriver, providing fast CDP-based browser automation with built-in anti-detection and no Selenium dependency.

parse5

HTML5 spec-compliant parser for Node.js

Stars
3.9K
Forks
251
Last commit
5 months ago

parse5 is the fastest spec-compliant HTML5 parser for Node.js, parsing HTML exactly as modern browsers do.

htmlparser2

The fast and forgiving HTML/XML parser

Stars
4.8K
Forks
394
Last commit
5 months ago

htmlparser2 is the fastest Node.js HTML and XML parser, offering a streaming, event-driven API with a forgiving approach to malformed markup.

Mechanicalsoup

Automate website interaction with Python

Stars
4.8K
Forks
385
Last commit
6 months ago

MechanicalSoup is a Python library for automating website interaction, handling cookies, redirects, forms, and links without a browser.

Camoufox

Anti-detect browser with fingerprint injection

Stars
4.9K
Forks
459
Last commit
5 months ago

Camoufox is an open-source anti-detect browser based on Firefox with robust fingerprint injection to evade bot detection systems.

LLM-Scraper

Turn Any Webpage into Structured Data with LLMs

Stars
6.2K
Forks
369
Last commit
7 months ago

LLM Scraper is a TypeScript library that extracts structured data from any webpage using LLMs with Zod schemas, Playwright, and streaming support.

Rod

High-Level DevTools Driver for Go

Stars
6.6K
Forks
437
Last commit
9 months ago

Rod is a Go library for web automation and scraping that drives browsers via the DevTools Protocol with a simple, chainable API.

Scrapling

Adaptive web scraping for the modern web

Stars
8.8K
Forks
537
Last commit
5 months ago

Scrapling is an adaptive Python scraping framework that learns from website changes, handling everything from single requests to full-scale crawls.

WebdriverIO

Next-Gen Browser and Mobile Automation for Node.js

Stars
9.7K
Forks
2.7K
Last commit
5 months ago

WebdriverIO is an all-in-one Node.js automation framework for web and mobile testing with smart selectors, component testing, and extensive plugins.

TestCafe

WebDriver-Free End-to-End Browser Testing

Stars
9.9K
Forks
677
Last commit
5 months ago

TestCafe is a free, open-source end-to-end testing framework that works across browsers without WebDriver, with intuitive syntax and built-in waiting.

Browserless

Headless browsers as a service

Stars
12.3K
Forks
949
Last commit
5 months ago

Browserless provides headless browser infrastructure in Docker for web scraping and automation, with built-in bot detection bypass and CAPTCHA solving.

Chromedp

Chrome DevTools Protocol Driver for Go

Stars
12.7K
Forks
853
Last commit
9 months ago

Chromedp is a fast, dependency-free Go package for driving browsers via the Chrome DevTools Protocol for web scraping, testing, and automation.

Goquery

jQuery-like HTML parsing for Go

Stars
14.9K
Forks
930
Last commit
6 months ago

GoQuery brings jQuery-like syntax to Go for parsing and manipulating HTML documents, making web scraping in Go intuitive and efficient.

Skyvern

AI Browser Automation with Computer Vision

Stars
20.2K
Forks
1.8K
Last commit
5 months ago

Skyvern automates browser-based workflows using LLMs and computer vision, replacing brittle scripts with AI that adapts to any webpage automatically.

Mastra

TypeScript AI Agent Framework with Workflows

Stars
20.4K
Forks
1.5K
Last commit
5 months ago

Mastra is an all-in-one TypeScript framework for building AI agents and workflows with RAG, memory, MCP support, evals, and an interactive playground.

Crawlee

Build reliable web scrapers. Fast.

Stars
21.3K
Forks
1.2K
Last commit
5 months ago

Crawlee is a JavaScript and Python web scraping library that handles blocking, proxies, and browsers so you can build reliable crawlers fast.

ScrapeGraphAI

AI-Powered Web Scraping API for Agents

Stars
22.4K
Forks
1.9K
Last commit
5 months ago

ScrapeGraphAI extracts structured data from any website using AI, handling proxies, rendering, and adapting to site changes automatically.

Colly

Elegant scraping and crawler framework for Golang

Stars
25K
Forks
1.8K
Last commit
6 months ago

Colly is a fast and elegant Go framework for building web scrapers, crawlers, and spiders with a clean, chainable API.

Cheerio

Fast, flexible HTML parsing and manipulation for JS

Stars
30K
Forks
1.7K
Last commit
5 months ago

Cheerio is the industry-standard JavaScript library for parsing and manipulating HTML and XML using a familiar jQuery-like syntax.

Selenium

Browser Automation Framework and Ecosystem

Stars
33.9K
Forks
8.6K
Last commit
5 months ago

Selenium is the original open-source browser automation framework for driving web browsers natively, supporting all major browsers and programming languages.

Crawl4AI

Open-Source LLM-Friendly Web Crawler

Stars
58.9K
Forks
6K
Last commit
5 months ago

Crawl4AI is an open-source web crawler and scraper built for LLMs, AI agents, and data pipelines with blazing-fast async crawling and AI-ready output.

SearchAPI.io

Multi-Engine Real-Time SERP API

SERP APIs

SearchAPI.io delivers real-time SERP data from Google, Bing, Baidu, YouTube, and Amazon with structured JSON output.

Zyte API

All-in-One Web Scraping API with AI Extraction

Scraping FrameworksSERP APIs

Zyte API is a full-stack web scraping API that combines anti-bot bypassing, browser rendering, and AI-powered data extraction in a single endpoint.

Exa AI

Semantic Web Search API Powered by AI

SERP APIs

Exa AI is a modern search engine and web search API that uses semantic embeddings to deliver high-quality, real-time web data for AI applications.

Tavily

Real-Time Search API for AI Agents and RAG

SERP APIs

Tavily is a real-time search engine for AI agents and RAG workflows, delivering fast web search and content extraction through a single secure API.

Brave Search API

Independent Web Search API with 35B+ Pages

SERP APIs

Brave Search API provides enterprise-grade access to an independent index of 35+ billion web pages with real-time results optimized for AI and RAG pipelines.

ScraperAPI

Scale data collection with a simple API

Stars
2
Forks
1
Last commit
2 years ago

ScraperAPI handles proxy rotation, browsers, and CAPTCHAs so developers can scrape any public page with a single API call at scale.

ScrapingBee

Web scraping API with proxy and browser handling

Stars
29
Forks
4
Last commit
5 months ago

ScrapingBee is a web scraping API that handles headless browsers, rotates proxies, and offers AI-powered extraction so you can focus on your data.

DataForSEO

Comprehensive API Stack for SEO and Marketing Data

Stars
39
Forks
12
Last commit
5 months ago

DataForSEO provides SEO and digital marketing data via API, serving as a trusted data partner for 750+ software companies and agencies worldwide.

Oxylabs

Premium proxy service for data at scale

Stars
93
Forks
5
Last commit
6 months ago

Oxylabs offers 175M+ residential and datacenter proxies with web scraping APIs and browser automation tools for large-scale public data extraction.

SerpAPI

Real-Time Google Search Results API

Stars
116
Forks
11
Last commit
7 months ago

SerpAPI is a real-time API that delivers structured Google search results, handling proxies, captchas, and rich data parsing automatically.

Apify

Full-stack web scraping and data extraction platform

Stars
161
Forks
21
Last commit
5 months ago

Apify is a cloud platform for web scraping, browser automation, and data extraction with 19,000+ ready-made tools and code templates.

BeautifulSoup

Python library for parsing HTML and XML

Stars
204
Forks
57
Last commit
3 years ago

Beautiful Soup is a Python library for pulling data out of HTML and XML files, providing Pythonic idioms for navigating and searching parse trees.

Zendriver

Async-first undetected browser automation

Stars
1.1K
Forks
75
Last commit
7 months ago

Zendriver is a fast, async-first undetected browser automation framework based on nodriver with Docker support for scalable scraping.

Patchright

Undetected Playwright fork for bot evasion

Stars
2.2K
Forks
88
Last commit
5 months ago

Patchright is a patched fork of Playwright that evades bot detection while maintaining full API compatibility for seamless migration.

BeeAI Framework

Production-Ready AI Agent Framework

Stars
3.1K
Forks
400
Last commit
5 months ago

BeeAI Framework is an open-source platform for building production-ready multi-agent systems in Python and TypeScript with MCP and workflow support.

Nodriver

Undetected browser automation without Selenium

Stars
3.5K
Forks
341
Last commit
8 months ago

Nodriver is the successor to undetected-chromedriver, providing fast CDP-based browser automation with built-in anti-detection and no Selenium dependency.

parse5

HTML5 spec-compliant parser for Node.js

Stars
3.9K
Forks
251
Last commit
5 months ago

parse5 is the fastest spec-compliant HTML5 parser for Node.js, parsing HTML exactly as modern browsers do.

htmlparser2

The fast and forgiving HTML/XML parser

Stars
4.8K
Forks
394
Last commit
5 months ago

htmlparser2 is the fastest Node.js HTML and XML parser, offering a streaming, event-driven API with a forgiving approach to malformed markup.

Mechanicalsoup

Automate website interaction with Python

Stars
4.8K
Forks
385
Last commit
6 months ago

MechanicalSoup is a Python library for automating website interaction, handling cookies, redirects, forms, and links without a browser.

Camoufox

Anti-detect browser with fingerprint injection

Stars
4.9K
Forks
459
Last commit
5 months ago

Camoufox is an open-source anti-detect browser based on Firefox with robust fingerprint injection to evade bot detection systems.

LLM-Scraper

Turn Any Webpage into Structured Data with LLMs

Stars
6.2K
Forks
369
Last commit
7 months ago

LLM Scraper is a TypeScript library that extracts structured data from any webpage using LLMs with Zod schemas, Playwright, and streaming support.

Rod

High-Level DevTools Driver for Go

Stars
6.6K
Forks
437
Last commit
9 months ago

Rod is a Go library for web automation and scraping that drives browsers via the DevTools Protocol with a simple, chainable API.

Scrapling

Adaptive web scraping for the modern web

Stars
8.8K
Forks
537
Last commit
5 months ago

Scrapling is an adaptive Python scraping framework that learns from website changes, handling everything from single requests to full-scale crawls.

WebdriverIO

Next-Gen Browser and Mobile Automation for Node.js

Stars
9.7K
Forks
2.7K
Last commit
5 months ago

WebdriverIO is an all-in-one Node.js automation framework for web and mobile testing with smart selectors, component testing, and extensive plugins.

TestCafe

WebDriver-Free End-to-End Browser Testing

Stars
9.9K
Forks
677
Last commit
5 months ago

TestCafe is a free, open-source end-to-end testing framework that works across browsers without WebDriver, with intuitive syntax and built-in waiting.

Browserless

Headless browsers as a service

Stars
12.3K
Forks
949
Last commit
5 months ago

Browserless provides headless browser infrastructure in Docker for web scraping and automation, with built-in bot detection bypass and CAPTCHA solving.

Chromedp

Chrome DevTools Protocol Driver for Go

Stars
12.7K
Forks
853
Last commit
9 months ago

Chromedp is a fast, dependency-free Go package for driving browsers via the Chrome DevTools Protocol for web scraping, testing, and automation.

Goquery

jQuery-like HTML parsing for Go

Stars
14.9K
Forks
930
Last commit
6 months ago

GoQuery brings jQuery-like syntax to Go for parsing and manipulating HTML documents, making web scraping in Go intuitive and efficient.

Skyvern

AI Browser Automation with Computer Vision

Stars
20.2K
Forks
1.8K
Last commit
5 months ago

Skyvern automates browser-based workflows using LLMs and computer vision, replacing brittle scripts with AI that adapts to any webpage automatically.

Mastra

TypeScript AI Agent Framework with Workflows

Stars
20.4K
Forks
1.5K
Last commit
5 months ago

Mastra is an all-in-one TypeScript framework for building AI agents and workflows with RAG, memory, MCP support, evals, and an interactive playground.

Crawlee

Build reliable web scrapers. Fast.

Stars
21.3K
Forks
1.2K
Last commit
5 months ago

Crawlee is a JavaScript and Python web scraping library that handles blocking, proxies, and browsers so you can build reliable crawlers fast.

ScrapeGraphAI

AI-Powered Web Scraping API for Agents

Stars
22.4K
Forks
1.9K
Last commit
5 months ago

ScrapeGraphAI extracts structured data from any website using AI, handling proxies, rendering, and adapting to site changes automatically.

Colly

Elegant scraping and crawler framework for Golang

Stars
25K
Forks
1.8K
Last commit
6 months ago

Colly is a fast and elegant Go framework for building web scrapers, crawlers, and spiders with a clean, chainable API.

Cheerio

Fast, flexible HTML parsing and manipulation for JS

Stars
30K
Forks
1.7K
Last commit
5 months ago

Cheerio is the industry-standard JavaScript library for parsing and manipulating HTML and XML using a familiar jQuery-like syntax.

Selenium

Browser Automation Framework and Ecosystem

Stars
33.9K
Forks
8.6K
Last commit
5 months ago

Selenium is the original open-source browser automation framework for driving web browsers natively, supporting all major browsers and programming languages.

Crawl4AI

Open-Source LLM-Friendly Web Crawler

Stars
58.9K
Forks
6K
Last commit
5 months ago

Crawl4AI is an open-source web crawler and scraper built for LLMs, AI agents, and data pipelines with blazing-fast async crawling and AI-ready output.

Browse 8 subcategories of Web Scraping tools and find the perfect solution for your needs.

Quick answer

Related guides

Top picks in Web Scraping

Related categories