Web scraping tools directory

Kadoa

Self-Healing AI Web Data Pipelines

AI Web ScrapingWeb Scraping

AI web scraping platform for building self-healing extraction agents that monitor pages and collect structured data at scale.

pgvector

Add Vector Search To PostgreSQL

Stars
23.3K
Forks
1.2K
Last recorded push
3 months ago

Open-source PostgreSQL extension for vector similarity search that keeps embeddings and scraped data in the same database.

Apache StormCrawler

Scalable Java Crawlers For Teams

Stars
1K
Forks
282
Last recorded push
3 months ago

Build scalable Java crawlers on Apache Storm with low-latency processing, flexible customization, and an Apache-2.0 license.

got-scraping

Browser-Like Requests For Reliable Scraping

Stars
774
Forks
60
Last recorded push
4 months ago

A Node.js HTTP client for scraping that sends browser-like requests with realistic headers and TLS behavior.

Parsel

Extract Web Data Cleanly With Python

Stars
1.4K
Forks
162
Last recorded push
3 months ago

A Python library for extracting data from HTML, XML, and JSON with CSS, XPath, JMESPath, and regex.

Apache Solr

Scalable Search For Scraped Content

Stars
1.7K
Forks
839
Last recorded push
3 months ago

Open-source Java search software for indexing scraped content with full-text search and faceted navigation.

jsoup

Parse Real-World HTML In Java

Stars
11.4K
Forks
2.3K
Last recorded push
3 months ago

A Java library for parsing real-world HTML into a standards-based DOM with CSS selectors and DOM traversal.

selectolax

Fast HTML Parsing for Python Workflows

Stars
1.7K
Forks
91
Last recorded push
4 months ago

Parse HTML with CSS selectors in Python using a fast, MIT-licensed library built for large-scale extraction.

SeleniumBase

Reliable Browser Automation For Python

Stars
13.1K
Forks
1.6K
Last recorded push
3 months ago

An open-source Python framework for browser automation, testing, and practical anti-detection scraping with UC Mode and CDP Mode.

puppeteer-extra

Stealth Plugins For Puppeteer Scraping

Stars
7.4K
Forks
781
Last recorded push
2 years ago

A JavaScript plugin framework for Puppeteer that adds stealth options to make headless Chrome harder to detect.

curl_cffi

Browser-Like Requests Without Full Browsers

Stars
6.7K
Forks
501
Last recorded push
3 months ago

Python HTTP client for scraping with browser-like TLS, JA3, and HTTP/2 fingerprint impersonation without running a full browser.

Unstructured

Clean Documents For AI Pipelines

Stars
15.6K
Forks
1.3K
Last recorded push
3 months ago

An open-source Python document ETL tool that turns PDFs, HTML, images, and more into structured elements for RAG and LLM workflows.

Diffbot

Turn Web Pages Into Usable Data

AI Web Scraping

An AI extraction platform that turns web pages into structured data and a knowledge graph for web data workflows.

lxml

Fast XML And HTML Parsing For Python

Stars
3.1K
Forks
619
Last recorded push
3 months ago

A fast Python library for parsing XML and HTML, often used as a foundation for scraping and data tools.

Apache Tika

Extract Text From Any Document

Stars
4.1K
Forks
941
Last recorded push
3 months ago

Detects file types and extracts text and document details from 1,000+ formats through one Java interface.

Docling

Turn Documents Into AI-Ready Data

Stars
68.7K
Forks
4.4K
Last recorded push
3 months ago

Open-source Python software that converts PDFs, DOCX, PPTX, and images into structured data for GenAI and RAG pipelines.

Scrapfly

AI-Ready Web Data Extraction

Scraping APIs

A commercial scraping platform that helps teams collect structured web data, automate cloud browsers, crawl sites, handle anti-bot blocks, and use proxies.

ZenRows

AI-Ready Scraping With Anti-Bot Bypass

Scraping APIs

A commercial scraping API for bypassing bot defenses, rendering JavaScript, using residential proxies, and returning AI-ready results.

Stagehand

Automate Web Workflows With AI

Stars
25.6K
Forks
1.6K
Last recorded push
3 months ago

Stagehand is an open-source SDK for AI-assisted browser automation with natural-language actions, structured extraction, and Playwright-style APIs.

Browserbase

Reliable Cloud Browsers for AI Agents

Headless Browsers

Run hosted browser sessions for AI automation and scraping with search, fetch, stealth, observability, and replay built in.

Octoparse

Turn web pages into structured data, no code

Web Scraping

Build scrapers visually and turn web pages into structured data in minutes, with scheduling and an optional managed data service.

Perplexity Sonar

Generative AI Search API with Real-Time Research

SERP APIs

Perplexity Sonar provides models for web-grounded answers. Support for Sonar Chat Completions ended on September 27, 2026. Perplexity recommends Agent API for new integrations; existing clients should follow its migration guide.

SearchAPI.io

Multi-Engine Real-Time SERP API

SERP APIs

SearchAPI.io delivers real-time SERP data from Google, Bing, Baidu, YouTube, and Amazon with structured JSON output.

Zyte API

All-in-One Web Scraping API with AI Extraction

Scraping FrameworksSERP APIs

Zyte API is a full-stack web scraping API that combines anti-bot bypassing, browser rendering, and AI-powered data extraction in a single endpoint.

Exa AI

Semantic Web Search API Powered by AI

SERP APIs

Exa AI is a modern search engine and web search API that uses semantic embeddings to deliver high-quality, real-time web data for AI applications.

Tavily

Real-Time Search API for AI Agents and RAG

SERP APIs

Tavily is a real-time search engine for AI agents and RAG workflows, delivering fast web search and content extraction through a single secure API.

Brave Search API

Independent Web Search API with 35B+ Pages

SERP APIs

Brave Search API provides enterprise-grade access to an independent index of 35+ billion web pages with real-time results optimized for AI and RAG pipelines.

Cloudflare R2

S3-compatible storage with zero egress fees

Object Storage

Cloudflare R2 is S3-compatible object storage with zero egress fees, making it ideal for storing and serving frequently accessed scraped data.

AWS Athena

Serverless SQL Queries on Amazon S3

ETL Tools

Amazon Athena is a serverless, interactive query service that lets you analyze petabyte-scale data in S3 using standard SQL with pay-per-query pricing.

AWS Glue

Serverless Data Integration Service by AWS

ETL Tools

AWS Glue is a fully managed, serverless ETL service that discovers, prepares, and integrates data from over 100 sources into lakes and warehouses.

Webshare

Fast and affordable proxy servers

Proxy Services

Webshare provides self-serve proxy servers across many locations, offering datacenter and residential proxy options for scraping workflows.

IPRoyal

Premium proxies at affordable prices

Proxy Services

IPRoyal offers residential, datacenter, ISP, and mobile proxy products with self-service access for scraping workflows.

Smartproxy

AI-ready proxy and scraping solutions

Proxy Services

Smartproxy (now Decodo) offers residential and datacenter proxy products with rotating and sticky sessions for scalable scraping.

Render

Cloud platform for web services and cron jobs

Cloud Compute

Render is a cloud platform that deploys web services, background workers, and cron jobs with simple scaling for scraping workloads.

Railway

Deploy apps from GitHub through configured exporters or custom item pipelines

Cloud Compute

Railway deploys web apps and services from GitHub with auto-scaling, built-in databases, and instant previews for scraping workloads.

AWS Lambda

Serverless compute for event-driven pipelines

Cloud Compute

AWS Lambda runs code without managing servers, ideal for event-driven scraping pipelines triggered by schedules, queues, or API calls.

Kadoa

Self-Healing AI Web Data Pipelines

AI Web ScrapingWeb Scraping

AI web scraping platform for building self-healing extraction agents that monitor pages and collect structured data at scale.

pgvector

Add Vector Search To PostgreSQL

Stars
23.3K
Forks
1.2K
Last recorded push
3 months ago

Open-source PostgreSQL extension for vector similarity search that keeps embeddings and scraped data in the same database.

Apache StormCrawler

Scalable Java Crawlers For Teams

Stars
1K
Forks
282
Last recorded push
3 months ago

Build scalable Java crawlers on Apache Storm with low-latency processing, flexible customization, and an Apache-2.0 license.

got-scraping

Browser-Like Requests For Reliable Scraping

Stars
774
Forks
60
Last recorded push
4 months ago

A Node.js HTTP client for scraping that sends browser-like requests with realistic headers and TLS behavior.

Parsel

Extract Web Data Cleanly With Python

Stars
1.4K
Forks
162
Last recorded push
3 months ago

A Python library for extracting data from HTML, XML, and JSON with CSS, XPath, JMESPath, and regex.

Apache Solr

Scalable Search For Scraped Content

Stars
1.7K
Forks
839
Last recorded push
3 months ago

Open-source Java search software for indexing scraped content with full-text search and faceted navigation.

jsoup

Parse Real-World HTML In Java

Stars
11.4K
Forks
2.3K
Last recorded push
3 months ago

A Java library for parsing real-world HTML into a standards-based DOM with CSS selectors and DOM traversal.

selectolax

Fast HTML Parsing for Python Workflows

Stars
1.7K
Forks
91
Last recorded push
4 months ago

Parse HTML with CSS selectors in Python using a fast, MIT-licensed library built for large-scale extraction.

SeleniumBase

Reliable Browser Automation For Python

Stars
13.1K
Forks
1.6K
Last recorded push
3 months ago

An open-source Python framework for browser automation, testing, and practical anti-detection scraping with UC Mode and CDP Mode.

puppeteer-extra

Stealth Plugins For Puppeteer Scraping

Stars
7.4K
Forks
781
Last recorded push
2 years ago

A JavaScript plugin framework for Puppeteer that adds stealth options to make headless Chrome harder to detect.

curl_cffi

Browser-Like Requests Without Full Browsers

Stars
6.7K
Forks
501
Last recorded push
3 months ago

Python HTTP client for scraping with browser-like TLS, JA3, and HTTP/2 fingerprint impersonation without running a full browser.

Unstructured

Clean Documents For AI Pipelines

Stars
15.6K
Forks
1.3K
Last recorded push
3 months ago

An open-source Python document ETL tool that turns PDFs, HTML, images, and more into structured elements for RAG and LLM workflows.

Diffbot

Turn Web Pages Into Usable Data

AI Web Scraping

An AI extraction platform that turns web pages into structured data and a knowledge graph for web data workflows.

lxml

Fast XML And HTML Parsing For Python

Stars
3.1K
Forks
619
Last recorded push
3 months ago

A fast Python library for parsing XML and HTML, often used as a foundation for scraping and data tools.

Apache Tika

Extract Text From Any Document

Stars
4.1K
Forks
941
Last recorded push
3 months ago

Detects file types and extracts text and document details from 1,000+ formats through one Java interface.

Docling

Turn Documents Into AI-Ready Data

Stars
68.7K
Forks
4.4K
Last recorded push
3 months ago

Open-source Python software that converts PDFs, DOCX, PPTX, and images into structured data for GenAI and RAG pipelines.

Scrapfly

AI-Ready Web Data Extraction

Scraping APIs

A commercial scraping platform that helps teams collect structured web data, automate cloud browsers, crawl sites, handle anti-bot blocks, and use proxies.

ZenRows

AI-Ready Scraping With Anti-Bot Bypass

Scraping APIs

A commercial scraping API for bypassing bot defenses, rendering JavaScript, using residential proxies, and returning AI-ready results.

Stagehand

Automate Web Workflows With AI

Stars
25.6K
Forks
1.6K
Last recorded push
3 months ago

Stagehand is an open-source SDK for AI-assisted browser automation with natural-language actions, structured extraction, and Playwright-style APIs.

Browserbase

Reliable Cloud Browsers for AI Agents

Headless Browsers

Run hosted browser sessions for AI automation and scraping with search, fetch, stealth, observability, and replay built in.

Octoparse

Turn web pages into structured data, no code

Web Scraping

Build scrapers visually and turn web pages into structured data in minutes, with scheduling and an optional managed data service.

Perplexity Sonar

Generative AI Search API with Real-Time Research

SERP APIs

SearchAPI.io

Multi-Engine Real-Time SERP API

SERP APIs

SearchAPI.io delivers real-time SERP data from Google, Bing, Baidu, YouTube, and Amazon with structured JSON output.

Zyte API

All-in-One Web Scraping API with AI Extraction

Scraping FrameworksSERP APIs

Zyte API is a full-stack web scraping API that combines anti-bot bypassing, browser rendering, and AI-powered data extraction in a single endpoint.

Exa AI

Semantic Web Search API Powered by AI

SERP APIs

Exa AI is a modern search engine and web search API that uses semantic embeddings to deliver high-quality, real-time web data for AI applications.

Tavily

Real-Time Search API for AI Agents and RAG

SERP APIs

Tavily is a real-time search engine for AI agents and RAG workflows, delivering fast web search and content extraction through a single secure API.

Brave Search API

Independent Web Search API with 35B+ Pages

SERP APIs

Brave Search API provides enterprise-grade access to an independent index of 35+ billion web pages with real-time results optimized for AI and RAG pipelines.

Cloudflare R2

S3-compatible storage with zero egress fees

Object Storage

Cloudflare R2 is S3-compatible object storage with zero egress fees, making it ideal for storing and serving frequently accessed scraped data.

AWS Athena

Serverless SQL Queries on Amazon S3

ETL Tools

Amazon Athena is a serverless, interactive query service that lets you analyze petabyte-scale data in S3 using standard SQL with pay-per-query pricing.

AWS Glue

Serverless Data Integration Service by AWS

ETL Tools

AWS Glue is a fully managed, serverless ETL service that discovers, prepares, and integrates data from over 100 sources into lakes and warehouses.

Webshare

Fast and affordable proxy servers

Proxy Services

Webshare provides self-serve proxy servers across many locations, offering datacenter and residential proxy options for scraping workflows.

IPRoyal

Premium proxies at affordable prices

Proxy Services

IPRoyal offers residential, datacenter, ISP, and mobile proxy products with self-service access for scraping workflows.

Smartproxy

AI-ready proxy and scraping solutions

Proxy Services

Smartproxy (now Decodo) offers residential and datacenter proxy products with rotating and sticky sessions for scalable scraping.

Render

Cloud platform for web services and cron jobs

Cloud Compute

Render is a cloud platform that deploys web services, background workers, and cron jobs with simple scaling for scraping workloads.

Railway

Deploy apps from GitHub through configured exporters or custom item pipelines

Cloud Compute

Railway deploys web apps and services from GitHub with auto-scaling, built-in databases, and instant previews for scraping workloads.

AWS Lambda

Serverless compute for event-driven pipelines

Cloud Compute

AWS Lambda runs code without managing servers, ideal for event-driven scraping pipelines triggered by schedules, queues, or API calls.

Find your next web scraping tool.

Discover and compare scraping APIs, no-code tools and open-source libraries.