Web scraping tools directory

Nodriver

Undetected browser automation without Selenium

Stars
4.8K
Forks
341
Last recorded push
11 months ago

Nodriver is the successor to undetected-chromedriver, providing fast CDP-based browser automation with built-in anti-detection and no Selenium dependency.

parse5

HTML5 spec-compliant parser for Node.js

Stars
3.9K
Forks
251
Last recorded push
8 months ago

parse5 is a spec-compliant HTML5 parser for Node.js, parsing HTML exactly as modern browsers do.

ParseHub

Visual scraper with free and paid plans

Web Scraping

Extract data from dynamic websites by clicking elements, handle AJAX, forms, and dropdowns, then export to JSON, Excel, or API.

ValueSERP

Low-Cost Google SERP API for Data Collection

SERP APIs

ValueSERP provides Google search results through an API, with location targeting, batch processing, and JSON, HTML, or CSV output.

Screaming Frog SEO Spider

Desktop SEO Crawler With Sitemap Auditing

Sitemap Scrapers

Desktop SEO crawler that can crawl linked XML sitemaps, download a sitemap in List Mode to crawl only its URLs, and export flat URL lists to CSV.

AutoScraper

Example-Based HTML Extraction For Python

Stars
8K
Forks
764
Last recorded push
last year

Python library that learns simple scraping rules from example values and extracts similar data from HTML pages.

Pyppeteer

Unmaintained Python Port of Puppeteer

Stars
3.9K
Forks
347
Last recorded push
2 years ago

Python port of Puppeteer for Chrome and Chromium automation. Its maintainers mark the repository as unmaintained and suggest Playwright Python.

Chat4Data

Chat-Based Web Data Extraction

AI Web ScrapingWeb Scraping

Chrome-based AI scraping tool that lets users extract page data through conversational instructions and export structured results.

Browse AI

No-code web scraping and change monitoring

AI Web Scraping

Extract and monitor web data through recorded browser workflows, with spreadsheet exports and integrations.

DataForSEO

Comprehensive API Stack for SEO and Marketing Data

Stars
51
Forks
12
Last recorded push
8 months ago

DataForSEO provides SEO and digital marketing data via API, serving as a trusted data partner for 750+ software companies and agencies worldwide.

Apify

Full-stack web scraping and data extraction platform

Stars
180
Forks
21
Last recorded push
8 months ago

Apify is a cloud platform for web scraping, browser automation, and data extraction with an Actor marketplace and developer SDKs.

Scrapy Cluster

Distributed on-demand scraping with Scrapy

Stars
1.2K
Forks
322
Last recorded push
2 years ago

Scrapy Cluster uses Redis and Kafka to create a distributed, on-demand Scrapy crawling cluster for coordinated large-scale web scraping. Its repository is archived.

Patchright

Undetected Playwright fork for bot evasion

Stars
4.8K
Forks
88
Last recorded push
8 months ago

Patchright is a patched fork of Playwright that evades bot detection with a Playwright-style API and documented compatibility limits.

BeeAI Framework

Production-Ready AI Agent Framework

Stars
3.4K
Forks
400
Last recorded push
8 months ago

BeeAI Framework is an open-source platform for building production-ready multi-agent systems in Python and TypeScript with MCP and workflow support.

Heritrix

Web-scale archival web crawler

Stars
3.3K
Forks
782
Last recorded push
9 months ago

Heritrix is the Internet Archive's open-source, extensible, archival-quality web crawler designed for large-scale web preservation and data collection.

Camoufox

Anti-detect browser with fingerprint injection

Stars
12.5K
Forks
459
Last recorded push
8 months ago

Camoufox is an open-source anti-detect browser based on Firefox with robust fingerprint injection to evade bot detection systems.

Scrapy Redis

Redis-based distributed components for Scrapy

Stars
5.6K
Forks
1.6K
Last recorded push
2 years ago

Scrapy-Redis provides Redis-backed components for Scrapy, enabling distributed crawling with shared request queues and item pipelines.

LLM-Scraper

Structured Web Extraction with LLMs

Stars
6.9K
Forks
369
Last recorded push
10 months ago

LLM Scraper is a TypeScript library that extracts structured data from web pages using LLMs with Zod schemas, Playwright, and streaming support.

Rod

High-Level DevTools Driver for Go

Stars
7.1K
Forks
437
Last recorded push
last year

Rod is a Go library for web automation and scraping that drives browsers via the DevTools Protocol with a simple, chainable API.

Crawl4AI

Open-Source LLM-Friendly Web Crawler

Stars
85.3K
Forks
6K
Last recorded push
8 months ago

Crawl4AI is an open-source web crawler and scraper built for LLMs, AI agents, and data pipelines with blazing-fast async crawling and AI-ready output.

Scrapy

Fast, high-level web crawling framework for Python

Stars
64.7K
Forks
11.2K
Last recorded push
8 months ago

Scrapy is an open-source Python framework for building fast, scalable web crawlers that extract structured data from websites efficiently.

Sitebulb

SEO Audits With Sitemap Discovery And Exports

Sitemap Scrapers

Desktop and cloud SEO auditing tool that discovers XML sitemaps, accepts a sitemap index as a crawl source, and exports URLs to CSV or Google Sheets.

Ben Balter sitemap-parser

Simple Sitemaps.org XML Parsing For Ruby

Stars
33
Forks
47
Last recorded push
last year

Ruby parser for sitemaps.org-compliant XML sitemaps; a recurse option flattens nested indexes into one URL list returned by to_a.

TurnerSoftware SitemapTools

Discover And Parse A Domain's Sitemaps In C#

Stars
38
Forks
10
Last recorded push
2 years ago

C# library whose GetAllSitemapsForDomainAsync() takes a domain and discovers plus parses all its sitemaps, with lenient parsing for messy real-world files.

VIPnytt SitemapParser

Recursive Sitemap Parsing For PHP

Stars
78
Forks
33
Last recorded push
last year

PHP library with parse() and parseRecursive() APIs; parseRecursive on a robots.txt URL walks every nested sitemap for a URL list.

Crawler-Commons

Sitemap and Robots.txt Parsing for Java

Stars
259
Forks
91
Last recorded push
3 months ago

Java library with a SAX-based sitemap parser for sitemaps, indexes, extensions, and feeds; you supply the bytes and it parses and normalizes them.

gopher-parse-sitemap

Zero-Dependency Streaming Sitemap Parser In Go

Stars
40
Forks
19
Last recorded push
3 years ago

Streaming Go parser for huge XML sitemaps with minimal memory and no external dependencies, consuming entries on the fly via callbacks.

go-sitemap-parser

Resolve Robots, Indexes, And Sitemaps In Go

Stars
8
Forks
1
Last recorded push
5 months ago

Go library whose Parse() accepts a robots.txt, sitemap index, or plain sitemap URL and resolves the rest, with concurrent fetching and depth limits.

sitemap-stream-parser

Stream Very Large Sitemaps With Low Memory

Stars
39
Forks
19
Last recorded push
5 years ago

Streaming Node.js XML sitemap parser for very large, deeply nested sitemaps, with a callback API and a helper that finds sitemaps in robots.txt.

Sitemapper

Parse Sitemaps And Indexes In Node.js

Stars
137
Forks
83
Last recorded push
4 months ago

Node.js library that parses a sitemap URL including sitemap indexes with many child sitemaps, with concurrency, retries, timeout, and proxy options.

Ultimate Sitemap Parser

Auto-Discover And Parse Sitemaps In Python

Stars
257
Forks
74
Last recorded push
8 months ago

Python library that finds a site's sitemaps from robots.txt and common paths, recurses sitemap indexes, and streams pages with low memory.

Crawlbase

Web data infrastructure for scraping at scale

AI Web Scraping

+2 more

Web data platform for crawling, extracting, proxying, and storing scraped data through APIs, managed crawlers, cloud storage, and MCP integrations.

Requests-HTML

Python Requests With HTML Parsing

Stars
13.8K
Forks
999
Last recorded push
2 years ago

Python library combining HTTP requests, HTML parsing, CSS selectors, and optional JavaScript rendering for scraping workflows.

pyquery

jQuery-Style HTML Parsing For Python

Stars
2.4K
Forks
184
Last recorded push
7 months ago

Python library that applies a jQuery-style API to HTML and XML documents for selection, traversal, and extraction.

Helium

Readable Selenium Automation For Python

Stars
8.3K
Forks
513
Last recorded push
3 months ago

Python browser automation library that wraps Selenium with a higher-level API for clicking, typing, and finding visible elements.

Thunderbit

AI Web Scraping From Browser Pages

AI Web ScrapingWeb Scraping

No-code AI web scraping tool for turning pages, PDFs, and images into structured tables from the browser.

Nodriver

Undetected browser automation without Selenium

Stars
4.8K
Forks
341
Last recorded push
11 months ago

Nodriver is the successor to undetected-chromedriver, providing fast CDP-based browser automation with built-in anti-detection and no Selenium dependency.

parse5

HTML5 spec-compliant parser for Node.js

Stars
3.9K
Forks
251
Last recorded push
8 months ago

parse5 is a spec-compliant HTML5 parser for Node.js, parsing HTML exactly as modern browsers do.

ParseHub

Visual scraper with free and paid plans

Web Scraping

Extract data from dynamic websites by clicking elements, handle AJAX, forms, and dropdowns, then export to JSON, Excel, or API.

ValueSERP

Low-Cost Google SERP API for Data Collection

SERP APIs

ValueSERP provides Google search results through an API, with location targeting, batch processing, and JSON, HTML, or CSV output.

Screaming Frog SEO Spider

Desktop SEO Crawler With Sitemap Auditing

Sitemap Scrapers

Desktop SEO crawler that can crawl linked XML sitemaps, download a sitemap in List Mode to crawl only its URLs, and export flat URL lists to CSV.

AutoScraper

Example-Based HTML Extraction For Python

Stars
8K
Forks
764
Last recorded push
last year

Python library that learns simple scraping rules from example values and extracts similar data from HTML pages.

Pyppeteer

Unmaintained Python Port of Puppeteer

Stars
3.9K
Forks
347
Last recorded push
2 years ago

Python port of Puppeteer for Chrome and Chromium automation. Its maintainers mark the repository as unmaintained and suggest Playwright Python.

Chat4Data

Chat-Based Web Data Extraction

AI Web ScrapingWeb Scraping

Chrome-based AI scraping tool that lets users extract page data through conversational instructions and export structured results.

Browse AI

No-code web scraping and change monitoring

AI Web Scraping

Extract and monitor web data through recorded browser workflows, with spreadsheet exports and integrations.

DataForSEO

Comprehensive API Stack for SEO and Marketing Data

Stars
51
Forks
12
Last recorded push
8 months ago

DataForSEO provides SEO and digital marketing data via API, serving as a trusted data partner for 750+ software companies and agencies worldwide.

Apify

Full-stack web scraping and data extraction platform

Stars
180
Forks
21
Last recorded push
8 months ago

Apify is a cloud platform for web scraping, browser automation, and data extraction with an Actor marketplace and developer SDKs.

Scrapy Cluster

Distributed on-demand scraping with Scrapy

Stars
1.2K
Forks
322
Last recorded push
2 years ago

Scrapy Cluster uses Redis and Kafka to create a distributed, on-demand Scrapy crawling cluster for coordinated large-scale web scraping. Its repository is archived.

Patchright

Undetected Playwright fork for bot evasion

Stars
4.8K
Forks
88
Last recorded push
8 months ago

Patchright is a patched fork of Playwright that evades bot detection with a Playwright-style API and documented compatibility limits.

BeeAI Framework

Production-Ready AI Agent Framework

Stars
3.4K
Forks
400
Last recorded push
8 months ago

BeeAI Framework is an open-source platform for building production-ready multi-agent systems in Python and TypeScript with MCP and workflow support.

Heritrix

Web-scale archival web crawler

Stars
3.3K
Forks
782
Last recorded push
9 months ago

Heritrix is the Internet Archive's open-source, extensible, archival-quality web crawler designed for large-scale web preservation and data collection.

Camoufox

Anti-detect browser with fingerprint injection

Stars
12.5K
Forks
459
Last recorded push
8 months ago

Camoufox is an open-source anti-detect browser based on Firefox with robust fingerprint injection to evade bot detection systems.

Scrapy Redis

Redis-based distributed components for Scrapy

Stars
5.6K
Forks
1.6K
Last recorded push
2 years ago

Scrapy-Redis provides Redis-backed components for Scrapy, enabling distributed crawling with shared request queues and item pipelines.

LLM-Scraper

Structured Web Extraction with LLMs

Stars
6.9K
Forks
369
Last recorded push
10 months ago

LLM Scraper is a TypeScript library that extracts structured data from web pages using LLMs with Zod schemas, Playwright, and streaming support.

Rod

High-Level DevTools Driver for Go

Stars
7.1K
Forks
437
Last recorded push
last year

Rod is a Go library for web automation and scraping that drives browsers via the DevTools Protocol with a simple, chainable API.

Crawl4AI

Open-Source LLM-Friendly Web Crawler

Stars
85.3K
Forks
6K
Last recorded push
8 months ago

Crawl4AI is an open-source web crawler and scraper built for LLMs, AI agents, and data pipelines with blazing-fast async crawling and AI-ready output.

Scrapy

Fast, high-level web crawling framework for Python

Stars
64.7K
Forks
11.2K
Last recorded push
8 months ago

Scrapy is an open-source Python framework for building fast, scalable web crawlers that extract structured data from websites efficiently.

Sitebulb

SEO Audits With Sitemap Discovery And Exports

Sitemap Scrapers

Desktop and cloud SEO auditing tool that discovers XML sitemaps, accepts a sitemap index as a crawl source, and exports URLs to CSV or Google Sheets.

Ben Balter sitemap-parser

Simple Sitemaps.org XML Parsing For Ruby

Stars
33
Forks
47
Last recorded push
last year

Ruby parser for sitemaps.org-compliant XML sitemaps; a recurse option flattens nested indexes into one URL list returned by to_a.

TurnerSoftware SitemapTools

Discover And Parse A Domain's Sitemaps In C#

Stars
38
Forks
10
Last recorded push
2 years ago

C# library whose GetAllSitemapsForDomainAsync() takes a domain and discovers plus parses all its sitemaps, with lenient parsing for messy real-world files.

VIPnytt SitemapParser

Recursive Sitemap Parsing For PHP

Stars
78
Forks
33
Last recorded push
last year

PHP library with parse() and parseRecursive() APIs; parseRecursive on a robots.txt URL walks every nested sitemap for a URL list.

Crawler-Commons

Sitemap and Robots.txt Parsing for Java

Stars
259
Forks
91
Last recorded push
3 months ago

Java library with a SAX-based sitemap parser for sitemaps, indexes, extensions, and feeds; you supply the bytes and it parses and normalizes them.

gopher-parse-sitemap

Zero-Dependency Streaming Sitemap Parser In Go

Stars
40
Forks
19
Last recorded push
3 years ago

Streaming Go parser for huge XML sitemaps with minimal memory and no external dependencies, consuming entries on the fly via callbacks.

go-sitemap-parser

Resolve Robots, Indexes, And Sitemaps In Go

Stars
8
Forks
1
Last recorded push
5 months ago

Go library whose Parse() accepts a robots.txt, sitemap index, or plain sitemap URL and resolves the rest, with concurrent fetching and depth limits.

sitemap-stream-parser

Stream Very Large Sitemaps With Low Memory

Stars
39
Forks
19
Last recorded push
5 years ago

Streaming Node.js XML sitemap parser for very large, deeply nested sitemaps, with a callback API and a helper that finds sitemaps in robots.txt.

Sitemapper

Parse Sitemaps And Indexes In Node.js

Stars
137
Forks
83
Last recorded push
4 months ago

Node.js library that parses a sitemap URL including sitemap indexes with many child sitemaps, with concurrency, retries, timeout, and proxy options.

Ultimate Sitemap Parser

Auto-Discover And Parse Sitemaps In Python

Stars
257
Forks
74
Last recorded push
8 months ago

Python library that finds a site's sitemaps from robots.txt and common paths, recurses sitemap indexes, and streams pages with low memory.

Crawlbase

Web data infrastructure for scraping at scale

AI Web Scraping

+2 more

Web data platform for crawling, extracting, proxying, and storing scraped data through APIs, managed crawlers, cloud storage, and MCP integrations.

Requests-HTML

Python Requests With HTML Parsing

Stars
13.8K
Forks
999
Last recorded push
2 years ago

Python library combining HTTP requests, HTML parsing, CSS selectors, and optional JavaScript rendering for scraping workflows.

pyquery

jQuery-Style HTML Parsing For Python

Stars
2.4K
Forks
184
Last recorded push
7 months ago

Python library that applies a jQuery-style API to HTML and XML documents for selection, traversal, and extraction.

Helium

Readable Selenium Automation For Python

Stars
8.3K
Forks
513
Last recorded push
3 months ago

Python browser automation library that wraps Selenium with a higher-level API for clicking, typing, and finding visible elements.

Thunderbit

AI Web Scraping From Browser Pages

AI Web ScrapingWeb Scraping

No-code AI web scraping tool for turning pages, PDFs, and images into structured tables from the browser.

Find your next web scraping tool.

Discover and compare scraping APIs, no-code tools and open-source libraries.