Undetected browser automation without Selenium
Library or tool · Library / Self-hosted
Stars: 4.8K · Code license: AGPL-3.0 Nodriver is the successor to undetected-chromedriver, providing fast CDP-based browser automation with built-in anti-detection and no Selenium dependency.
Browser Automation +1 more
HTML5 spec-compliant parser for Node.js
Library or tool · Library / Self-hosted
Stars: 3.9K · Code license: MIT parse5 is a spec-compliant HTML5 parser for Node.js, parsing HTML exactly as modern browsers do.
HTML Parsers
Visual scraper with free and paid plans
Platform or service
Extract data from dynamic websites by clicking elements, handle AJAX, forms, and dropdowns, then export to JSON, Excel, or API.
Web Scraping
Low-Cost Google SERP API for Data Collection
Platform or service · API
ValueSERP provides Google search results through an API, with location targeting, batch processing, and JSON, HTML, or CSV output.
SERP Extraction APIs
Desktop SEO Crawler With Sitemap Auditing
Platform or service · Desktop app
Desktop SEO crawler that can crawl linked XML sitemaps, download a sitemap in List Mode to crawl only its URLs, and export flat URL lists to CSV.
Sitemap Scrapers
Example-Based HTML Extraction For Python
Library or tool · Library / Self-hosted
Stars: 8.1K · Code license: MIT Python library that learns simple scraping rules from example values and extracts similar data from HTML pages.
Unmaintained Python Port of Puppeteer
Library or tool · Library / Self-hosted
Python port of Puppeteer for Chrome and Chromium automation. Its maintainers mark the repository as unmaintained and suggest Playwright Python.
Browser Automation +1 more
Chat-Based Web Data Extraction
Platform or service · Browser extension
Chrome-based AI scraping tool that lets users extract page data through conversational instructions and export structured results.
No-code web scraping and change monitoring
Platform or service · SaaS
Extract and monitor web data through recorded browser workflows, with spreadsheet exports and integrations.
AI Web Scraping
Comprehensive API Stack for SEO and Marketing Data
Platform or service · API
DataForSEO provides APIs for search engine results, keyword data, backlinks and other SEO and digital marketing datasets.
SERP Extraction APIs
Full-stack web scraping and data extraction platform
Platform or service · SaaS / API
Apify is a cloud platform for web scraping, browser automation, and data extraction with an Actor marketplace and developer SDKs.
Scraping APIs
Distributed on-demand scraping with Scrapy
Library or tool · Self-hosted
Scrapy Cluster uses Redis and Kafka to create a distributed, on-demand Scrapy crawling cluster for coordinated large-scale web scraping. Its repository is archived.
Large-Scale & Distributed Crawling
Undetected Playwright fork for bot evasion
Library or tool · Self-hosted
Patchright is a patched fork of Playwright that evades bot detection with a Playwright-style API and documented compatibility limits.
Browser Automation +1 more
AI agents and multi-agent workflows
Library or tool · Framework / Self-hosted
Stars: 3.4K · Code license: Apache-2.0 Python and TypeScript framework for building AI agents and multi-agent workflows with model providers, memory, tools and MCP integrations.
AI Agent Tooling
Web-scale archival web crawler
Library or tool · Self-hosted
Heritrix is the Internet Archive's open-source, extensible, archival-quality web crawler designed for large-scale web preservation and data collection.
Large-Scale & Distributed Crawling
Anti-detect browser with fingerprint injection
Library or tool · Self-hosted
Stars: 12.6K · Code license: MPL-2.0 Camoufox is an open-source anti-detect browser based on Firefox with robust fingerprint injection to evade bot detection systems.
Browser Automation +1 more
Redis-based distributed components for Scrapy
Library or tool · Self-hosted
Scrapy-Redis provides Redis-backed components for Scrapy, enabling distributed crawling with shared request queues and item pipelines.
Large-Scale & Distributed Crawling
Structured Web Extraction with LLMs
Library or tool · Library / Self-hosted
Stars: 6.9K · Code license: MIT LLM Scraper is a TypeScript library that extracts structured data from web pages using LLMs with Zod schemas, Playwright, and streaming support.
High-Level DevTools Driver for Go
Library or tool · Self-hosted
Rod is a Go library for web automation and scraping that drives browsers via the DevTools Protocol with a simple, chainable API.
Browser Automation
Open-Source LLM-Friendly Web Crawler
Library or tool · Framework / Self-hosted
Stars: 85.6K · Code license: Apache-2.0 Crawl4AI is an open-source Python crawler with asynchronous crawling, Markdown output and structured extraction for LLM and data workflows.
Fast, high-level web crawling framework for Python
Library or tool · Framework / Self-hosted
Stars: 64.8K · Code license: BSD-3-Clause Scrapy is an open-source Python framework for building fast, scalable web crawlers that extract structured data from websites efficiently.
Scraping Frameworks
SEO Audits With Sitemap Discovery And Exports
Platform or service · Desktop app
Desktop and cloud SEO auditing tool that discovers XML sitemaps, accepts a sitemap index as a crawl source, and exports URLs to CSV or Google Sheets.
Sitemap Scrapers
Simple Sitemaps.org XML Parsing For Ruby
Library or tool · Library / Self-hosted
Stars: 33 · Code license: MIT Ruby parser for sitemaps.org-compliant XML sitemaps; a recurse option flattens nested indexes into one URL list returned by to_a.
Sitemap Scrapers
Discover And Parse A Domain's Sitemaps In C#
Library or tool · Library / Self-hosted
C# library whose GetAllSitemapsForDomainAsync() takes a domain and discovers plus parses all its sitemaps, with lenient parsing for messy real-world files.
Sitemap Scrapers
Recursive Sitemap Parsing For PHP
Library or tool · Library / Self-hosted
Stars: 78 · Code license: MIT PHP library with parse() and parseRecursive() APIs; parseRecursive on a robots.txt URL walks every nested sitemap for a URL list.
Sitemap Scrapers
Sitemap and Robots.txt Parsing for Java
Library or tool · Library / Self-hosted
Stars: 259 · Code license: Apache-2.0 Java library with a SAX-based sitemap parser for sitemaps, indexes, extensions, and feeds; you supply the bytes and it parses and normalizes them.
Sitemap Scrapers
Zero-Dependency Streaming Sitemap Parser In Go
Library or tool · Library / Self-hosted
Streaming Go parser for huge XML sitemaps with minimal memory and no external dependencies, consuming entries on the fly via callbacks.
Sitemap Scrapers
Resolve Robots, Indexes, And Sitemaps In Go
Library or tool · Library / Self-hosted
Stars: 8 · Code license: MIT Go library whose Parse() accepts a robots.txt, sitemap index, or plain sitemap URL and resolves the rest, with concurrent fetching and depth limits.
Sitemap Scrapers
Stream Very Large Sitemaps With Low Memory
Library or tool · Library / Self-hosted
Streaming Node.js XML sitemap parser for very large, deeply nested sitemaps, with a callback API and a helper that finds sitemaps in robots.txt.
Sitemap Scrapers
Parse Sitemaps And Indexes In Node.js
Library or tool · Library / Self-hosted
Stars: 137 · Code license: MIT Node.js library that parses a sitemap URL including sitemap indexes with many child sitemaps, with concurrency, retries, timeout, and proxy options.
Sitemap Scrapers
Auto-Discover And Parse Sitemaps In Python
Library or tool · Library / Self-hosted
Stars: 257 · Code license: GPL-3.0-or-later Python library that finds a site's sitemaps from robots.txt and common paths, recurses sitemap indexes, and streams pages with low memory.
Sitemap Scrapers
Web data infrastructure for scraping at scale
Platform or service · API
Web data platform for crawling, extracting, proxying, and storing scraped data through APIs, managed crawlers, cloud storage, and MCP integrations.
Python Requests With HTML Parsing
Library or tool · Self-hosted
Python library combining HTTP requests, HTML parsing, CSS selectors, and optional JavaScript rendering for scraping workflows.
Browser Automation +2 more
jQuery-Style HTML Parsing For Python
Library or tool · Library / Self-hosted
Python library that applies a jQuery-style API to HTML and XML documents for selection, traversal, and extraction.
HTML Parsers
Readable Selenium Automation For Python
Library or tool · Self-hosted
Python browser automation library that wraps Selenium with a higher-level API for clicking, typing, and finding visible elements.
Browser Automation +1 more
AI Web Scraping From Browser Pages
Platform or service · Browser extension / SaaS / API / CLI
No-code AI web scraping tool for turning pages, PDFs, and images into structured tables from the browser.
