All tools

Every tool in the directory - scraping, anti-blocking, data processing, databases, infrastructure, and utilities.

Sitebulb

SEO Audits With Sitemap Discovery And Exports

Platform or service · Desktop app

Desktop and cloud SEO auditing tool that discovers XML sitemaps, accepts a sitemap index as a crawl source, and exports URLs to CSV or Google Sheets.

Sitemap Scrapers

Screaming Frog SEO Spider

Desktop SEO Crawler With Sitemap Auditing

Platform or service · Desktop app

Desktop SEO crawler that can crawl linked XML sitemaps, download a sitemap in List Mode to crawl only its URLs, and export flat URL lists to CSV.

Sitemap Scrapers

Ben Balter sitemap-parser

Simple Sitemaps.org XML Parsing For Ruby

Library or tool · Library / Self-hosted

Stars: 33
Code license: MIT

Ruby parser for sitemaps.org-compliant XML sitemaps; a recurse option flattens nested indexes into one URL list returned by to_a.

Sitemap Scrapers

TurnerSoftware SitemapTools

Discover And Parse A Domain's Sitemaps In C#

Library or tool · Library / Self-hosted

C# library whose GetAllSitemapsForDomainAsync() takes a domain and discovers plus parses all its sitemaps, with lenient parsing for messy real-world files.

Sitemap Scrapers

VIPnytt SitemapParser

Recursive Sitemap Parsing For PHP

Library or tool · Library / Self-hosted

Stars: 78
Code license: MIT

PHP library with parse() and parseRecursive() APIs; parseRecursive on a robots.txt URL walks every nested sitemap for a URL list.

Sitemap Scrapers

Crawler-Commons

Sitemap and Robots.txt Parsing for Java

Library or tool · Library / Self-hosted

Stars: 259
Code license: Apache-2.0

Java library with a SAX-based sitemap parser for sitemaps, indexes, extensions, and feeds; you supply the bytes and it parses and normalizes them.

Sitemap Scrapers

gopher-parse-sitemap

Zero-Dependency Streaming Sitemap Parser In Go

Library or tool · Library / Self-hosted

Streaming Go parser for huge XML sitemaps with minimal memory and no external dependencies, consuming entries on the fly via callbacks.

Sitemap Scrapers

go-sitemap-parser

Resolve Robots, Indexes, And Sitemaps In Go

Library or tool · Library / Self-hosted

Stars: 8
Code license: MIT

Go library whose Parse() accepts a robots.txt, sitemap index, or plain sitemap URL and resolves the rest, with concurrent fetching and depth limits.

Sitemap Scrapers

sitemap-stream-parser

Stream Very Large Sitemaps With Low Memory

Library or tool · Library / Self-hosted

Streaming Node.js XML sitemap parser for very large, deeply nested sitemaps, with a callback API and a helper that finds sitemaps in robots.txt.

Sitemap Scrapers

Sitemapper

Parse Sitemaps And Indexes In Node.js

Library or tool · Library / Self-hosted

Stars: 137
Code license: MIT

Node.js library that parses a sitemap URL including sitemap indexes with many child sitemaps, with concurrency, retries, timeout, and proxy options.

Sitemap Scrapers

Ultimate Sitemap Parser

Auto-Discover And Parse Sitemaps In Python

Library or tool · Library / Self-hosted

Stars: 257
Code license: GPL-3.0-or-later

Python library that finds a site's sitemaps from robots.txt and common paths, recurses sitemap indexes, and streams pages with low memory.

Sitemap Scrapers

Crawlbase

Web data infrastructure for scraping at scale

Platform or service · API

Web data platform for crawling, extracting, proxying, and storing scraped data through APIs, managed crawlers, cloud storage, and MCP integrations.

AI Web Scraping

+2 more

Requests-HTML

Python Requests With HTML Parsing

Library or tool · Self-hosted

Python library combining HTTP requests, HTML parsing, CSS selectors, and optional JavaScript rendering for scraping workflows.

Browser Automation

+2 more

pyquery

jQuery-Style HTML Parsing For Python

Library or tool · Library / Self-hosted

Python library that applies a jQuery-style API to HTML and XML documents for selection, traversal, and extraction.

HTML Parsers

Helium

Readable Selenium Automation For Python

Library or tool · Self-hosted

Python browser automation library that wraps Selenium with a higher-level API for clicking, typing, and finding visible elements.

Browser Automation

+1 more

AutoScraper

Example-Based HTML Extraction For Python

Library or tool · Library / Self-hosted

Stars: 8.1K
Code license: MIT

Python library that learns simple scraping rules from example values and extracts similar data from HTML pages.

HTML Parsers

+1 more

Pyppeteer

Unmaintained Python Port of Puppeteer

Library or tool · Library / Self-hosted

Python port of Puppeteer for Chrome and Chromium automation. Its maintainers mark the repository as unmaintained and suggest Playwright Python.

Browser Automation

+1 more

Chat4Data

Chat-Based Web Data Extraction

Platform or service · Browser extension

Chrome-based AI scraping tool that lets users extract page data through conversational instructions and export structured results.

AI Web Scraping

+1 more

Thunderbit

AI Web Scraping From Browser Pages

Platform or service · Browser extension / SaaS / API / CLI

No-code AI web scraping tool for turning pages, PDFs, and images into structured tables from the browser.

AI Web Scraping

+1 more

Kadoa

Self-Healing AI Web Data Pipelines

Platform or service · SaaS

AI web scraping platform for building self-healing extraction agents that monitor pages and collect structured data at scale.

AI Web Scraping

+1 more

pgvector

Add Vector Search To PostgreSQL

Infrastructure · Self-hosted

Open-source PostgreSQL extension for vector similarity search that keeps embeddings and scraped data in the same database.

Vector Databases

Apache StormCrawler

Scalable Java Crawlers For Teams

Library or tool · Self-hosted

Build scalable Java crawlers on Apache Storm with low-latency processing, flexible customization, and an Apache-2.0 license.

Large-Scale & Distributed Crawling

got-scraping

Browser-Like Requests For Reliable Scraping

Library or tool · Library / Self-hosted

Stars: 776
Code license: Apache-2.0

A Node.js HTTP client for scraping that sends browser-like requests with realistic headers and TLS behavior.

Scraping Frameworks

Parsel

Extract Web Data Cleanly With Python

Library or tool · Library / Self-hosted

Stars: 1.4K
Code license: BSD-3-Clause

A Python library for extracting data from HTML, XML, and JSON with CSS, XPath, JMESPath, and regex.

HTML Parsers

Apache Solr

Scalable Search For Scraped Content

Infrastructure · Self-hosted

Open-source Java search software for indexing scraped content with full-text search and faceted navigation.

Search Engines

jsoup

Parse Real-World HTML In Java

Library or tool · Library / Self-hosted

Stars: 11.4K
Code license: MIT

A Java library for parsing real-world HTML into a standards-based DOM with CSS selectors and DOM traversal.

HTML Parsers

selectolax

Fast HTML Parsing for Python Workflows

Library or tool · Library / Self-hosted

Stars: 1.7K
Code license: MIT

Parse HTML with CSS selectors in Python using a fast, MIT-licensed library built for large-scale extraction.

HTML Parsers

SeleniumBase

Reliable Browser Automation For Python

Library or tool · Self-hosted

An open-source Python framework for browser automation, testing, and practical anti-detection scraping with UC Mode and CDP Mode.

Browser Automation

puppeteer-extra

Stealth Plugins For Puppeteer Scraping

Library or tool · Self-hosted

A JavaScript plugin framework for Puppeteer that adds stealth options to make headless Chrome harder to detect.

Fingerprint & Stealth Tooling

curl_cffi

Browser-Like Requests Without Full Browsers

Library or tool · Self-hosted

Python HTTP client for scraping with browser-like TLS, JA3, and HTTP/2 fingerprint impersonation without running a full browser.

Fingerprint & Stealth Tooling

Unstructured

Clean Documents For AI Pipelines

Library or tool · Self-hosted

An open-source Python document ETL tool that turns PDFs, HTML, images, and more into structured elements for RAG and LLM workflows.

Document Extraction

Diffbot

Turn Web Pages Into Usable Data

Platform or service · API

An AI extraction platform that turns web pages into structured data and a knowledge graph for web data workflows.

AI Web Scraping

lxml

Fast XML And HTML Parsing For Python

Library or tool · Library / Self-hosted

Stars: 3.1K
Code license: BSD-3-Clause

A fast Python library for parsing XML and HTML, often used as a foundation for scraping and data tools.

HTML Parsers

Apache Tika

Extract Text From Any Document

Library or tool · Self-hosted

Detects file types and extracts text and document details from 1,000+ formats through one Java interface.

Document Extraction

Docling

Turn Documents Into AI-Ready Data

Library or tool · Self-hosted

Open-source Python software that converts PDFs, DOCX, PPTX, and images into structured data for GenAI and RAG pipelines.

Document Extraction

Scrapfly

AI-Ready Web Data Extraction

Platform or service · API

A commercial scraping platform that helps teams collect structured web data, automate cloud browsers, crawl sites, handle anti-bot blocks, and use proxies.

Scraping APIs

Sitebulb

SEO Audits With Sitemap Discovery And Exports

Platform or service · Desktop app

Desktop and cloud SEO auditing tool that discovers XML sitemaps, accepts a sitemap index as a crawl source, and exports URLs to CSV or Google Sheets.

Sitemap Scrapers

Screaming Frog SEO Spider

Desktop SEO Crawler With Sitemap Auditing

Platform or service · Desktop app

Desktop SEO crawler that can crawl linked XML sitemaps, download a sitemap in List Mode to crawl only its URLs, and export flat URL lists to CSV.

Sitemap Scrapers

Ben Balter sitemap-parser

Simple Sitemaps.org XML Parsing For Ruby

Library or tool · Library / Self-hosted

Stars: 33
Code license: MIT

Ruby parser for sitemaps.org-compliant XML sitemaps; a recurse option flattens nested indexes into one URL list returned by to_a.

Sitemap Scrapers

TurnerSoftware SitemapTools

Discover And Parse A Domain's Sitemaps In C#

Library or tool · Library / Self-hosted

C# library whose GetAllSitemapsForDomainAsync() takes a domain and discovers plus parses all its sitemaps, with lenient parsing for messy real-world files.

Sitemap Scrapers

VIPnytt SitemapParser

Recursive Sitemap Parsing For PHP

Library or tool · Library / Self-hosted

Stars: 78
Code license: MIT

PHP library with parse() and parseRecursive() APIs; parseRecursive on a robots.txt URL walks every nested sitemap for a URL list.

Sitemap Scrapers

Crawler-Commons

Sitemap and Robots.txt Parsing for Java

Library or tool · Library / Self-hosted

Stars: 259
Code license: Apache-2.0

Java library with a SAX-based sitemap parser for sitemaps, indexes, extensions, and feeds; you supply the bytes and it parses and normalizes them.

Sitemap Scrapers

gopher-parse-sitemap

Zero-Dependency Streaming Sitemap Parser In Go

Library or tool · Library / Self-hosted

Streaming Go parser for huge XML sitemaps with minimal memory and no external dependencies, consuming entries on the fly via callbacks.

Sitemap Scrapers

go-sitemap-parser

Resolve Robots, Indexes, And Sitemaps In Go

Library or tool · Library / Self-hosted

Stars: 8
Code license: MIT

Go library whose Parse() accepts a robots.txt, sitemap index, or plain sitemap URL and resolves the rest, with concurrent fetching and depth limits.

Sitemap Scrapers

sitemap-stream-parser

Stream Very Large Sitemaps With Low Memory

Library or tool · Library / Self-hosted

Streaming Node.js XML sitemap parser for very large, deeply nested sitemaps, with a callback API and a helper that finds sitemaps in robots.txt.

Sitemap Scrapers

Sitemapper

Parse Sitemaps And Indexes In Node.js

Library or tool · Library / Self-hosted

Stars: 137
Code license: MIT

Node.js library that parses a sitemap URL including sitemap indexes with many child sitemaps, with concurrency, retries, timeout, and proxy options.

Sitemap Scrapers

Ultimate Sitemap Parser

Auto-Discover And Parse Sitemaps In Python

Library or tool · Library / Self-hosted

Stars: 257
Code license: GPL-3.0-or-later

Python library that finds a site's sitemaps from robots.txt and common paths, recurses sitemap indexes, and streams pages with low memory.

Sitemap Scrapers

Crawlbase

Web data infrastructure for scraping at scale

Platform or service · API

Web data platform for crawling, extracting, proxying, and storing scraped data through APIs, managed crawlers, cloud storage, and MCP integrations.

AI Web Scraping

+2 more

Requests-HTML

Python Requests With HTML Parsing

Library or tool · Self-hosted

Python library combining HTTP requests, HTML parsing, CSS selectors, and optional JavaScript rendering for scraping workflows.

Browser Automation

+2 more

pyquery

jQuery-Style HTML Parsing For Python

Library or tool · Library / Self-hosted

Python library that applies a jQuery-style API to HTML and XML documents for selection, traversal, and extraction.

HTML Parsers

Helium

Readable Selenium Automation For Python

Library or tool · Self-hosted

Python browser automation library that wraps Selenium with a higher-level API for clicking, typing, and finding visible elements.

Browser Automation

+1 more

AutoScraper

Example-Based HTML Extraction For Python

Library or tool · Library / Self-hosted

Stars: 8.1K
Code license: MIT

Python library that learns simple scraping rules from example values and extracts similar data from HTML pages.

HTML Parsers

+1 more

Pyppeteer

Unmaintained Python Port of Puppeteer

Library or tool · Library / Self-hosted

Python port of Puppeteer for Chrome and Chromium automation. Its maintainers mark the repository as unmaintained and suggest Playwright Python.

Browser Automation

+1 more

Chat4Data

Chat-Based Web Data Extraction

Platform or service · Browser extension

Chrome-based AI scraping tool that lets users extract page data through conversational instructions and export structured results.

AI Web Scraping

+1 more

Thunderbit

AI Web Scraping From Browser Pages

Platform or service · Browser extension / SaaS / API / CLI

No-code AI web scraping tool for turning pages, PDFs, and images into structured tables from the browser.

AI Web Scraping

+1 more

Kadoa

Self-Healing AI Web Data Pipelines

Platform or service · SaaS

AI web scraping platform for building self-healing extraction agents that monitor pages and collect structured data at scale.

AI Web Scraping

+1 more

pgvector

Add Vector Search To PostgreSQL

Infrastructure · Self-hosted

Open-source PostgreSQL extension for vector similarity search that keeps embeddings and scraped data in the same database.

Vector Databases

Apache StormCrawler

Scalable Java Crawlers For Teams

Library or tool · Self-hosted

Build scalable Java crawlers on Apache Storm with low-latency processing, flexible customization, and an Apache-2.0 license.

Large-Scale & Distributed Crawling

got-scraping

Browser-Like Requests For Reliable Scraping

Library or tool · Library / Self-hosted

Stars: 776
Code license: Apache-2.0

A Node.js HTTP client for scraping that sends browser-like requests with realistic headers and TLS behavior.

Scraping Frameworks

Parsel

Extract Web Data Cleanly With Python

Library or tool · Library / Self-hosted

Stars: 1.4K
Code license: BSD-3-Clause

A Python library for extracting data from HTML, XML, and JSON with CSS, XPath, JMESPath, and regex.

HTML Parsers

Apache Solr

Scalable Search For Scraped Content

Infrastructure · Self-hosted

Open-source Java search software for indexing scraped content with full-text search and faceted navigation.

Search Engines

jsoup

Parse Real-World HTML In Java

Library or tool · Library / Self-hosted

Stars: 11.4K
Code license: MIT

A Java library for parsing real-world HTML into a standards-based DOM with CSS selectors and DOM traversal.

HTML Parsers

selectolax

Fast HTML Parsing for Python Workflows

Library or tool · Library / Self-hosted

Stars: 1.7K
Code license: MIT

Parse HTML with CSS selectors in Python using a fast, MIT-licensed library built for large-scale extraction.

HTML Parsers

SeleniumBase

Reliable Browser Automation For Python

Library or tool · Self-hosted

An open-source Python framework for browser automation, testing, and practical anti-detection scraping with UC Mode and CDP Mode.

Browser Automation

puppeteer-extra

Stealth Plugins For Puppeteer Scraping

Library or tool · Self-hosted

A JavaScript plugin framework for Puppeteer that adds stealth options to make headless Chrome harder to detect.

Fingerprint & Stealth Tooling

curl_cffi

Browser-Like Requests Without Full Browsers

Library or tool · Self-hosted

Python HTTP client for scraping with browser-like TLS, JA3, and HTTP/2 fingerprint impersonation without running a full browser.

Fingerprint & Stealth Tooling

Unstructured

Clean Documents For AI Pipelines

Library or tool · Self-hosted

An open-source Python document ETL tool that turns PDFs, HTML, images, and more into structured elements for RAG and LLM workflows.

Document Extraction

Diffbot

Turn Web Pages Into Usable Data

Platform or service · API

An AI extraction platform that turns web pages into structured data and a knowledge graph for web data workflows.

AI Web Scraping

lxml

Fast XML And HTML Parsing For Python

Library or tool · Library / Self-hosted

Stars: 3.1K
Code license: BSD-3-Clause

A fast Python library for parsing XML and HTML, often used as a foundation for scraping and data tools.

HTML Parsers

Apache Tika

Extract Text From Any Document

Library or tool · Self-hosted

Detects file types and extracts text and document details from 1,000+ formats through one Java interface.

Document Extraction

Docling

Turn Documents Into AI-Ready Data

Library or tool · Self-hosted

Open-source Python software that converts PDFs, DOCX, PPTX, and images into structured data for GenAI and RAG pipelines.

Document Extraction

Scrapfly

AI-Ready Web Data Extraction

Platform or service · API

A commercial scraping platform that helps teams collect structured web data, automate cloud browsers, crawl sites, handle anti-bot blocks, and use proxies.

Scraping APIs