All tools

Sitebulb

SEO Audits With Sitemap Discovery And Exports

Sitemap Scrapers

Desktop and cloud SEO auditing tool that discovers XML sitemaps, accepts a sitemap index as a crawl source, and exports URLs to CSV or Google Sheets.

Screaming Frog SEO Spider

Desktop SEO Crawler With Sitemap Auditing

Sitemap Scrapers

Desktop SEO crawler that can crawl linked XML sitemaps, download a sitemap in List Mode to crawl only its URLs, and export flat URL lists to CSV.

Ben Balter sitemap-parser

Simple Sitemaps.org XML Parsing For Ruby

Stars
33
Forks
47
Last commit
last year

Ruby parser for sitemaps.org-compliant XML sitemaps; a recurse option flattens nested indexes into one URL list returned by to_a.

TurnerSoftware SitemapTools

Discover And Parse A Domain's Sitemaps In C#

Stars
38
Forks
10
Last commit
2 years ago

C# library whose GetAllSitemapsForDomainAsync() takes a domain and discovers plus parses all its sitemaps, with lenient parsing for messy real-world files.

VIPnytt SitemapParser

Recursive Sitemap Parsing For PHP

Stars
78
Forks
33
Last commit
10 months ago

PHP library with parse() and parseRecursive() APIs; parseRecursive on a robots.txt URL walks every nested sitemap for a complete URL list.

Crawler-Commons

The De Facto Java Sitemap Parsing Library

Stars
259
Forks
91
Last commit
2 months ago

Java library with a SAX-based sitemap parser for sitemaps, indexes, extensions, and feeds; you supply the bytes and it parses and normalizes them.

gopher-parse-sitemap

Zero-Dependency Streaming Sitemap Parser In Go

Stars
40
Forks
19
Last commit
3 years ago

Streaming Go parser for huge XML sitemaps with minimal memory and no external dependencies, consuming entries on the fly via callbacks.

go-sitemap-parser

Resolve Robots, Indexes, And Sitemaps In Go

Stars
6
Forks
1
Last commit
3 months ago

Go library whose Parse() accepts a robots.txt, sitemap index, or plain sitemap URL and resolves the rest, with concurrent fetching and depth limits.

sitemap-stream-parser

Stream Very Large Sitemaps With Low Memory

Stars
39
Forks
19
Last commit
5 years ago

Streaming Node.js XML sitemap parser for very large, deeply nested sitemaps, with a callback API and a helper that finds sitemaps in robots.txt.

Sitemapper

Parse Sitemaps And Indexes In Node.js

Stars
135
Forks
83
Last commit
3 months ago

Node.js library that parses a sitemap URL including sitemap indexes with many child sitemaps, with concurrency, retries, timeout, and proxy options.

Ultimate Sitemap Parser

Auto-Discover And Parse Sitemaps In Python

Stars
242
Forks
74
Last commit
7 months ago

Python library that finds a site's sitemaps from robots.txt and common paths, recurses sitemap indexes, and streams pages with low memory.

Crawlbase

Web data infrastructure for scraping at scale

AI Web Scraping

+2 more

Web data platform for crawling, extracting, proxying, and storing scraped data through APIs, managed crawlers, cloud storage, and MCP integrations.

Requests-HTML

Python Requests With HTML Parsing

Stars
13.8K
Forks
999
Last commit
2 years ago

Python library combining HTTP requests, HTML parsing, CSS selectors, and optional JavaScript rendering for scraping workflows.

pyquery

jQuery-Style HTML Parsing For Python

Stars
2.4K
Forks
184
Last commit
6 months ago

Python library that applies a jQuery-style API to HTML and XML documents for selection, traversal, and extraction.

Helium

Readable Selenium Automation For Python

Stars
8.3K
Forks
513
Last commit
2 months ago

Python browser automation library that wraps Selenium with a higher-level API for clicking, typing, and finding visible elements.

AutoScraper

Example-Based HTML Extraction For Python

Stars
7.5K
Forks
764
Last commit
last year

Python library that learns simple scraping rules from example values and extracts similar data from HTML pages.

Pyppeteer

Python Control For Headless Chromium

Stars
4K
Forks
347
Last commit
2 years ago

Python library for controlling Chrome and Chromium through a Puppeteer-like API for screenshots, automation, and scraping.

Chat4Data

Chat-Based Web Data Extraction

AI Web ScrapingWeb Scraping

Chrome-based AI scraping tool that lets users extract page data through conversational instructions and export structured results.

Thunderbit

AI Web Scraping From Browser Pages

AI Web ScrapingWeb Scraping

No-code AI web scraping tool for turning pages, PDFs, and images into structured tables from the browser.

Kadoa

Self-Healing AI Web Data Pipelines

AI Web ScrapingWeb Scraping

AI web scraping platform for building self-healing extraction agents that monitor pages and collect structured data at scale.

pgvector

Add Vector Search To PostgreSQL

Stars
21.9K
Forks
1.2K
Last commit
2 months ago

Open-source PostgreSQL extension for vector similarity search that keeps embeddings and scraped data in the same database.

Apache StormCrawler

Scalable Java Crawlers For Teams

Stars
981
Forks
282
Last commit
2 months ago

Build scalable Java crawlers on Apache Storm with low-latency processing, flexible customization, and an Apache-2.0 license.

got-scraping

Browser-Like Requests For Reliable Scraping

Stars
760
Forks
60
Last commit
3 months ago

A Node.js HTTP client for scraping that sends browser-like requests with realistic headers and TLS behavior.

Parsel

Extract Web Data Cleanly With Python

Stars
1.3K
Forks
162
Last commit
2 months ago

A Python library for extracting data from HTML, XML, and JSON with CSS, XPath, JMESPath, and regex.

Apache Solr

Scalable Search For Scraped Content

Stars
1.6K
Forks
839
Last commit
2 months ago

Open-source Java search software for indexing scraped content with full-text search and faceted navigation.

jsoup

Parse Real-World HTML In Java

Stars
11.4K
Forks
2.3K
Last commit
2 months ago

A Java library for parsing real-world HTML into a standards-based DOM with CSS selectors and DOM traversal.

selectolax

Fast HTML Parsing for Python Workflows

Stars
1.6K
Forks
91
Last commit
3 months ago

Parse HTML with CSS selectors in Python using a fast, MIT-licensed library built for large-scale extraction.

SeleniumBase

Reliable Browser Automation For Python

Stars
12.8K
Forks
1.6K
Last commit
2 months ago

An open-source Python framework for browser automation, testing, and practical anti-detection scraping with UC Mode and CDP Mode.

puppeteer-extra

Stealth Plugins For Puppeteer Scraping

Stars
7.4K
Forks
781
Last commit
2 years ago

A JavaScript plugin framework for Puppeteer that adds stealth options to make headless Chrome harder to detect.

curl_cffi

Browser-Like Requests Without Full Browsers

Stars
5.9K
Forks
501
Last commit
2 months ago

Python HTTP client for scraping with browser-like TLS, JA3, and HTTP/2 fingerprint impersonation without running a full browser.

Unstructured

Clean Documents For AI Pipelines

Stars
15K
Forks
1.3K
Last commit
2 months ago

An open-source Python document ETL tool that turns PDFs, HTML, images, and more into structured elements for RAG and LLM workflows.

Diffbot

Turn Web Pages Into Usable Data

AI Web Scraping

An AI extraction platform that turns web pages into structured data and a knowledge graph for web data workflows.

lxml

Fast XML And HTML Parsing For Python

Stars
3K
Forks
619
Last commit
2 months ago

A fast Python library for parsing XML and HTML, often used as a foundation for scraping and data tools.

Apache Tika

Extract Text From Any Document

Stars
3.8K
Forks
941
Last commit
2 months ago

Detects file types and extracts text and document details from 1,000+ formats through one Java interface.

Docling

Turn Documents Into AI-Ready Data

Stars
62.2K
Forks
4.4K
Last commit
2 months ago

Open-source Python software that converts PDFs, DOCX, PPTX, and images into structured data for GenAI and RAG pipelines.

Scrapfly

AI-Ready Web Data Extraction

Scraping APIs

A commercial scraping platform that helps teams collect structured web data, automate cloud browsers, crawl sites, handle anti-bot blocks, and use proxies.

Sitebulb

SEO Audits With Sitemap Discovery And Exports

Sitemap Scrapers

Desktop and cloud SEO auditing tool that discovers XML sitemaps, accepts a sitemap index as a crawl source, and exports URLs to CSV or Google Sheets.

Screaming Frog SEO Spider

Desktop SEO Crawler With Sitemap Auditing

Sitemap Scrapers

Desktop SEO crawler that can crawl linked XML sitemaps, download a sitemap in List Mode to crawl only its URLs, and export flat URL lists to CSV.

Ben Balter sitemap-parser

Simple Sitemaps.org XML Parsing For Ruby

Stars
33
Forks
47
Last commit
last year

Ruby parser for sitemaps.org-compliant XML sitemaps; a recurse option flattens nested indexes into one URL list returned by to_a.

TurnerSoftware SitemapTools

Discover And Parse A Domain's Sitemaps In C#

Stars
38
Forks
10
Last commit
2 years ago

C# library whose GetAllSitemapsForDomainAsync() takes a domain and discovers plus parses all its sitemaps, with lenient parsing for messy real-world files.

VIPnytt SitemapParser

Recursive Sitemap Parsing For PHP

Stars
78
Forks
33
Last commit
10 months ago

PHP library with parse() and parseRecursive() APIs; parseRecursive on a robots.txt URL walks every nested sitemap for a complete URL list.

Crawler-Commons

The De Facto Java Sitemap Parsing Library

Stars
259
Forks
91
Last commit
2 months ago

Java library with a SAX-based sitemap parser for sitemaps, indexes, extensions, and feeds; you supply the bytes and it parses and normalizes them.

gopher-parse-sitemap

Zero-Dependency Streaming Sitemap Parser In Go

Stars
40
Forks
19
Last commit
3 years ago

Streaming Go parser for huge XML sitemaps with minimal memory and no external dependencies, consuming entries on the fly via callbacks.

go-sitemap-parser

Resolve Robots, Indexes, And Sitemaps In Go

Stars
6
Forks
1
Last commit
3 months ago

Go library whose Parse() accepts a robots.txt, sitemap index, or plain sitemap URL and resolves the rest, with concurrent fetching and depth limits.

sitemap-stream-parser

Stream Very Large Sitemaps With Low Memory

Stars
39
Forks
19
Last commit
5 years ago

Streaming Node.js XML sitemap parser for very large, deeply nested sitemaps, with a callback API and a helper that finds sitemaps in robots.txt.

Sitemapper

Parse Sitemaps And Indexes In Node.js

Stars
135
Forks
83
Last commit
3 months ago

Node.js library that parses a sitemap URL including sitemap indexes with many child sitemaps, with concurrency, retries, timeout, and proxy options.

Ultimate Sitemap Parser

Auto-Discover And Parse Sitemaps In Python

Stars
242
Forks
74
Last commit
7 months ago

Python library that finds a site's sitemaps from robots.txt and common paths, recurses sitemap indexes, and streams pages with low memory.

Crawlbase

Web data infrastructure for scraping at scale

AI Web Scraping

+2 more

Web data platform for crawling, extracting, proxying, and storing scraped data through APIs, managed crawlers, cloud storage, and MCP integrations.

Requests-HTML

Python Requests With HTML Parsing

Stars
13.8K
Forks
999
Last commit
2 years ago

Python library combining HTTP requests, HTML parsing, CSS selectors, and optional JavaScript rendering for scraping workflows.

pyquery

jQuery-Style HTML Parsing For Python

Stars
2.4K
Forks
184
Last commit
6 months ago

Python library that applies a jQuery-style API to HTML and XML documents for selection, traversal, and extraction.

Helium

Readable Selenium Automation For Python

Stars
8.3K
Forks
513
Last commit
2 months ago

Python browser automation library that wraps Selenium with a higher-level API for clicking, typing, and finding visible elements.

AutoScraper

Example-Based HTML Extraction For Python

Stars
7.5K
Forks
764
Last commit
last year

Python library that learns simple scraping rules from example values and extracts similar data from HTML pages.

Pyppeteer

Python Control For Headless Chromium

Stars
4K
Forks
347
Last commit
2 years ago

Python library for controlling Chrome and Chromium through a Puppeteer-like API for screenshots, automation, and scraping.

Chat4Data

Chat-Based Web Data Extraction

AI Web ScrapingWeb Scraping

Chrome-based AI scraping tool that lets users extract page data through conversational instructions and export structured results.

Thunderbit

AI Web Scraping From Browser Pages

AI Web ScrapingWeb Scraping

No-code AI web scraping tool for turning pages, PDFs, and images into structured tables from the browser.

Kadoa

Self-Healing AI Web Data Pipelines

AI Web ScrapingWeb Scraping

AI web scraping platform for building self-healing extraction agents that monitor pages and collect structured data at scale.

pgvector

Add Vector Search To PostgreSQL

Stars
21.9K
Forks
1.2K
Last commit
2 months ago

Open-source PostgreSQL extension for vector similarity search that keeps embeddings and scraped data in the same database.

Apache StormCrawler

Scalable Java Crawlers For Teams

Stars
981
Forks
282
Last commit
2 months ago

Build scalable Java crawlers on Apache Storm with low-latency processing, flexible customization, and an Apache-2.0 license.

got-scraping

Browser-Like Requests For Reliable Scraping

Stars
760
Forks
60
Last commit
3 months ago

A Node.js HTTP client for scraping that sends browser-like requests with realistic headers and TLS behavior.

Parsel

Extract Web Data Cleanly With Python

Stars
1.3K
Forks
162
Last commit
2 months ago

A Python library for extracting data from HTML, XML, and JSON with CSS, XPath, JMESPath, and regex.

Apache Solr

Scalable Search For Scraped Content

Stars
1.6K
Forks
839
Last commit
2 months ago

Open-source Java search software for indexing scraped content with full-text search and faceted navigation.

jsoup

Parse Real-World HTML In Java

Stars
11.4K
Forks
2.3K
Last commit
2 months ago

A Java library for parsing real-world HTML into a standards-based DOM with CSS selectors and DOM traversal.

selectolax

Fast HTML Parsing for Python Workflows

Stars
1.6K
Forks
91
Last commit
3 months ago

Parse HTML with CSS selectors in Python using a fast, MIT-licensed library built for large-scale extraction.

SeleniumBase

Reliable Browser Automation For Python

Stars
12.8K
Forks
1.6K
Last commit
2 months ago

An open-source Python framework for browser automation, testing, and practical anti-detection scraping with UC Mode and CDP Mode.

puppeteer-extra

Stealth Plugins For Puppeteer Scraping

Stars
7.4K
Forks
781
Last commit
2 years ago

A JavaScript plugin framework for Puppeteer that adds stealth options to make headless Chrome harder to detect.

curl_cffi

Browser-Like Requests Without Full Browsers

Stars
5.9K
Forks
501
Last commit
2 months ago

Python HTTP client for scraping with browser-like TLS, JA3, and HTTP/2 fingerprint impersonation without running a full browser.

Unstructured

Clean Documents For AI Pipelines

Stars
15K
Forks
1.3K
Last commit
2 months ago

An open-source Python document ETL tool that turns PDFs, HTML, images, and more into structured elements for RAG and LLM workflows.

Diffbot

Turn Web Pages Into Usable Data

AI Web Scraping

An AI extraction platform that turns web pages into structured data and a knowledge graph for web data workflows.

lxml

Fast XML And HTML Parsing For Python

Stars
3K
Forks
619
Last commit
2 months ago

A fast Python library for parsing XML and HTML, often used as a foundation for scraping and data tools.

Apache Tika

Extract Text From Any Document

Stars
3.8K
Forks
941
Last commit
2 months ago

Detects file types and extracts text and document details from 1,000+ formats through one Java interface.

Docling

Turn Documents Into AI-Ready Data

Stars
62.2K
Forks
4.4K
Last commit
2 months ago

Open-source Python software that converts PDFs, DOCX, PPTX, and images into structured data for GenAI and RAG pipelines.

Scrapfly

AI-Ready Web Data Extraction

Scraping APIs

A commercial scraping platform that helps teams collect structured web data, automate cloud browsers, crawl sites, handle anti-bot blocks, and use proxies.

Every tool in the directory - scraping, anti-blocking, data processing, databases, infrastructure, and utilities.