Open Source Web Scraping Tools

Ben Balter sitemap-parser

Simple Sitemaps.org XML Parsing For Ruby

Stars
33
Forks
47
Last recorded push
last year

Ruby parser for sitemaps.org-compliant XML sitemaps; a recurse option flattens nested indexes into one URL list returned by to_a.

TurnerSoftware SitemapTools

Discover And Parse A Domain's Sitemaps In C#

Stars
38
Forks
10
Last recorded push
2 years ago

C# library whose GetAllSitemapsForDomainAsync() takes a domain and discovers plus parses all its sitemaps, with lenient parsing for messy real-world files.

VIPnytt SitemapParser

Recursive Sitemap Parsing For PHP

Stars
78
Forks
33
Last recorded push
last year

PHP library with parse() and parseRecursive() APIs; parseRecursive on a robots.txt URL walks every nested sitemap for a URL list.

Crawler-Commons

Sitemap and Robots.txt Parsing for Java

Stars
259
Forks
91
Last recorded push
3 months ago

Java library with a SAX-based sitemap parser for sitemaps, indexes, extensions, and feeds; you supply the bytes and it parses and normalizes them.

gopher-parse-sitemap

Zero-Dependency Streaming Sitemap Parser In Go

Stars
40
Forks
19
Last recorded push
3 years ago

Streaming Go parser for huge XML sitemaps with minimal memory and no external dependencies, consuming entries on the fly via callbacks.

go-sitemap-parser

Resolve Robots, Indexes, And Sitemaps In Go

Stars
8
Forks
1
Last recorded push
5 months ago

Go library whose Parse() accepts a robots.txt, sitemap index, or plain sitemap URL and resolves the rest, with concurrent fetching and depth limits.

sitemap-stream-parser

Stream Very Large Sitemaps With Low Memory

Stars
39
Forks
19
Last recorded push
5 years ago

Streaming Node.js XML sitemap parser for very large, deeply nested sitemaps, with a callback API and a helper that finds sitemaps in robots.txt.

Sitemapper

Parse Sitemaps And Indexes In Node.js

Stars
137
Forks
83
Last recorded push
4 months ago

Node.js library that parses a sitemap URL including sitemap indexes with many child sitemaps, with concurrency, retries, timeout, and proxy options.

Ultimate Sitemap Parser

Auto-Discover And Parse Sitemaps In Python

Stars
257
Forks
74
Last recorded push
8 months ago

Python library that finds a site's sitemaps from robots.txt and common paths, recurses sitemap indexes, and streams pages with low memory.

Requests-HTML

Python Requests With HTML Parsing

Stars
13.8K
Forks
999
Last recorded push
2 years ago

Python library combining HTTP requests, HTML parsing, CSS selectors, and optional JavaScript rendering for scraping workflows.

pyquery

jQuery-Style HTML Parsing For Python

Stars
2.4K
Forks
184
Last recorded push
7 months ago

Python library that applies a jQuery-style API to HTML and XML documents for selection, traversal, and extraction.

Helium

Readable Selenium Automation For Python

Stars
8.3K
Forks
513
Last recorded push
3 months ago

Python browser automation library that wraps Selenium with a higher-level API for clicking, typing, and finding visible elements.

AutoScraper

Example-Based HTML Extraction For Python

Stars
8K
Forks
764
Last recorded push
last year

Python library that learns simple scraping rules from example values and extracts similar data from HTML pages.

Pyppeteer

Unmaintained Python Port of Puppeteer

Stars
3.9K
Forks
347
Last recorded push
2 years ago

Python port of Puppeteer for Chrome and Chromium automation. Its maintainers mark the repository as unmaintained and suggest Playwright Python.

pgvector

Add Vector Search To PostgreSQL

Stars
23.3K
Forks
1.2K
Last recorded push
3 months ago

Open-source PostgreSQL extension for vector similarity search that keeps embeddings and scraped data in the same database.

Apache StormCrawler

Scalable Java Crawlers For Teams

Stars
1K
Forks
282
Last recorded push
3 months ago

Build scalable Java crawlers on Apache Storm with low-latency processing, flexible customization, and an Apache-2.0 license.

got-scraping

Browser-Like Requests For Reliable Scraping

Stars
774
Forks
60
Last recorded push
4 months ago

A Node.js HTTP client for scraping that sends browser-like requests with realistic headers and TLS behavior.

Parsel

Extract Web Data Cleanly With Python

Stars
1.4K
Forks
162
Last recorded push
3 months ago

A Python library for extracting data from HTML, XML, and JSON with CSS, XPath, JMESPath, and regex.

Apache Solr

Scalable Search For Scraped Content

Stars
1.7K
Forks
839
Last recorded push
3 months ago

Open-source Java search software for indexing scraped content with full-text search and faceted navigation.

jsoup

Parse Real-World HTML In Java

Stars
11.4K
Forks
2.3K
Last recorded push
3 months ago

A Java library for parsing real-world HTML into a standards-based DOM with CSS selectors and DOM traversal.

selectolax

Fast HTML Parsing for Python Workflows

Stars
1.7K
Forks
91
Last recorded push
4 months ago

Parse HTML with CSS selectors in Python using a fast, MIT-licensed library built for large-scale extraction.

SeleniumBase

Reliable Browser Automation For Python

Stars
13.1K
Forks
1.6K
Last recorded push
3 months ago

An open-source Python framework for browser automation, testing, and practical anti-detection scraping with UC Mode and CDP Mode.

puppeteer-extra

Stealth Plugins For Puppeteer Scraping

Stars
7.4K
Forks
781
Last recorded push
2 years ago

A JavaScript plugin framework for Puppeteer that adds stealth options to make headless Chrome harder to detect.

curl_cffi

Browser-Like Requests Without Full Browsers

Stars
6.7K
Forks
501
Last recorded push
3 months ago

Python HTTP client for scraping with browser-like TLS, JA3, and HTTP/2 fingerprint impersonation without running a full browser.

Unstructured

Clean Documents For AI Pipelines

Stars
15.6K
Forks
1.3K
Last recorded push
3 months ago

An open-source Python document ETL tool that turns PDFs, HTML, images, and more into structured elements for RAG and LLM workflows.

lxml

Fast XML And HTML Parsing For Python

Stars
3.1K
Forks
619
Last recorded push
3 months ago

A fast Python library for parsing XML and HTML, often used as a foundation for scraping and data tools.

Apache Tika

Extract Text From Any Document

Stars
4.1K
Forks
941
Last recorded push
3 months ago

Detects file types and extracts text and document details from 1,000+ formats through one Java interface.

Docling

Turn Documents Into AI-Ready Data

Stars
68.7K
Forks
4.4K
Last recorded push
3 months ago

Open-source Python software that converts PDFs, DOCX, PPTX, and images into structured data for GenAI and RAG pipelines.

Stagehand

Automate Web Workflows With AI

Stars
25.6K
Forks
1.6K
Last recorded push
3 months ago

Stagehand is an open-source SDK for AI-assisted browser automation with natural-language actions, structured extraction, and Playwright-style APIs.

BeautifulSoup

Python library for parsing HTML and XML

Stars
226
Forks
57
Last recorded push
3 years ago

Beautiful Soup is a Python library for pulling data out of HTML and XML files, providing Pythonic idioms for navigating and searching parse trees.

Zendriver

Async-first undetected browser automation

Stars
1.5K
Forks
75
Last recorded push
10 months ago

Zendriver is a fast, async-first undetected browser automation framework based on nodriver with Docker support for scalable scraping.

Scrapy Cluster

Distributed on-demand scraping with Scrapy

Stars
1.2K
Forks
322
Last recorded push
2 years ago

Scrapy Cluster uses Redis and Kafka to create a distributed, on-demand Scrapy crawling cluster for coordinated large-scale web scraping. Its repository is archived.

Frontera

Scalable crawl frontier framework

Stars
1.3K
Forks
217
Last recorded push
last year

Frontera is a Python crawl frontier framework for managing when and what to crawl, enabling web crawlers of any scale with Scrapy integration.

Patchright

Undetected Playwright fork for bot evasion

Stars
4.8K
Forks
88
Last recorded push
8 months ago

Patchright is a patched fork of Playwright that evades bot detection with a Playwright-style API and documented compatibility limits.

Meltano

CLI-First ELT Engine Built on Singer

Stars
2.7K
Forks
0

Meltano is an open-source, CLI-first ELT platform with 600+ connectors, GitOps workflows, and native dbt integration for data pipeline management.

Scrapoxy

End-of-life proxy orchestration project

Stars
2.4K
Forks
0

Scrapoxy is an open-source proxy orchestration project that reached End of Life as announced on scrapoxy.io, so it is historical context rather than a new-deployment choice.