Simple Sitemaps.org XML Parsing For Ruby
Library or tool · Library / Self-hosted
Stars: 33 · Code license: MIT Ruby parser for sitemaps.org-compliant XML sitemaps; a recurse option flattens nested indexes into one URL list returned by to_a.
Sitemap Scrapers
Discover And Parse A Domain's Sitemaps In C#
Library or tool · Library / Self-hosted
C# library whose GetAllSitemapsForDomainAsync() takes a domain and discovers plus parses all its sitemaps, with lenient parsing for messy real-world files.
Sitemap Scrapers
Recursive Sitemap Parsing For PHP
Library or tool · Library / Self-hosted
Stars: 78 · Code license: MIT PHP library with parse() and parseRecursive() APIs; parseRecursive on a robots.txt URL walks every nested sitemap for a URL list.
Sitemap Scrapers
Sitemap and Robots.txt Parsing for Java
Library or tool · Library / Self-hosted
Stars: 259 · Code license: Apache-2.0 Java library with a SAX-based sitemap parser for sitemaps, indexes, extensions, and feeds; you supply the bytes and it parses and normalizes them.
Sitemap Scrapers
Zero-Dependency Streaming Sitemap Parser In Go
Library or tool · Library / Self-hosted
Streaming Go parser for huge XML sitemaps with minimal memory and no external dependencies, consuming entries on the fly via callbacks.
Sitemap Scrapers
Resolve Robots, Indexes, And Sitemaps In Go
Library or tool · Library / Self-hosted
Stars: 8 · Code license: MIT Go library whose Parse() accepts a robots.txt, sitemap index, or plain sitemap URL and resolves the rest, with concurrent fetching and depth limits.
Sitemap Scrapers
Stream Very Large Sitemaps With Low Memory
Library or tool · Library / Self-hosted
Streaming Node.js XML sitemap parser for very large, deeply nested sitemaps, with a callback API and a helper that finds sitemaps in robots.txt.
Sitemap Scrapers
Parse Sitemaps And Indexes In Node.js
Library or tool · Library / Self-hosted
Stars: 137 · Code license: MIT Node.js library that parses a sitemap URL including sitemap indexes with many child sitemaps, with concurrency, retries, timeout, and proxy options.
Sitemap Scrapers
Auto-Discover And Parse Sitemaps In Python
Library or tool · Library / Self-hosted
Stars: 257 · Code license: GPL-3.0-or-later Python library that finds a site's sitemaps from robots.txt and common paths, recurses sitemap indexes, and streams pages with low memory.
Sitemap Scrapers
Python Requests With HTML Parsing
Library or tool · Self-hosted
Python library combining HTTP requests, HTML parsing, CSS selectors, and optional JavaScript rendering for scraping workflows.
Browser Automation +2 more
jQuery-Style HTML Parsing For Python
Library or tool · Library / Self-hosted
Python library that applies a jQuery-style API to HTML and XML documents for selection, traversal, and extraction.
HTML Parsers
Readable Selenium Automation For Python
Library or tool · Self-hosted
Python browser automation library that wraps Selenium with a higher-level API for clicking, typing, and finding visible elements.
Browser Automation +1 more
Example-Based HTML Extraction For Python
Library or tool · Library / Self-hosted
Stars: 8.1K · Code license: MIT Python library that learns simple scraping rules from example values and extracts similar data from HTML pages.
Unmaintained Python Port of Puppeteer
Library or tool · Library / Self-hosted
Python port of Puppeteer for Chrome and Chromium automation. Its maintainers mark the repository as unmaintained and suggest Playwright Python.
Browser Automation +1 more
Add Vector Search To PostgreSQL
Infrastructure · Self-hosted
Open-source PostgreSQL extension for vector similarity search that keeps embeddings and scraped data in the same database.
Vector Databases
Scalable Java Crawlers For Teams
Library or tool · Self-hosted
Build scalable Java crawlers on Apache Storm with low-latency processing, flexible customization, and an Apache-2.0 license.
Large-Scale & Distributed Crawling
Browser-Like Requests For Reliable Scraping
Library or tool · Library / Self-hosted
Stars: 776 · Code license: Apache-2.0 A Node.js HTTP client for scraping that sends browser-like requests with realistic headers and TLS behavior.
Scraping Frameworks
Extract Web Data Cleanly With Python
Library or tool · Library / Self-hosted
Stars: 1.4K · Code license: BSD-3-Clause A Python library for extracting data from HTML, XML, and JSON with CSS, XPath, JMESPath, and regex.
HTML Parsers
Scalable Search For Scraped Content
Infrastructure · Self-hosted
Open-source Java search software for indexing scraped content with full-text search and faceted navigation.
Search Engines
Parse Real-World HTML In Java
Library or tool · Library / Self-hosted
Stars: 11.4K · Code license: MIT A Java library for parsing real-world HTML into a standards-based DOM with CSS selectors and DOM traversal.
HTML Parsers
Fast HTML Parsing for Python Workflows
Library or tool · Library / Self-hosted
Stars: 1.7K · Code license: MIT Parse HTML with CSS selectors in Python using a fast, MIT-licensed library built for large-scale extraction.
HTML Parsers
Reliable Browser Automation For Python
Library or tool · Self-hosted
An open-source Python framework for browser automation, testing, and practical anti-detection scraping with UC Mode and CDP Mode.
Browser Automation
Stealth Plugins For Puppeteer Scraping
Library or tool · Self-hosted
A JavaScript plugin framework for Puppeteer that adds stealth options to make headless Chrome harder to detect.
Fingerprint & Stealth Tooling
Browser-Like Requests Without Full Browsers
Library or tool · Self-hosted
Python HTTP client for scraping with browser-like TLS, JA3, and HTTP/2 fingerprint impersonation without running a full browser.
Fingerprint & Stealth Tooling
Clean Documents For AI Pipelines
Library or tool · Self-hosted
An open-source Python document ETL tool that turns PDFs, HTML, images, and more into structured elements for RAG and LLM workflows.
Document Extraction
Fast XML And HTML Parsing For Python
Library or tool · Library / Self-hosted
Stars: 3.1K · Code license: BSD-3-Clause A fast Python library for parsing XML and HTML, often used as a foundation for scraping and data tools.
HTML Parsers
Extract Text From Any Document
Library or tool · Self-hosted
Detects file types and extracts text and document details from 1,000+ formats through one Java interface.
Document Extraction
Turn Documents Into AI-Ready Data
Library or tool · Self-hosted
Open-source Python software that converts PDFs, DOCX, PPTX, and images into structured data for GenAI and RAG pipelines.
Document Extraction
Automate Web Workflows With AI
Library or tool · Library / Self-hosted
Stars: 25.7K · Code license: MIT Stagehand is an open-source SDK for AI-assisted browser automation with natural-language actions, structured extraction, and Playwright-style APIs.
Python library for parsing HTML and XML
Library or tool · Library / Self-hosted
Beautiful Soup is a Python library for pulling data out of HTML and XML files, providing Pythonic idioms for navigating and searching parse trees.
HTML Parsers
Async-first undetected browser automation
Library or tool · Self-hosted
Zendriver is a fast, async-first undetected browser automation framework based on nodriver with Docker support for scalable scraping.
Browser Automation +1 more
Distributed on-demand scraping with Scrapy
Library or tool · Self-hosted
Scrapy Cluster uses Redis and Kafka to create a distributed, on-demand Scrapy crawling cluster for coordinated large-scale web scraping. Its repository is archived.
Large-Scale & Distributed Crawling
Scalable crawl frontier framework
Library or tool · Self-hosted
Frontera is a Python crawl frontier framework for managing when and what to crawl, enabling web crawlers of any scale with Scrapy integration.
Large-Scale & Distributed Crawling
Undetected Playwright fork for bot evasion
Library or tool · Self-hosted
Patchright is a patched fork of Playwright that evades bot detection with a Playwright-style API and documented compatibility limits.
Browser Automation +1 more
CLI-First ELT Engine Built on Singer
Library or tool · Self-hosted
Meltano is an open-source, CLI-first ELT platform with 600+ connectors, GitOps workflows, and native dbt integration for data pipeline management.
ETL Tools
End-of-life proxy orchestration project
Library or tool · Retired · Self-hosted
Scrapoxy is an open-source proxy orchestration project that reached End of Life as announced on scrapoxy.io, so it is historical context rather than a new-deployment choice.
Proxy Services
