Simple Sitemaps.org XML Parsing For Ruby
Library or tool · Library / Self-hosted
Stars: 33 · Code license: MIT Ruby parser for sitemaps.org-compliant XML sitemaps; a recurse option flattens nested indexes into one URL list returned by to_a.
Sitemap Scrapers
Discover And Parse A Domain's Sitemaps In C#
Library or tool · Library / Self-hosted
C# library whose GetAllSitemapsForDomainAsync() takes a domain and discovers plus parses all its sitemaps, with lenient parsing for messy real-world files.
Sitemap Scrapers
Recursive Sitemap Parsing For PHP
Library or tool · Library / Self-hosted
Stars: 78 · Code license: MIT PHP library with parse() and parseRecursive() APIs; parseRecursive on a robots.txt URL walks every nested sitemap for a URL list.
Sitemap Scrapers
Sitemap and Robots.txt Parsing for Java
Library or tool · Library / Self-hosted
Stars: 259 · Code license: Apache-2.0 Java library with a SAX-based sitemap parser for sitemaps, indexes, extensions, and feeds; you supply the bytes and it parses and normalizes them.
Sitemap Scrapers
Zero-Dependency Streaming Sitemap Parser In Go
Library or tool · Library / Self-hosted
Streaming Go parser for huge XML sitemaps with minimal memory and no external dependencies, consuming entries on the fly via callbacks.
Sitemap Scrapers
Resolve Robots, Indexes, And Sitemaps In Go
Library or tool · Library / Self-hosted
Stars: 8 · Code license: MIT Go library whose Parse() accepts a robots.txt, sitemap index, or plain sitemap URL and resolves the rest, with concurrent fetching and depth limits.
Sitemap Scrapers
Stream Very Large Sitemaps With Low Memory
Library or tool · Library / Self-hosted
Streaming Node.js XML sitemap parser for very large, deeply nested sitemaps, with a callback API and a helper that finds sitemaps in robots.txt.
Sitemap Scrapers
Parse Sitemaps And Indexes In Node.js
Library or tool · Library / Self-hosted
Stars: 137 · Code license: MIT Node.js library that parses a sitemap URL including sitemap indexes with many child sitemaps, with concurrency, retries, timeout, and proxy options.
Sitemap Scrapers
Auto-Discover And Parse Sitemaps In Python
Library or tool · Library / Self-hosted
Stars: 257 · Code license: GPL-3.0-or-later Python library that finds a site's sitemaps from robots.txt and common paths, recurses sitemap indexes, and streams pages with low memory.
Sitemap Scrapers
Python Requests With HTML Parsing
Library or tool · Self-hosted
Python library combining HTTP requests, HTML parsing, CSS selectors, and optional JavaScript rendering for scraping workflows.
Browser Automation +2 more
jQuery-Style HTML Parsing For Python
Library or tool · Library / Self-hosted
Python library that applies a jQuery-style API to HTML and XML documents for selection, traversal, and extraction.
HTML Parsers
Readable Selenium Automation For Python
Library or tool · Self-hosted
Python browser automation library that wraps Selenium with a higher-level API for clicking, typing, and finding visible elements.
Browser Automation +1 more
Example-Based HTML Extraction For Python
Library or tool · Library / Self-hosted
Stars: 8.1K · Code license: MIT Python library that learns simple scraping rules from example values and extracts similar data from HTML pages.
Unmaintained Python Port of Puppeteer
Library or tool · Library / Self-hosted
Python port of Puppeteer for Chrome and Chromium automation. Its maintainers mark the repository as unmaintained and suggest Playwright Python.
Browser Automation +1 more
Add Vector Search To PostgreSQL
Infrastructure · Self-hosted
Open-source PostgreSQL extension for vector similarity search that keeps embeddings and scraped data in the same database.
Vector Databases
Parse Real-World HTML In Java
Library or tool · Library / Self-hosted
Stars: 11.4K · Code license: MIT A Java library for parsing real-world HTML into a standards-based DOM with CSS selectors and DOM traversal.
HTML Parsers
Fast XML And HTML Parsing For Python
Library or tool · Library / Self-hosted
Stars: 3.1K · Code license: BSD-3-Clause A fast Python library for parsing XML and HTML, often used as a foundation for scraping and data tools.
HTML Parsers
Python library for parsing HTML and XML
Library or tool · Library / Self-hosted
Beautiful Soup is a Python library for pulling data out of HTML and XML files, providing Pythonic idioms for navigating and searching parse trees.
HTML Parsers
Async-first undetected browser automation
Library or tool · Self-hosted
Zendriver is a fast, async-first undetected browser automation framework based on nodriver with Docker support for scalable scraping.
Browser Automation +1 more
Distributed on-demand scraping with Scrapy
Library or tool · Self-hosted
Scrapy Cluster uses Redis and Kafka to create a distributed, on-demand Scrapy crawling cluster for coordinated large-scale web scraping. Its repository is archived.
Large-Scale & Distributed Crawling
Scalable crawl frontier framework
Library or tool · Self-hosted
Frontera is a Python crawl frontier framework for managing when and what to crawl, enabling web crawlers of any scale with Scrapy integration.
Large-Scale & Distributed Crawling
Undetected Playwright fork for bot evasion
Library or tool · Self-hosted
Patchright is a patched fork of Playwright that evades bot detection with a Playwright-style API and documented compatibility limits.
Browser Automation +1 more
CLI-First ELT Engine Built on Singer
Library or tool · Self-hosted
Meltano is an open-source, CLI-first ELT platform with 600+ connectors, GitOps workflows, and native dbt integration for data pipeline management.
ETL Tools
End-of-life proxy orchestration project
Library or tool · Retired · Self-hosted
Scrapoxy is an open-source proxy orchestration project that reached End of Life as announced on scrapoxy.io, so it is historical context rather than a new-deployment choice.
Proxy Services
AI agents and multi-agent workflows
Library or tool · Framework / Self-hosted
Stars: 3.4K · Code license: Apache-2.0 Python and TypeScript framework for building AI agents and multi-agent workflows with model providers, memory, tools and MCP integrations.
AI Agent Tooling
Highly extensible and scalable web crawler
Library or tool · Self-hosted
Apache Nutch is a highly extensible, production-ready web crawler built on Hadoop for large-scale batch crawling and data acquisition tasks.
Large-Scale & Distributed Crawling
Web-scale archival web crawler
Library or tool · Self-hosted
Heritrix is the Internet Archive's open-source, extensible, archival-quality web crawler designed for large-scale web preservation and data collection.
Large-Scale & Distributed Crawling
Undetected browser automation without Selenium
Library or tool · Library / Self-hosted
Stars: 4.8K · Code license: AGPL-3.0 Nodriver is the successor to undetected-chromedriver, providing fast CDP-based browser automation with built-in anti-detection and no Selenium dependency.
Browser Automation +1 more
HTML5 spec-compliant parser for Node.js
Library or tool · Library / Self-hosted
Stars: 3.9K · Code license: MIT parse5 is a spec-compliant HTML5 parser for Node.js, parsing HTML exactly as modern browsers do.
HTML Parsers
Durable Functions for Workflows and Queues
Library or tool · Self-hosted
Inngest replaces traditional queues with durable step functions, enabling reliable multi-step workflows on serverless, servers, or the edge.
Workflow Orchestration
The fast and forgiving HTML/XML parser
Library or tool · Library / Self-hosted
Stars: 4.8K · Code license: MIT htmlparser2 is a streaming Node.js HTML and XML parser, offering a streaming, event-driven API with a forgiving approach to malformed markup.
HTML Parsers
Python Library for Data Loading
Library or tool · Self-hosted
dlt is a lightweight, open-source Python library for loading data from any source into well-structured datasets with automatic schema inference.
Data Transformation & Validation +1 more
Automate website interaction with Python
Library or tool · Library / Self-hosted
Stars: 4.9K · Code license: MIT MechanicalSoup is a Python library for automating website interaction, handling cookies, redirects, forms, and links without a browser.
Scraping Frameworks
Anti-detect browser with fingerprint injection
Library or tool · Self-hosted
Stars: 12.6K · Code license: MPL-2.0 Camoufox is an open-source anti-detect browser based on Firefox with robust fingerprint injection to evade bot detection systems.
Browser Automation +1 more
Redis-based distributed components for Scrapy
Library or tool · Self-hosted
Scrapy-Redis provides Redis-backed components for Scrapy, enabling distributed crawling with shared request queues and item pipelines.
Large-Scale & Distributed Crawling
Lightweight Python Task Queue
Library or tool · Self-hosted
Huey is a lightweight Python task queue with a clean API supporting Redis, SQLite, or in-memory storage for background job processing.
Task Queues
