Auto-Discover And Parse Sitemaps In Python
Library or tool · Library / Self-hosted
Stars: 257 · Code license: GPL-3.0-or-later Python library that finds a site's sitemaps from robots.txt and common paths, recurses sitemap indexes, and streams pages with low memory.
Sitemap Scrapers
Python Requests With HTML Parsing
Library or tool · Self-hosted
Python library combining HTTP requests, HTML parsing, CSS selectors, and optional JavaScript rendering for scraping workflows.
Browser Automation +2 more
jQuery-Style HTML Parsing For Python
Library or tool · Library / Self-hosted
Python library that applies a jQuery-style API to HTML and XML documents for selection, traversal, and extraction.
HTML Parsers
Readable Selenium Automation For Python
Library or tool · Self-hosted
Python browser automation library that wraps Selenium with a higher-level API for clicking, typing, and finding visible elements.
Browser Automation +1 more
Example-Based HTML Extraction For Python
Library or tool · Library / Self-hosted
Stars: 8.1K · Code license: MIT Python library that learns simple scraping rules from example values and extracts similar data from HTML pages.
Unmaintained Python Port of Puppeteer
Library or tool · Library / Self-hosted
Python port of Puppeteer for Chrome and Chromium automation. Its maintainers mark the repository as unmaintained and suggest Playwright Python.
Browser Automation +1 more
Extract Web Data Cleanly With Python
Library or tool · Library / Self-hosted
Stars: 1.4K · Code license: BSD-3-Clause A Python library for extracting data from HTML, XML, and JSON with CSS, XPath, JMESPath, and regex.
HTML Parsers
Fast HTML Parsing for Python Workflows
Library or tool · Library / Self-hosted
Stars: 1.7K · Code license: MIT Parse HTML with CSS selectors in Python using a fast, MIT-licensed library built for large-scale extraction.
HTML Parsers
Reliable Browser Automation For Python
Library or tool · Self-hosted
An open-source Python framework for browser automation, testing, and practical anti-detection scraping with UC Mode and CDP Mode.
Browser Automation
Browser-Like Requests Without Full Browsers
Library or tool · Self-hosted
Python HTTP client for scraping with browser-like TLS, JA3, and HTTP/2 fingerprint impersonation without running a full browser.
Fingerprint & Stealth Tooling
Clean Documents For AI Pipelines
Library or tool · Self-hosted
An open-source Python document ETL tool that turns PDFs, HTML, images, and more into structured elements for RAG and LLM workflows.
Document Extraction
Fast XML And HTML Parsing For Python
Library or tool · Library / Self-hosted
Stars: 3.1K · Code license: BSD-3-Clause A fast Python library for parsing XML and HTML, often used as a foundation for scraping and data tools.
HTML Parsers
Turn Documents Into AI-Ready Data
Library or tool · Self-hosted
Open-source Python software that converts PDFs, DOCX, PPTX, and images into structured data for GenAI and RAG pipelines.
Document Extraction
Serverless compute for event-driven pipelines
Infrastructure · API / SaaS
AWS Lambda runs code without managing servers, ideal for event-driven scraping pipelines triggered by schedules, queues, or API calls.
Cloud Compute
Python library for parsing HTML and XML
Library or tool · Library / Self-hosted
Beautiful Soup is a Python library for pulling data out of HTML and XML files, providing Pythonic idioms for navigating and searching parse trees.
HTML Parsers
Async-first undetected browser automation
Library or tool · Self-hosted
Zendriver is a fast, async-first undetected browser automation framework based on nodriver with Docker support for scalable scraping.
Browser Automation +1 more
Scalable crawl frontier framework
Library or tool · Self-hosted
Frontera is a Python crawl frontier framework for managing when and what to crawl, enabling web crawlers of any scale with Scrapy integration.
Large-Scale & Distributed Crawling
Undetected Playwright fork for bot evasion
Library or tool · Self-hosted
Patchright is a patched fork of Playwright that evades bot detection with a Playwright-style API and documented compatibility limits.
Browser Automation +1 more
CLI-First ELT Engine Built on Singer
Library or tool · Self-hosted
Meltano is an open-source, CLI-first ELT platform with 600+ connectors, GitOps workflows, and native dbt integration for data pipeline management.
ETL Tools
AI agents and multi-agent workflows
Library or tool · Framework / Self-hosted
Stars: 3.4K · Code license: Apache-2.0 Python and TypeScript framework for building AI agents and multi-agent workflows with model providers, memory, tools and MCP integrations.
AI Agent Tooling
Undetected browser automation without Selenium
Library or tool · Library / Self-hosted
Stars: 4.8K · Code license: AGPL-3.0 Nodriver is the successor to undetected-chromedriver, providing fast CDP-based browser automation with built-in anti-detection and no Selenium dependency.
Browser Automation +1 more
Python Library for Data Loading
Library or tool · Self-hosted
dlt is a lightweight, open-source Python library for loading data from any source into well-structured datasets with automatic schema inference.
Data Transformation & Validation +1 more
Automate website interaction with Python
Library or tool · Library / Self-hosted
Stars: 4.9K · Code license: MIT MechanicalSoup is a Python library for automating website interaction, handling cookies, redirects, forms, and links without a browser.
Scraping Frameworks
Anti-detect browser with fingerprint injection
Library or tool · Self-hosted
Stars: 12.6K · Code license: MPL-2.0 Camoufox is an open-source anti-detect browser based on Firefox with robust fingerprint injection to evade bot detection systems.
Browser Automation +1 more
Redis-based distributed components for Scrapy
Library or tool · Self-hosted
Scrapy-Redis provides Redis-backed components for Scrapy, enabling distributed crawling with shared request queues and item pipelines.
Large-Scale & Distributed Crawling
Lightweight Python Task Queue
Library or tool · Self-hosted
Huey is a lightweight Python task queue with a clean API supporting Redis, SQLite, or in-memory storage for background job processing.
Task Queues
Lakehouse Platform for Incremental Processing
Infrastructure · Self-hosted
Apache Hudi is an open data lakehouse platform that enables efficient record-level upserts, incremental processing, and ACID transactions on data lakes.
Lakehouse & Table Formats
Message Queue and Background Jobs for Node.js
Library or tool · Self-hosted
BullMQ is a task queue for background jobs. Its JavaScript/TypeScript package supports Redis; version 6.3.11 also includes an optional PostgreSQL backend. Backend support must be checked for each language binding.
Task Queues
Open Table Format for Huge Analytic Datasets
Infrastructure · Self-hosted
Apache Iceberg is a high-performance open table format enabling multi-engine analytics with schema evolution, time travel, and hidden partitioning.
Lakehouse & Table Formats
Open-Source Lakehouse Storage Framework
Infrastructure · Self-hosted
Delta Lake is an open-source storage framework that brings ACID transactions, schema evolution, and time travel to data lakes at petabyte scale.
Lakehouse & Table Formats
Serverless vector database, no server management
Infrastructure · Self-hosted
LanceDB is a serverless vector database built in Rust that runs embedded or in the cloud with native DataFrame integration for AI workloads.
Vector Databases
AI-Powered Data Pipeline Builder
Library or tool · Self-hosted
Mage AI is a data engineering platform that lets you build, deploy, and run scalable data pipelines with AI assistance across Python, SQL, and R.
Workflow Orchestration
Adaptive web scraping for the modern web
Library or tool · Framework / Self-hosted
Stars: 87.9K · Code license: BSD-3-Clause Scrapling is an adaptive Python scraping framework that learns from website changes, handling everything from single requests to full-scale crawls.
Scraping Frameworks
Distributed NoSQL with high availability at scale
Infrastructure · Self-hosted
Apache Cassandra is a distributed NoSQL database that handles massive amounts of scraped data with high availability and linear scalability.
NoSQL Databases
Simple Job Queues for Python
Library or tool · Self-hosted
RQ is a lightweight Python library for queueing jobs and processing them in the background with Redis-backed workers.
Task Queues
Transform Data in Your Warehouse with SQL
Library or tool · Self-hosted
dbt empowers data teams to transform, test, and document data directly in the warehouse using SQL, with version control and governance built in.
Data Transformation & Validation
