AI agents and multi-agent workflows
Library or tool · Framework / Self-hosted
Stars: 3.4K · Code license: Apache-2.0 Python and TypeScript framework for building AI agents and multi-agent workflows with model providers, memory, tools and MCP integrations.
AI Agent Tooling
Highly extensible and scalable web crawler
Library or tool · Self-hosted
Apache Nutch is a highly extensible, production-ready web crawler built on Hadoop for large-scale batch crawling and data acquisition tasks.
Large-Scale & Distributed Crawling
Web-scale archival web crawler
Library or tool · Self-hosted
Heritrix is the Internet Archive's open-source, extensible, archival-quality web crawler designed for large-scale web preservation and data collection.
Large-Scale & Distributed Crawling
Undetected browser automation without Selenium
Library or tool · Library / Self-hosted
Stars: 4.8K · Code license: AGPL-3.0 Nodriver is the successor to undetected-chromedriver, providing fast CDP-based browser automation with built-in anti-detection and no Selenium dependency.
Browser Automation +1 more
HTML5 spec-compliant parser for Node.js
Library or tool · Library / Self-hosted
Stars: 3.9K · Code license: MIT parse5 is a spec-compliant HTML5 parser for Node.js, parsing HTML exactly as modern browsers do.
HTML Parsers
Durable Functions for Workflows and Queues
Library or tool · Self-hosted
Inngest replaces traditional queues with durable step functions, enabling reliable multi-step workflows on serverless, servers, or the edge.
Workflow Orchestration
The fast and forgiving HTML/XML parser
Library or tool · Library / Self-hosted
Stars: 4.8K · Code license: MIT htmlparser2 is a streaming Node.js HTML and XML parser, offering a streaming, event-driven API with a forgiving approach to malformed markup.
HTML Parsers
Python Library for Data Loading
Library or tool · Self-hosted
dlt is a lightweight, open-source Python library for loading data from any source into well-structured datasets with automatic schema inference.
Data Transformation & Validation +1 more
Automate website interaction with Python
Library or tool · Library / Self-hosted
Stars: 4.9K · Code license: MIT MechanicalSoup is a Python library for automating website interaction, handling cookies, redirects, forms, and links without a browser.
Scraping Frameworks
Anti-detect browser with fingerprint injection
Library or tool · Self-hosted
Stars: 12.6K · Code license: MPL-2.0 Camoufox is an open-source anti-detect browser based on Firefox with robust fingerprint injection to evade bot detection systems.
Browser Automation +1 more
Redis-based distributed components for Scrapy
Library or tool · Self-hosted
Scrapy-Redis provides Redis-backed components for Scrapy, enabling distributed crawling with shared request queues and item pipelines.
Large-Scale & Distributed Crawling
Lightweight Python Task Queue
Library or tool · Self-hosted
Huey is a lightweight Python task queue with a clean API supporting Redis, SQLite, or in-memory storage for background job processing.
Task Queues
Lakehouse Platform for Incremental Processing
Infrastructure · Self-hosted
Apache Hudi is an open data lakehouse platform that enables efficient record-level upserts, incremental processing, and ACID transactions on data lakes.
Lakehouse & Table Formats
Structured Web Extraction with LLMs
Library or tool · Library / Self-hosted
Stars: 6.9K · Code license: MIT LLM Scraper is a TypeScript library that extracts structured data from web pages using LLMs with Zod schemas, Playwright, and streaming support.
High-Level DevTools Driver for Go
Library or tool · Self-hosted
Rod is a Go library for web automation and scraping that drives browsers via the DevTools Protocol with a simple, chainable API.
Browser Automation
Message Queue and Background Jobs for Node.js
Library or tool · Self-hosted
BullMQ is a task queue for background jobs. Its JavaScript/TypeScript package supports Redis; version 6.3.11 also includes an optional PostgreSQL backend. Backend support must be checked for each language binding.
Task Queues
Open Table Format for Huge Analytic Datasets
Infrastructure · Self-hosted
Apache Iceberg is a high-performance open table format enabling multi-engine analytics with schema evolution, time travel, and hidden partitioning.
Lakehouse & Table Formats
Open-Source Lakehouse Storage Framework
Infrastructure · Self-hosted
Delta Lake is an open-source storage framework that brings ACID transactions, schema evolution, and time travel to data lakes at petabyte scale.
Lakehouse & Table Formats
Serverless vector database, no server management
Infrastructure · Self-hosted
LanceDB is a serverless vector database built in Rust that runs embedded or in the cloud with native DataFrame integration for AI workloads.
Vector Databases
AI-Powered Data Pipeline Builder
Library or tool · Self-hosted
Mage AI is a data engineering platform that lets you build, deploy, and run scalable data pipelines with AI assistance across Python, SQL, and R.
Workflow Orchestration
Adaptive web scraping for the modern web
Library or tool · Framework / Self-hosted
Stars: 87.9K · Code license: BSD-3-Clause Scrapling is an adaptive Python scraping framework that learns from website changes, handling everything from single requests to full-scale crawls.
Scraping Frameworks
Distributed NoSQL with high availability at scale
Infrastructure · Self-hosted
Apache Cassandra is a distributed NoSQL database that handles massive amounts of scraped data with high availability and linear scalability.
NoSQL Databases
Next-Gen Browser and Mobile Automation for Node.js
Library or tool · Self-hosted
WebdriverIO is an all-in-one Node.js automation framework for web and mobile testing with smart selectors, component testing, and extensive plugins.
Browser Automation +1 more
WebDriver-Free End-to-End Browser Testing
Library or tool · Self-hosted
TestCafe is a free, open-source end-to-end testing framework that works across browsers without WebDriver, with intuitive syntax and built-in waiting.
Browser Automation
Simple Job Queues for Python
Library or tool · Self-hosted
RQ is a lightweight Python library for queueing jobs and processing them in the background with Redis-backed workers.
Task Queues
Transform Data in Your Warehouse with SQL
Library or tool · Self-hosted
dbt empowers data teams to transform, test, and document data directly in the warehouse using SQL, with version control and governance built in.
Data Transformation & Validation
Data Quality Testing and Validation Platform
Library or tool · Self-hosted
Great Expectations helps data teams validate, document, and monitor data quality across pipelines with automated testing and observability.
Data Transformation & Validation
Sub-second analytics for enterprise-scale data
Infrastructure · Self-hosted
StarRocks is an open-source analytical database that delivers sub-second query latency for complex joins and aggregations at enterprise scale.
Analytics Databases
Open-source search database with SQL and HTTP APIs
Infrastructure · Self-hosted
Manticore Search is an open-source search database for full-text search and analytical queries through SQL and HTTP APIs.
Search Engines
Open-source search and analytics suite
Infrastructure · Self-hosted
OpenSearch is an Apache 2.0-licensed search and analytics suite that lets you ingest, search, visualize, and analyze scraped data at scale.
Search Engines
Multithreaded Redis fork with higher throughput
Infrastructure · Self-hosted
KeyDB is a multithreaded Redis-compatible database for caching, data structures, and background-job infrastructure.
Caching Databases
Chrome DevTools Protocol Driver for Go
Library or tool · Library / Self-hosted
Stars: 13.3K · Code license: MIT Chromedp is a Go package for driving CDP-compatible browsers for scraping, testing, and automation without Selenium or WebDriver.
Browser Automation
Real-time OLAP database for event-driven data
Infrastructure · Self-hosted
Apache Druid is a real-time OLAP database that delivers sub-second queries on streaming and batch data for analytics at scale.
Analytics Databases
Simple, fast distributed memory caching system
Infrastructure · Self-hosted
Memcached is a free, high-performance distributed memory caching system that speeds up dynamic applications by caching scraped data in RAM.
Caching Databases
Unified Platform for AI and Data Pipelines
Library or tool · Self-hosted
Dagster is a data orchestration platform that helps teams build, schedule, and monitor reliable data pipelines with integrated lineage and observability.
Workflow Orchestration
jQuery-like HTML parsing for Go
Library or tool · Library / Self-hosted
Stars: 15K · Code license: BSD-3-Clause GoQuery brings jQuery-like syntax to Go for parsing and manipulating HTML documents, making web scraping in Go intuitive and efficient.
HTML Parsers
