Lakehouse Platform for Incremental Processing
Infrastructure · Self-hosted
Apache Hudi is an open data lakehouse platform that enables efficient record-level upserts, incremental processing, and ACID transactions on data lakes.
Lakehouse & Table Formats
Structured Web Extraction with LLMs
Library or tool · Library / Self-hosted
Stars: 6.9K · Code license: MIT LLM Scraper is a TypeScript library that extracts structured data from web pages using LLMs with Zod schemas, Playwright, and streaming support.
High-Level DevTools Driver for Go
Library or tool · Self-hosted
Rod is a Go library for web automation and scraping that drives browsers via the DevTools Protocol with a simple, chainable API.
Browser Automation
Message Queue and Background Jobs for Node.js
Library or tool · Self-hosted
BullMQ is a task queue for background jobs. Its JavaScript/TypeScript package supports Redis; version 6.3.11 also includes an optional PostgreSQL backend. Backend support must be checked for each language binding.
Task Queues
Open Table Format for Huge Analytic Datasets
Infrastructure · Self-hosted
Apache Iceberg is a high-performance open table format enabling multi-engine analytics with schema evolution, time travel, and hidden partitioning.
Lakehouse & Table Formats
Open-Source Lakehouse Storage Framework
Infrastructure · Self-hosted
Delta Lake is an open-source storage framework that brings ACID transactions, schema evolution, and time travel to data lakes at petabyte scale.
Lakehouse & Table Formats
Serverless vector database, no server management
Infrastructure · Self-hosted
LanceDB is a serverless vector database built in Rust that runs embedded or in the cloud with native DataFrame integration for AI workloads.
Vector Databases
AI-Powered Data Pipeline Builder
Library or tool · Self-hosted
Mage AI is a data engineering platform that lets you build, deploy, and run scalable data pipelines with AI assistance across Python, SQL, and R.
Workflow Orchestration
Adaptive web scraping for the modern web
Library or tool · Framework / Self-hosted
Stars: 87.9K · Code license: BSD-3-Clause Scrapling is an adaptive Python scraping framework that learns from website changes, handling everything from single requests to full-scale crawls.
Scraping Frameworks
Distributed NoSQL with high availability at scale
Infrastructure · Self-hosted
Apache Cassandra is a distributed NoSQL database that handles massive amounts of scraped data with high availability and linear scalability.
NoSQL Databases
Next-Gen Browser and Mobile Automation for Node.js
Library or tool · Self-hosted
WebdriverIO is an all-in-one Node.js automation framework for web and mobile testing with smart selectors, component testing, and extensive plugins.
Browser Automation +1 more
WebDriver-Free End-to-End Browser Testing
Library or tool · Self-hosted
TestCafe is a free, open-source end-to-end testing framework that works across browsers without WebDriver, with intuitive syntax and built-in waiting.
Browser Automation
Simple Job Queues for Python
Library or tool · Self-hosted
RQ is a lightweight Python library for queueing jobs and processing them in the background with Redis-backed workers.
Task Queues
Transform Data in Your Warehouse with SQL
Library or tool · Self-hosted
dbt empowers data teams to transform, test, and document data directly in the warehouse using SQL, with version control and governance built in.
Data Transformation & Validation
Data Quality Testing and Validation Platform
Library or tool · Self-hosted
Great Expectations helps data teams validate, document, and monitor data quality across pipelines with automated testing and observability.
Data Transformation & Validation
Sub-second analytics for enterprise-scale data
Infrastructure · Self-hosted
StarRocks is an open-source analytical database that delivers sub-second query latency for complex joins and aggregations at enterprise scale.
Analytics Databases
Open-source search database with SQL and HTTP APIs
Infrastructure · Self-hosted
Manticore Search is an open-source search database for full-text search and analytical queries through SQL and HTTP APIs.
Search Engines
Open-source search and analytics suite
Infrastructure · Self-hosted
OpenSearch is an Apache 2.0-licensed search and analytics suite that lets you ingest, search, visualize, and analyze scraped data at scale.
Search Engines
Headless browsers as a service
Platform or service · API / Self-hosted
Browserless provides headless browser infrastructure in Docker for web scraping and automation, with self-hosted or cloud deployment options.
Multithreaded Redis fork with higher throughput
Infrastructure · Self-hosted
KeyDB is a multithreaded Redis-compatible database for caching, data structures, and background-job infrastructure.
Caching Databases
Chrome DevTools Protocol Driver for Go
Library or tool · Library / Self-hosted
Stars: 13.3K · Code license: MIT Chromedp is a Go package for driving CDP-compatible browsers for scraping, testing, and automation without Selenium or WebDriver.
Browser Automation
Real-time OLAP database for event-driven data
Infrastructure · Self-hosted
Apache Druid is a real-time OLAP database that delivers sub-second queries on streaming and batch data for analytics at scale.
Analytics Databases
Simple, fast distributed memory caching system
Infrastructure · Self-hosted
Memcached is a free, high-performance distributed memory caching system that speeds up dynamic applications by caching scraped data in RAM.
Caching Databases
Unified Platform for AI and Data Pipelines
Library or tool · Self-hosted
Dagster is a data orchestration platform that helps teams build, schedule, and monitor reliable data pipelines with integrated lineage and observability.
Workflow Orchestration
jQuery-like HTML parsing for Go
Library or tool · Library / Self-hosted
Stars: 15K · Code license: BSD-3-Clause GoQuery brings jQuery-like syntax to Go for parsing and manipulating HTML documents, making web scraping in Go intuitive and efficient.
HTML Parsers
Real-time analytical database with MySQL syntax
Infrastructure · Self-hosted
Apache Doris is a real-time analytical database with MySQL-compatible syntax for high-concurrency queries over large scraped datasets.
Analytics Databases
AI-native vector database with built-in ML models
Infrastructure · Self-hosted
Weaviate is an AI-native vector database with built-in ML models for hybrid search, RAG, and semantic retrieval over scraped content.
Vector Databases
Build and Deploy Internal Tools at Scale
Library or tool · Self-hosted
Windmill is an open-source platform for building workflows, internal tools, and data pipelines with full code flexibility and a visual flow builder.
Workflow Orchestration
Time-series database with ultra-low latency
Infrastructure · Self-hosted
QuestDB is an open-source time-series database delivering ultra-low latency ingestion and queries for scraping metrics and monitoring data.
Analytics Databases
Durable Execution for Invincible Applications
Library or tool · Self-hosted
Temporal is a durable execution platform that makes distributed applications fault-tolerant by automatically capturing state and recovering from failures.
Workflow Orchestration
Python Batch Pipeline Framework by Spotify
Library or tool · Self-hosted
Luigi is a Python package by Spotify for building complex pipelines of batch jobs with dependency resolution, workflow management, and Hadoop support.
Workflow Orchestration
Relational SQL database with native JSON document support
Infrastructure · Self-hosted
PostgreSQL with JSONB combines relational SQL with flexible JSON document storage for scraped data that needs both structure and flexibility.
Databases
AI Browser Automation with Computer Vision
Library or tool · Framework / Self-hosted
Stars: 23.2K · Code license: AGPL-3.0 Skyvern automates browser-based workflows using LLMs and computer vision, replacing brittle scripts with AI that can adapt workflows to page layouts.
High-performance S3-compatible storage in Rust
Infrastructure · Self-hosted
RustFS is a high-performance distributed object storage system built in Rust with S3 compatibility and multi-cloud support.
Object Storage
TypeScript AI Agent Framework with Workflows
Library or tool · Framework / Self-hosted
Stars: 28.8K · Code license: Apache-2.0 (outside ee/) Mastra is an all-in-one TypeScript framework for building AI agents and workflows with RAG, memory, MCP support, evals, and an interactive playground.
AI Agent Tooling
Data Integration Platform with Source-Available Core
Platform or service · API / SaaS / Self-hosted
Airbyte is a data integration platform with a source-available core and connectors for moving data into warehouses and data lakes.
ETL Tools
