Build reliable web scrapers. Fast.
Library or tool · Framework / Self-hosted
Stars: 26.2K · Code license: Apache-2.0 Crawlee is a JavaScript and Python web scraping library that handles blocking, proxies, and browsers so you can build reliable crawlers fast.
Browser Automation +2 more
Workflow Orchestration for Python Data Pipelines
Library or tool · Self-hosted
Prefect is an open-source Python workflow orchestration framework for building resilient data pipelines with full observability and zero rewrites.
Workflow Orchestration
AI-Powered Web Scraping API for Agents
Library or tool · Library / Self-hosted
Stars: 31.8K · Code license: MIT ScrapeGraphAI extracts structured data from web pages using AI, handling proxies, rendering, and adapting to site changes automatically.
Elegant scraping and crawler framework for Golang
Library or tool · Framework / Self-hosted
Stars: 25.6K · Code license: Apache-2.0 Colly is a fast and elegant Go framework for building web scrapers, crawlers, and spiders with a clean, chainable API.
Scraping Frameworks
Typo-tolerant search engine with faceted filtering
Infrastructure · Self-hosted
Typesense is a fast, typo-tolerant search engine that delivers instant search-as-you-type results over your scraped and structured data.
Search Engines
AI-native embedding database with simple API
Infrastructure · Self-hosted
Chroma is an open-source embedding database with a simple API for storing and querying vectors, ideal for AI search over scraped content.
Vector Databases
Declarative Data Orchestration Platform
Library or tool · Self-hosted
Kestra is an open-source orchestration platform that uses declarative YAML to build, schedule, and monitor event-driven workflows at scale.
Workflow Orchestration
Data Validation Using Python Type Hints
Library or tool · Self-hosted
Pydantic is a Python data validation library, offering fast schema enforcement with type hints and a Rust-powered core engine.
Data Transformation & Validation
Distributed Task Queue for Python
Library or tool · Self-hosted
Celery is a simple, flexible, and reliable distributed task queue for processing vast amounts of messages with real-time and scheduled execution.
Task Queues
Flexible document database for any data shape
Infrastructure · API / SaaS / Self-hosted
MongoDB is a flexible document database that stores scraped data as JSON-like documents, making it easy to ingest varied web content.
NoSQL Databases
High-performance vector search engine in Rust
Infrastructure · Self-hosted
Stars: 35.1K · Code license: Apache-2.0 Qdrant is a high-performance vector search engine built in Rust for production-grade semantic search and RAG applications.
Vector Databases
Fast, flexible HTML parsing and manipulation for JS
Library or tool · Library / Self-hosted
Stars: 30.5K · Code license: MIT Cheerio is a JavaScript library for parsing and manipulating HTML and XML using a familiar jQuery-like syntax.
HTML Parsers
Browser Automation Framework and Ecosystem
Library or tool · Framework / Self-hosted
Stars: 34.6K · Code license: Apache-2.0 Selenium is the original open-source browser automation framework for driving web browsers natively, supporting all major browsers and programming languages.
Browser Automation
In-process SQL analytics, no server needed
Infrastructure · Self-hosted
DuckDB is an in-process analytical database that lets you query scraped CSV, Parquet, and JSON files with fast SQL, no server required.
Analytics Databases
Lightning-Fast DataFrame Library in Rust
Library or tool · Self-hosted
Polars is an open-source DataFrame library with a Rust-based query engine, columnar processing, lazy evaluation, and parallel execution.
Data Transformation & Validation
Open-source vector database for billion-scale AI
Infrastructure · Self-hosted
Milvus is an open-source vector database that handles billion-scale embeddings with GPU acceleration for AI-powered search and retrieval.
Vector Databases
Programmatically Author, Schedule, and Monitor Workflows
Library or tool · Self-hosted
Apache Airflow is an open-source platform for programmatically authoring, scheduling, and monitoring data pipelines and batch workflows.
Workflow Orchestration
Real-time analytics database for petabyte-scale data
Infrastructure · Self-hosted
ClickHouse is a fast, open-source columnar database that runs analytical queries on billions of scraped records in milliseconds.
Analytics Databases
Move and sync scraped datasets across cloud storage
Infrastructure · CLI / Self-hosted
Stars: 60.3K · Code license: MIT Rclone is a command-line data movement tool that syncs, copies, and mounts files across cloud storage providers for moving scraped datasets between systems.
Data Movement
Lightning-fast search engine through configured exporters or custom item pipelines
Infrastructure · Self-hosted
Meilisearch is a fast, typo-tolerant search engine that indexes scraped data and delivers instant search-as-you-type results.
Search Engines
Open-Source LLM-Friendly Web Crawler
Library or tool · Framework / Self-hosted
Stars: 85.6K · Code license: Apache-2.0 Crawl4AI is an open-source Python crawler with asynchronous crawling, Markdown output and structured extraction for LLM and data workflows.
Fast, high-level web crawling framework for Python
Library or tool · Framework / Self-hosted
Stars: 64.8K · Code license: BSD-3-Clause Scrapy is an open-source Python framework for building fast, scalable web crawlers that extract structured data from websites efficiently.
Scraping Frameworks
Archived community S3-compatible object store
Infrastructure · Self-hosted
MinIO is a self-hosted, S3-compatible object store for files, datasets, and media, with distributed storage and erasure coding. Its repository is archived.
Object Storage
In-memory data store for caching and queues
Infrastructure · Self-hosted
Redis is a blazing-fast in-memory data store used to cache scraped results, manage job queues, and power real-time data pipelines.
Caching Databases
Distributed RESTful search and analytics engine
Infrastructure · Self-hosted
Stars: 78.3K · Code license: AGPL-3.0-only OR SSPL-1.0 OR Elastic-2.0 (default source headers) Elasticsearch is a distributed search and analytics engine that lets you index, query, and analyze large volumes of scraped data in real time.
Search Engines
Open-Source AI Browser Automation Platform
Library or tool · Self-hosted
Browser Use provides open-source libraries for AI-assisted browser automation. Browser Use Cloud is a separate hosted service.
Web Data API for AI Applications
Platform or service · API / Self-hosted
Firecrawl is an open-source web crawling and scraping API that turns entire websites into clean, LLM-ready markdown or structured data for AI applications.
Cross-Browser Automation for Modern Web Apps
Library or tool · Library / Self-hosted
Stars: 97.8K · Code license: Apache-2.0 Playwright is an open-source framework for reliable end-to-end testing and browser automation across Chromium, Firefox, and WebKit with a single API.
Browser Automation +1 more
JavaScript API for Chrome and Firefox Automation
Library or tool · Library / Self-hosted
Stars: 95.7K · Code license: Apache-2.0 Puppeteer is a JavaScript library that provides a high-level API to control Chrome or Firefox for web scraping, testing, and browser automation tasks.
Browser Automation +1 more
AI Workflow Automation for Technical Teams
Platform or service · API / SaaS / Self-hosted
n8n is a workflow automation platform that combines AI capabilities with business process automation, offering code flexibility with no-code speed.
Workflow Orchestration
