Open Source Web Scraping Tools

Apache Doris

Real-time analytical database with MySQL syntax

Infrastructure · Self-hosted

Apache Doris is a real-time analytical database with MySQL-compatible syntax for high-concurrency queries over large scraped datasets.

Analytics Databases

Weaviate

AI-native vector database with built-in ML models

Infrastructure · Self-hosted

Weaviate is an AI-native vector database with built-in ML models for hybrid search, RAG, and semantic retrieval over scraped content.

Vector Databases

Windmill

Build and Deploy Internal Tools at Scale

Library or tool · Self-hosted

Windmill is an open-source platform for building workflows, internal tools, and data pipelines with full code flexibility and a visual flow builder.

Workflow Orchestration

QuestDB

Time-series database with ultra-low latency

Infrastructure · Self-hosted

QuestDB is an open-source time-series database delivering ultra-low latency ingestion and queries for scraping metrics and monitoring data.

Analytics Databases

Temporal

Durable Execution for Invincible Applications

Library or tool · Self-hosted

Temporal is a durable execution platform that makes distributed applications fault-tolerant by automatically capturing state and recovering from failures.

Workflow Orchestration

Luigi

Python Batch Pipeline Framework by Spotify

Library or tool · Self-hosted

Luigi is a Python package by Spotify for building complex pipelines of batch jobs with dependency resolution, workflow management, and Hadoop support.

Workflow Orchestration

PostgreSQL + JSONB

Relational SQL database with native JSON document support

Infrastructure · Self-hosted

PostgreSQL with JSONB combines relational SQL with flexible JSON document storage for scraped data that needs both structure and flexibility.

Databases

Skyvern

AI Browser Automation with Computer Vision

Library or tool · Framework / Self-hosted

Stars: 23.2K
Code license: AGPL-3.0

Skyvern automates browser-based workflows using LLMs and computer vision, replacing brittle scripts with AI that can adapt workflows to page layouts.

AI Web Scraping

+1 more

RustFS

High-performance S3-compatible storage in Rust

Infrastructure · Self-hosted

RustFS is a high-performance distributed object storage system built in Rust with S3 compatibility and multi-cloud support.

Object Storage

Mastra

TypeScript AI Agent Framework with Workflows

Library or tool · Framework / Self-hosted

Stars: 28.8K
Code license: Apache-2.0 (outside ee/)

Mastra is an all-in-one TypeScript framework for building AI agents and workflows with RAG, memory, MCP support, evals, and an interactive playground.

AI Agent Tooling

Crawlee

Build reliable web scrapers. Fast.

Library or tool · Framework / Self-hosted

Stars: 26.2K
Code license: Apache-2.0

Crawlee is a JavaScript and Python web scraping library that handles blocking, proxies, and browsers so you can build reliable crawlers fast.

Browser Automation

+2 more

Prefect

Workflow Orchestration for Python Data Pipelines

Library or tool · Self-hosted

Prefect is an open-source Python workflow orchestration framework for building resilient data pipelines with full observability and zero rewrites.

Workflow Orchestration

ScrapeGraphAI

AI-Powered Web Scraping API for Agents

Library or tool · Library / Self-hosted

Stars: 31.8K
Code license: MIT

ScrapeGraphAI extracts structured data from web pages using AI, handling proxies, rendering, and adapting to site changes automatically.

AI Web Scraping

+1 more

Colly

Elegant scraping and crawler framework for Golang

Library or tool · Framework / Self-hosted

Stars: 25.6K
Code license: Apache-2.0

Colly is a fast and elegant Go framework for building web scrapers, crawlers, and spiders with a clean, chainable API.

Scraping Frameworks

Typesense

Typo-tolerant search engine with faceted filtering

Infrastructure · Self-hosted

Typesense is a fast, typo-tolerant search engine that delivers instant search-as-you-type results over your scraped and structured data.

Search Engines

Chroma

AI-native embedding database with simple API

Infrastructure · Self-hosted

Chroma is an open-source embedding database with a simple API for storing and querying vectors, ideal for AI search over scraped content.

Vector Databases

Kestra

Declarative Data Orchestration Platform

Library or tool · Self-hosted

Kestra is an open-source orchestration platform that uses declarative YAML to build, schedule, and monitor event-driven workflows at scale.

Workflow Orchestration

Pydantic

Data Validation Using Python Type Hints

Library or tool · Self-hosted

Pydantic is a Python data validation library, offering fast schema enforcement with type hints and a Rust-powered core engine.

Data Transformation & Validation

Celery

Distributed Task Queue for Python

Library or tool · Self-hosted

Celery is a simple, flexible, and reliable distributed task queue for processing vast amounts of messages with real-time and scheduled execution.

Task Queues

Qdrant

High-performance vector search engine in Rust

Infrastructure · Self-hosted

Stars: 35.1K
Code license: Apache-2.0

Qdrant is a high-performance vector search engine built in Rust for production-grade semantic search and RAG applications.

Vector Databases

Cheerio

Fast, flexible HTML parsing and manipulation for JS

Library or tool · Library / Self-hosted

Stars: 30.5K
Code license: MIT

Cheerio is a JavaScript library for parsing and manipulating HTML and XML using a familiar jQuery-like syntax.

HTML Parsers

Selenium

Browser Automation Framework and Ecosystem

Library or tool · Framework / Self-hosted

Stars: 34.6K
Code license: Apache-2.0

Selenium is the original open-source browser automation framework for driving web browsers natively, supporting all major browsers and programming languages.

Browser Automation

DuckDB

In-process SQL analytics, no server needed

Infrastructure · Self-hosted

DuckDB is an in-process analytical database that lets you query scraped CSV, Parquet, and JSON files with fast SQL, no server required.

Analytics Databases

Polars

Lightning-Fast DataFrame Library in Rust

Library or tool · Self-hosted

Polars is an open-source DataFrame library with a Rust-based query engine, columnar processing, lazy evaluation, and parallel execution.

Data Transformation & Validation

Milvus

Open-source vector database for billion-scale AI

Infrastructure · Self-hosted

Milvus is an open-source vector database that handles billion-scale embeddings with GPU acceleration for AI-powered search and retrieval.

Vector Databases

Apache Airflow

Programmatically Author, Schedule, and Monitor Workflows

Library or tool · Self-hosted

Apache Airflow is an open-source platform for programmatically authoring, scheduling, and monitoring data pipelines and batch workflows.

Workflow Orchestration

ClickHouse

Real-time analytics database for petabyte-scale data

Infrastructure · Self-hosted

ClickHouse is a fast, open-source columnar database that runs analytical queries on billions of scraped records in milliseconds.

Analytics Databases

rclone

Move and sync scraped datasets across cloud storage

Infrastructure · CLI / Self-hosted

Stars: 60.3K
Code license: MIT

Rclone is a command-line data movement tool that syncs, copies, and mounts files across cloud storage providers for moving scraped datasets between systems.

Data Movement

Meilisearch

Lightning-fast search engine through configured exporters or custom item pipelines

Infrastructure · Self-hosted

Meilisearch is a fast, typo-tolerant search engine that indexes scraped data and delivers instant search-as-you-type results.

Search Engines

Crawl4AI

Open-Source LLM-Friendly Web Crawler

Library or tool · Framework / Self-hosted

Stars: 85.6K
Code license: Apache-2.0

Crawl4AI is an open-source Python crawler with asynchronous crawling, Markdown output and structured extraction for LLM and data workflows.

AI Web Scraping

+1 more

Scrapy

Fast, high-level web crawling framework for Python

Library or tool · Framework / Self-hosted

Stars: 64.8K
Code license: BSD-3-Clause

Scrapy is an open-source Python framework for building fast, scalable web crawlers that extract structured data from websites efficiently.

Scraping Frameworks

MinIO

Archived community S3-compatible object store

Infrastructure · Self-hosted

MinIO is a self-hosted, S3-compatible object store for files, datasets, and media, with distributed storage and erasure coding. Its repository is archived.

Object Storage

Redis

In-memory data store for caching and queues

Infrastructure · Self-hosted

Redis is a blazing-fast in-memory data store used to cache scraped results, manage job queues, and power real-time data pipelines.

Caching Databases

Elasticsearch

Distributed RESTful search and analytics engine

Infrastructure · Self-hosted

Stars: 78.3K
Code license: AGPL-3.0-only OR SSPL-1.0 OR Elastic-2.0 (default source headers)

Elasticsearch is a distributed search and analytics engine that lets you index, query, and analyze large volumes of scraped data in real time.

Search Engines

Browser Use

Open-Source AI Browser Automation Platform

Library or tool · Self-hosted

Browser Use provides open-source libraries for AI-assisted browser automation. Browser Use Cloud is a separate hosted service.

AI Web Scraping

+1 more

Firecrawl

Web Data API for AI Applications

Platform or service · API / Self-hosted

Firecrawl is an open-source web crawling and scraping API that turns entire websites into clean, LLM-ready markdown or structured data for AI applications.

AI Web Scraping

+1 more

Apache Doris

Real-time analytical database with MySQL syntax

Infrastructure · Self-hosted

Apache Doris is a real-time analytical database with MySQL-compatible syntax for high-concurrency queries over large scraped datasets.

Analytics Databases

Weaviate

AI-native vector database with built-in ML models

Infrastructure · Self-hosted

Weaviate is an AI-native vector database with built-in ML models for hybrid search, RAG, and semantic retrieval over scraped content.

Vector Databases

Windmill

Build and Deploy Internal Tools at Scale

Library or tool · Self-hosted

Windmill is an open-source platform for building workflows, internal tools, and data pipelines with full code flexibility and a visual flow builder.

Workflow Orchestration

QuestDB

Time-series database with ultra-low latency

Infrastructure · Self-hosted

QuestDB is an open-source time-series database delivering ultra-low latency ingestion and queries for scraping metrics and monitoring data.

Analytics Databases

Temporal

Durable Execution for Invincible Applications

Library or tool · Self-hosted

Temporal is a durable execution platform that makes distributed applications fault-tolerant by automatically capturing state and recovering from failures.

Workflow Orchestration

Luigi

Python Batch Pipeline Framework by Spotify

Library or tool · Self-hosted

Luigi is a Python package by Spotify for building complex pipelines of batch jobs with dependency resolution, workflow management, and Hadoop support.

Workflow Orchestration

PostgreSQL + JSONB

Relational SQL database with native JSON document support

Infrastructure · Self-hosted

PostgreSQL with JSONB combines relational SQL with flexible JSON document storage for scraped data that needs both structure and flexibility.

Databases

Skyvern

AI Browser Automation with Computer Vision

Library or tool · Framework / Self-hosted

Stars: 23.2K
Code license: AGPL-3.0

Skyvern automates browser-based workflows using LLMs and computer vision, replacing brittle scripts with AI that can adapt workflows to page layouts.

AI Web Scraping

+1 more

RustFS

High-performance S3-compatible storage in Rust

Infrastructure · Self-hosted

RustFS is a high-performance distributed object storage system built in Rust with S3 compatibility and multi-cloud support.

Object Storage

Mastra

TypeScript AI Agent Framework with Workflows

Library or tool · Framework / Self-hosted

Stars: 28.8K
Code license: Apache-2.0 (outside ee/)

Mastra is an all-in-one TypeScript framework for building AI agents and workflows with RAG, memory, MCP support, evals, and an interactive playground.

AI Agent Tooling

Crawlee

Build reliable web scrapers. Fast.

Library or tool · Framework / Self-hosted

Stars: 26.2K
Code license: Apache-2.0

Crawlee is a JavaScript and Python web scraping library that handles blocking, proxies, and browsers so you can build reliable crawlers fast.

Browser Automation

+2 more

Prefect

Workflow Orchestration for Python Data Pipelines

Library or tool · Self-hosted

Prefect is an open-source Python workflow orchestration framework for building resilient data pipelines with full observability and zero rewrites.

Workflow Orchestration

ScrapeGraphAI

AI-Powered Web Scraping API for Agents

Library or tool · Library / Self-hosted

Stars: 31.8K
Code license: MIT

ScrapeGraphAI extracts structured data from web pages using AI, handling proxies, rendering, and adapting to site changes automatically.

AI Web Scraping

+1 more

Colly

Elegant scraping and crawler framework for Golang

Library or tool · Framework / Self-hosted

Stars: 25.6K
Code license: Apache-2.0

Colly is a fast and elegant Go framework for building web scrapers, crawlers, and spiders with a clean, chainable API.

Scraping Frameworks

Typesense

Typo-tolerant search engine with faceted filtering

Infrastructure · Self-hosted

Typesense is a fast, typo-tolerant search engine that delivers instant search-as-you-type results over your scraped and structured data.

Search Engines

Chroma

AI-native embedding database with simple API

Infrastructure · Self-hosted

Chroma is an open-source embedding database with a simple API for storing and querying vectors, ideal for AI search over scraped content.

Vector Databases

Kestra

Declarative Data Orchestration Platform

Library or tool · Self-hosted

Kestra is an open-source orchestration platform that uses declarative YAML to build, schedule, and monitor event-driven workflows at scale.

Workflow Orchestration

Pydantic

Data Validation Using Python Type Hints

Library or tool · Self-hosted

Pydantic is a Python data validation library, offering fast schema enforcement with type hints and a Rust-powered core engine.

Data Transformation & Validation

Celery

Distributed Task Queue for Python

Library or tool · Self-hosted

Celery is a simple, flexible, and reliable distributed task queue for processing vast amounts of messages with real-time and scheduled execution.

Task Queues

Qdrant

High-performance vector search engine in Rust

Infrastructure · Self-hosted

Stars: 35.1K
Code license: Apache-2.0

Qdrant is a high-performance vector search engine built in Rust for production-grade semantic search and RAG applications.

Vector Databases

Cheerio

Fast, flexible HTML parsing and manipulation for JS

Library or tool · Library / Self-hosted

Stars: 30.5K
Code license: MIT

Cheerio is a JavaScript library for parsing and manipulating HTML and XML using a familiar jQuery-like syntax.

HTML Parsers

Selenium

Browser Automation Framework and Ecosystem

Library or tool · Framework / Self-hosted

Stars: 34.6K
Code license: Apache-2.0

Selenium is the original open-source browser automation framework for driving web browsers natively, supporting all major browsers and programming languages.

Browser Automation

DuckDB

In-process SQL analytics, no server needed

Infrastructure · Self-hosted

DuckDB is an in-process analytical database that lets you query scraped CSV, Parquet, and JSON files with fast SQL, no server required.

Analytics Databases

Polars

Lightning-Fast DataFrame Library in Rust

Library or tool · Self-hosted

Polars is an open-source DataFrame library with a Rust-based query engine, columnar processing, lazy evaluation, and parallel execution.

Data Transformation & Validation

Milvus

Open-source vector database for billion-scale AI

Infrastructure · Self-hosted

Milvus is an open-source vector database that handles billion-scale embeddings with GPU acceleration for AI-powered search and retrieval.

Vector Databases

Apache Airflow

Programmatically Author, Schedule, and Monitor Workflows

Library or tool · Self-hosted

Apache Airflow is an open-source platform for programmatically authoring, scheduling, and monitoring data pipelines and batch workflows.

Workflow Orchestration

ClickHouse

Real-time analytics database for petabyte-scale data

Infrastructure · Self-hosted

ClickHouse is a fast, open-source columnar database that runs analytical queries on billions of scraped records in milliseconds.

Analytics Databases

rclone

Move and sync scraped datasets across cloud storage

Infrastructure · CLI / Self-hosted

Stars: 60.3K
Code license: MIT

Rclone is a command-line data movement tool that syncs, copies, and mounts files across cloud storage providers for moving scraped datasets between systems.

Data Movement

Meilisearch

Lightning-fast search engine through configured exporters or custom item pipelines

Infrastructure · Self-hosted

Meilisearch is a fast, typo-tolerant search engine that indexes scraped data and delivers instant search-as-you-type results.

Search Engines

Crawl4AI

Open-Source LLM-Friendly Web Crawler

Library or tool · Framework / Self-hosted

Stars: 85.6K
Code license: Apache-2.0

Crawl4AI is an open-source Python crawler with asynchronous crawling, Markdown output and structured extraction for LLM and data workflows.

AI Web Scraping

+1 more

Scrapy

Fast, high-level web crawling framework for Python

Library or tool · Framework / Self-hosted

Stars: 64.8K
Code license: BSD-3-Clause

Scrapy is an open-source Python framework for building fast, scalable web crawlers that extract structured data from websites efficiently.

Scraping Frameworks

MinIO

Archived community S3-compatible object store

Infrastructure · Self-hosted

MinIO is a self-hosted, S3-compatible object store for files, datasets, and media, with distributed storage and erasure coding. Its repository is archived.

Object Storage

Redis

In-memory data store for caching and queues

Infrastructure · Self-hosted

Redis is a blazing-fast in-memory data store used to cache scraped results, manage job queues, and power real-time data pipelines.

Caching Databases

Elasticsearch

Distributed RESTful search and analytics engine

Infrastructure · Self-hosted

Stars: 78.3K
Code license: AGPL-3.0-only OR SSPL-1.0 OR Elastic-2.0 (default source headers)

Elasticsearch is a distributed search and analytics engine that lets you index, query, and analyze large volumes of scraped data in real time.

Search Engines

Browser Use

Open-Source AI Browser Automation Platform

Library or tool · Self-hosted

Browser Use provides open-source libraries for AI-assisted browser automation. Browser Use Cloud is a separate hosted service.

AI Web Scraping

+1 more

Firecrawl

Web Data API for AI Applications

Platform or service · API / Self-hosted

Firecrawl is an open-source web crawling and scraping API that turns entire websites into clean, LLM-ready markdown or structured data for AI applications.

AI Web Scraping

+1 more

Open Source Web Scraping Tools

Browse Open Source web scraping tools. Compare features, limitations, and value to find the right fit for your budget.