Best Free Web Scraping Tools

Weaviate

AI-native vector database with built-in ML models

Stars
15.5K
Forks
0

Weaviate is an AI-native vector database with built-in ML models for hybrid search, RAG, and semantic retrieval over scraped content.

Windmill

Build and Deploy Internal Tools at Scale

Stars
15.6K
Forks
869
Last commit
5 months ago

Windmill is an open-source platform for building workflows, internal tools, and data pipelines with full code flexibility and a visual flow builder.

QuestDB

Time-series database with ultra-low latency

Stars
16.5K
Forks
0

QuestDB is an open-source time-series database delivering ultra-low latency ingestion and queries for scraping metrics and monitoring data.

Temporal

Durable Execution for Invincible Applications

Stars
17.9K
Forks
1.3K
Last commit
5 months ago

Temporal is a durable execution platform that makes distributed applications fault-tolerant by automatically capturing state and recovering from failures.

Luigi

Python Batch Pipeline Framework by Spotify

Stars
18.6K
Forks
2.4K
Last commit
5 months ago

Luigi is a Python package by Spotify for building complex pipelines of batch jobs with dependency resolution, workflow management, and Hadoop support.

PostgreSQL + JSONB

Relational SQL database with native JSON document support

Stars
19.7K
Forks
5.4K
Last commit
5 months ago

PostgreSQL with JSONB combines relational SQL with flexible JSON document storage for scraped data that needs both structure and flexibility.

Skyvern

AI Browser Automation with Computer Vision

Stars
20.2K
Forks
1.8K
Last commit
5 months ago

Skyvern automates browser-based workflows using LLMs and computer vision, replacing brittle scripts with AI that adapts to any webpage automatically.

RustFS

High-performance S3-compatible storage in Rust

Stars
20.2K
Forks
868
Last commit
5 months ago

RustFS is a high-performance distributed object storage system built in Rust with S3 compatibility and multi-cloud support.

Mastra

TypeScript AI Agent Framework with Workflows

Stars
20.4K
Forks
1.5K
Last commit
5 months ago

Mastra is an all-in-one TypeScript framework for building AI agents and workflows with RAG, memory, MCP support, evals, and an interactive playground.

Airbyte

Open-Source Data Integration Platform

Stars
20.5K
Forks
0

Airbyte is an open-source ELT platform with 600+ connectors that moves data from APIs, databases, and files to warehouses and data lakes at scale.

Crawlee

Build reliable web scrapers. Fast.

Stars
21.3K
Forks
1.2K
Last commit
5 months ago

Crawlee is a JavaScript and Python web scraping library that handles blocking, proxies, and browsers so you can build reliable crawlers fast.

Prefect

Workflow Orchestration for Python Data Pipelines

Stars
21.4K
Forks
2.1K
Last commit
5 months ago

Prefect is an open-source Python workflow orchestration framework for building resilient data pipelines with full observability and zero rewrites.

ScrapeGraphAI

AI-Powered Web Scraping API for Agents

Stars
22.4K
Forks
1.9K
Last commit
5 months ago

ScrapeGraphAI extracts structured data from any website using AI, handling proxies, rendering, and adapting to site changes automatically.

Colly

Elegant scraping and crawler framework for Golang

Stars
25K
Forks
1.8K
Last commit
6 months ago

Colly is a fast and elegant Go framework for building web scrapers, crawlers, and spiders with a clean, chainable API.

Typesense

Typo-tolerant search with sub-50ms responses

Stars
25.1K
Forks
849
Last commit
5 months ago

Typesense is a fast, typo-tolerant search engine that delivers instant search-as-you-type results over your scraped and structured data.

Chroma

AI-native embedding database with simple API

Stars
25.7K
Forks
0

Chroma is an open-source embedding database with a simple API for storing and querying vectors, ideal for AI search over scraped content.

Kestra

Declarative Data Orchestration Platform

Stars
26.3K
Forks
2.5K
Last commit
5 months ago

Kestra is an open-source orchestration platform that uses declarative YAML to build, schedule, and monitor event-driven workflows at scale.

Pydantic

Data Validation Using Python Type Hints

Stars
26.5K
Forks
0

Pydantic is the most widely used Python data validation library, offering fast schema enforcement with type hints and a Rust-powered core engine.

Celery

Distributed Task Queue for Python

Stars
27.9K
Forks
4.9K
Last commit
5 months ago

Celery is a simple, flexible, and reliable distributed task queue for processing vast amounts of messages with real-time and scheduled execution.

MongoDB

Flexible document database for any data shape

Stars
28K
Forks
5.7K
Last commit
5 months ago

MongoDB is a flexible document database that stores scraped data as JSON-like documents, making it easy to ingest varied web content.

Qdrant

High-performance vector search engine in Rust

Stars
28.4K
Forks
0

Qdrant is a high-performance vector search engine built in Rust for production-grade semantic search and RAG applications.

Cheerio

Fast, flexible HTML parsing and manipulation for JS

Stars
30K
Forks
1.7K
Last commit
5 months ago

Cheerio is the industry-standard JavaScript library for parsing and manipulating HTML and XML using a familiar jQuery-like syntax.

Selenium

Browser Automation Framework and Ecosystem

Stars
33.9K
Forks
8.6K
Last commit
5 months ago

Selenium is the original open-source browser automation framework for driving web browsers natively, supporting all major browsers and programming languages.

DuckDB

In-process SQL analytics, no server needed

Stars
35.6K
Forks
2.9K
Last commit
5 months ago

DuckDB is an in-process analytical database that lets you query scraped CSV, Parquet, and JSON files with fast SQL, no server required.

Polars

Lightning-Fast DataFrame Library in Rust

Stars
37.1K
Forks
0

Polars is an open-source, Rust-powered DataFrame library delivering 10-100x faster data processing than pandas with lazy evaluation and parallelism.

Milvus

Open-source vector database for billion-scale AI

Stars
42.4K
Forks
0

Milvus is an open-source vector database that handles billion-scale embeddings with GPU acceleration for AI-powered search and retrieval.

Apache Airflow

Programmatically Author, Schedule, and Monitor Workflows

Stars
44K
Forks
16.3K
Last commit
5 months ago

Apache Airflow is an open-source platform for programmatically authoring, scheduling, and monitoring data pipelines and batch workflows.

ClickHouse

Real-time analytics database for petabyte-scale data

Stars
45.4K
Forks
8K
Last commit
5 months ago

ClickHouse is a fast, open-source columnar database that runs analytical queries on billions of scraped records in milliseconds.

rclone

Move and sync scraped datasets across cloud storage

Stars
55.1K
Forks
0

Rclone is a command-line data movement tool that syncs, copies, and mounts files across cloud storage providers for moving scraped datasets between systems.

Meilisearch

Lightning-fast search engine with zero config

Stars
55.5K
Forks
2.3K
Last commit
5 months ago

Meilisearch is a fast, typo-tolerant search engine that indexes scraped data and delivers instant search-as-you-type results.

Crawl4AI

Open-Source LLM-Friendly Web Crawler

Stars
58.9K
Forks
6K
Last commit
5 months ago

Crawl4AI is an open-source web crawler and scraper built for LLMs, AI agents, and data pipelines with blazing-fast async crawling and AI-ready output.

Scrapy

Fast, high-level web crawling framework for Python

Stars
59.5K
Forks
11.2K
Last commit
5 months ago

Scrapy is an open-source Python framework for building fast, scalable web crawlers that extract structured data from websites efficiently.

MinIO

S3-compatible object storage you can self-host

Stars
59.9K
Forks
6.9K
Last commit
6 months ago

MinIO is a high-performance, S3-compatible object store for storing scraped files, datasets, and media at exascale.

Redis

In-memory data store for caching and queues

Stars
72.6K
Forks
24.5K
Last commit
5 months ago

Redis is a blazing-fast in-memory data store used to cache scraped results, manage job queues, and power real-time data pipelines.

Elasticsearch

Distributed RESTful search and analytics engine

Stars
75.9K
Forks
25.8K
Last commit
5 months ago

Elasticsearch is a distributed search and analytics engine that lets you index, query, and analyze large volumes of scraped data in real time.

Browser Use

Open-Source AI Browser Automation Platform

Stars
76.4K
Forks
9.1K
Last commit
5 months ago

Browser Use is an open-source platform that lets AI agents automate web tasks using natural language, with stealth browsers and custom models for scale.