Web scraping tools directory

Manticore Search

Open-source search database with SQL and HTTP APIs

Infrastructure · Self-hosted

Manticore Search is an open-source search database for full-text search and analytical queries through SQL and HTTP APIs.

Search Engines

OpenSearch

Open-source search and analytics suite

Infrastructure · Self-hosted

OpenSearch is an Apache 2.0-licensed search and analytics suite that lets you ingest, search, visualize, and analyze scraped data at scale.

Search Engines

Browserless

Headless browsers as a service

Platform or service · API / Self-hosted

Browserless provides headless browser infrastructure in Docker for web scraping and automation, with self-hosted or cloud deployment options.

Headless Browsers

+1 more

KeyDB

Multithreaded Redis fork with higher throughput

Infrastructure · Self-hosted

KeyDB is a multithreaded Redis-compatible database for caching, data structures, and background-job infrastructure.

Caching Databases

Chromedp

Chrome DevTools Protocol Driver for Go

Library or tool · Library / Self-hosted

Stars: 13.3K
Code license: MIT

Chromedp is a Go package for driving CDP-compatible browsers for scraping, testing, and automation without Selenium or WebDriver.

Browser Automation

Apache Druid

Real-time OLAP database for event-driven data

Infrastructure · Self-hosted

Apache Druid is a real-time OLAP database that delivers sub-second queries on streaming and batch data for analytics at scale.

Analytics Databases

Memcached

Simple, fast distributed memory caching system

Infrastructure · Self-hosted

Memcached is a free, high-performance distributed memory caching system that speeds up dynamic applications by caching scraped data in RAM.

Caching Databases

Dagster

Unified Platform for AI and Data Pipelines

Library or tool · Self-hosted

Dagster is a data orchestration platform that helps teams build, schedule, and monitor reliable data pipelines with integrated lineage and observability.

Workflow Orchestration

Goquery

jQuery-like HTML parsing for Go

Library or tool · Library / Self-hosted

Stars: 15K
Code license: BSD-3-Clause

GoQuery brings jQuery-like syntax to Go for parsing and manipulating HTML documents, making web scraping in Go intuitive and efficient.

HTML Parsers

Apache Doris

Real-time analytical database with MySQL syntax

Infrastructure · Self-hosted

Apache Doris is a real-time analytical database with MySQL-compatible syntax for high-concurrency queries over large scraped datasets.

Analytics Databases

Weaviate

AI-native vector database with built-in ML models

Infrastructure · Self-hosted

Weaviate is an AI-native vector database with built-in ML models for hybrid search, RAG, and semantic retrieval over scraped content.

Vector Databases

Windmill

Build and Deploy Internal Tools at Scale

Library or tool · Self-hosted

Windmill is an open-source platform for building workflows, internal tools, and data pipelines with full code flexibility and a visual flow builder.

Workflow Orchestration

QuestDB

Time-series database with ultra-low latency

Infrastructure · Self-hosted

QuestDB is an open-source time-series database delivering ultra-low latency ingestion and queries for scraping metrics and monitoring data.

Analytics Databases

Temporal

Durable Execution for Invincible Applications

Library or tool · Self-hosted

Temporal is a durable execution platform that makes distributed applications fault-tolerant by automatically capturing state and recovering from failures.

Workflow Orchestration

Luigi

Python Batch Pipeline Framework by Spotify

Library or tool · Self-hosted

Luigi is a Python package by Spotify for building complex pipelines of batch jobs with dependency resolution, workflow management, and Hadoop support.

Workflow Orchestration

PostgreSQL + JSONB

Relational SQL database with native JSON document support

Infrastructure · Self-hosted

PostgreSQL with JSONB combines relational SQL with flexible JSON document storage for scraped data that needs both structure and flexibility.

Databases

Skyvern

AI Browser Automation with Computer Vision

Library or tool · Framework / Self-hosted

Stars: 23.2K
Code license: AGPL-3.0

Skyvern automates browser-based workflows using LLMs and computer vision, replacing brittle scripts with AI that can adapt workflows to page layouts.

AI Web Scraping

+1 more

RustFS

High-performance S3-compatible storage in Rust

Infrastructure · Self-hosted

RustFS is a high-performance distributed object storage system built in Rust with S3 compatibility and multi-cloud support.

Object Storage

Mastra

TypeScript AI Agent Framework with Workflows

Library or tool · Framework / Self-hosted

Stars: 28.8K
Code license: Apache-2.0 (outside ee/)

Mastra is an all-in-one TypeScript framework for building AI agents and workflows with RAG, memory, MCP support, evals, and an interactive playground.

AI Agent Tooling

Airbyte

Data Integration Platform with Source-Available Core

Platform or service · API / SaaS / Self-hosted

Airbyte is a data integration platform with a source-available core and connectors for moving data into warehouses and data lakes.

ETL Tools

Crawlee

Build reliable web scrapers. Fast.

Library or tool · Framework / Self-hosted

Stars: 26.2K
Code license: Apache-2.0

Crawlee is a JavaScript and Python web scraping library that handles blocking, proxies, and browsers so you can build reliable crawlers fast.

Browser Automation

+2 more

Prefect

Workflow Orchestration for Python Data Pipelines

Library or tool · Self-hosted

Prefect is an open-source Python workflow orchestration framework for building resilient data pipelines with full observability and zero rewrites.

Workflow Orchestration

ScrapeGraphAI

AI-Powered Web Scraping API for Agents

Library or tool · Library / Self-hosted

Stars: 31.8K
Code license: MIT

ScrapeGraphAI extracts structured data from web pages using AI, handling proxies, rendering, and adapting to site changes automatically.

AI Web Scraping

+1 more

Colly

Elegant scraping and crawler framework for Golang

Library or tool · Framework / Self-hosted

Stars: 25.6K
Code license: Apache-2.0

Colly is a fast and elegant Go framework for building web scrapers, crawlers, and spiders with a clean, chainable API.

Scraping Frameworks

Typesense

Typo-tolerant search engine with faceted filtering

Infrastructure · Self-hosted

Typesense is a fast, typo-tolerant search engine that delivers instant search-as-you-type results over your scraped and structured data.

Search Engines

Chroma

AI-native embedding database with simple API

Infrastructure · Self-hosted

Chroma is an open-source embedding database with a simple API for storing and querying vectors, ideal for AI search over scraped content.

Vector Databases

Kestra

Declarative Data Orchestration Platform

Library or tool · Self-hosted

Kestra is an open-source orchestration platform that uses declarative YAML to build, schedule, and monitor event-driven workflows at scale.

Workflow Orchestration

Pydantic

Data Validation Using Python Type Hints

Library or tool · Self-hosted

Pydantic is a Python data validation library, offering fast schema enforcement with type hints and a Rust-powered core engine.

Data Transformation & Validation

Celery

Distributed Task Queue for Python

Library or tool · Self-hosted

Celery is a simple, flexible, and reliable distributed task queue for processing vast amounts of messages with real-time and scheduled execution.

Task Queues

MongoDB

Flexible document database for any data shape

Infrastructure · API / SaaS / Self-hosted

MongoDB is a flexible document database that stores scraped data as JSON-like documents, making it easy to ingest varied web content.

NoSQL Databases

Qdrant

High-performance vector search engine in Rust

Infrastructure · Self-hosted

Stars: 35.1K
Code license: Apache-2.0

Qdrant is a high-performance vector search engine built in Rust for production-grade semantic search and RAG applications.

Vector Databases

Cheerio

Fast, flexible HTML parsing and manipulation for JS

Library or tool · Library / Self-hosted

Stars: 30.5K
Code license: MIT

Cheerio is a JavaScript library for parsing and manipulating HTML and XML using a familiar jQuery-like syntax.

HTML Parsers

Selenium

Browser Automation Framework and Ecosystem

Library or tool · Framework / Self-hosted

Stars: 34.6K
Code license: Apache-2.0

Selenium is the original open-source browser automation framework for driving web browsers natively, supporting all major browsers and programming languages.

Browser Automation