All tools

Every tool in the directory - scraping, anti-blocking, data processing, databases, infrastructure, and utilities.

Fly.io

Deploy containers globally on edge servers

Infrastructure · API / SaaS

Fly.io lets you deploy containerized scrapers and services on edge servers worldwide with persistent storage and instant scaling.

Cloud Compute

Patchright

Undetected Playwright fork for bot evasion

Library or tool · Self-hosted

Patchright is a patched fork of Playwright that evades bot detection with a Playwright-style API and documented compatibility limits.

Browser Automation

+1 more

Meltano

CLI-First ELT Engine Built on Singer

Library or tool · Self-hosted

Meltano is an open-source, CLI-first ELT platform with 600+ connectors, GitOps workflows, and native dbt integration for data pipeline management.

ETL Tools

Scrapoxy

End-of-life proxy orchestration project

Library or tool · Retired · Self-hosted

Scrapoxy is an open-source proxy orchestration project that reached End of Life as announced on scrapoxy.io, so it is historical context rather than a new-deployment choice.

Proxy Services

BeeAI Framework

AI agents and multi-agent workflows

Library or tool · Framework / Self-hosted

Stars: 3.4K
Code license: Apache-2.0

Python and TypeScript framework for building AI agents and multi-agent workflows with model providers, memory, tools and MCP integrations.

AI Agent Tooling

Nutch

Highly extensible and scalable web crawler

Library or tool · Self-hosted

Apache Nutch is a highly extensible, production-ready web crawler built on Hadoop for large-scale batch crawling and data acquisition tasks.

Large-Scale & Distributed Crawling

Heritrix

Web-scale archival web crawler

Library or tool · Self-hosted

Heritrix is the Internet Archive's open-source, extensible, archival-quality web crawler designed for large-scale web preservation and data collection.

Large-Scale & Distributed Crawling

Nodriver

Undetected browser automation without Selenium

Library or tool · Library / Self-hosted

Stars: 4.8K
Code license: AGPL-3.0

Nodriver is the successor to undetected-chromedriver, providing fast CDP-based browser automation with built-in anti-detection and no Selenium dependency.

Browser Automation

+1 more

parse5

HTML5 spec-compliant parser for Node.js

Library or tool · Library / Self-hosted

Stars: 3.9K
Code license: MIT

parse5 is a spec-compliant HTML5 parser for Node.js, parsing HTML exactly as modern browsers do.

HTML Parsers

Algolia

Managed search and AI retrieval platform

Infrastructure · API / SaaS

Algolia is a managed search platform that delivers fast, relevant search results over scraped and structured content for web applications.

Search Engines

Inngest

Durable Functions for Workflows and Queues

Library or tool · Self-hosted

Inngest replaces traditional queues with durable step functions, enabling reliable multi-step workflows on serverless, servers, or the edge.

Workflow Orchestration

htmlparser2

The fast and forgiving HTML/XML parser

Library or tool · Library / Self-hosted

Stars: 4.8K
Code license: MIT

htmlparser2 is a streaming Node.js HTML and XML parser, offering a streaming, event-driven API with a forgiving approach to malformed markup.

HTML Parsers

dlt (data load tool)

Python Library for Data Loading

Library or tool · Self-hosted

dlt is a lightweight, open-source Python library for loading data from any source into well-structured datasets with automatic schema inference.

Data Transformation & Validation

+1 more

Mechanicalsoup

Automate website interaction with Python

Library or tool · Library / Self-hosted

Stars: 4.9K
Code license: MIT

MechanicalSoup is a Python library for automating website interaction, handling cookies, redirects, forms, and links without a browser.

Scraping Frameworks

Camoufox

Anti-detect browser with fingerprint injection

Library or tool · Self-hosted

Stars: 12.6K
Code license: MPL-2.0

Camoufox is an open-source anti-detect browser based on Firefox with robust fingerprint injection to evade bot detection systems.

Browser Automation

+1 more

Scrapy Redis

Redis-based distributed components for Scrapy

Library or tool · Self-hosted

Scrapy-Redis provides Redis-backed components for Scrapy, enabling distributed crawling with shared request queues and item pipelines.

Large-Scale & Distributed Crawling

Huey

Lightweight Python Task Queue

Library or tool · Self-hosted

Huey is a lightweight Python task queue with a clean API supporting Redis, SQLite, or in-memory storage for background job processing.

Task Queues

Apache Hudi

Lakehouse Platform for Incremental Processing

Infrastructure · Self-hosted

Apache Hudi is an open data lakehouse platform that enables efficient record-level upserts, incremental processing, and ACID transactions on data lakes.

Lakehouse & Table Formats

LLM-Scraper

Structured Web Extraction with LLMs

Library or tool · Library / Self-hosted

Stars: 6.9K
Code license: MIT

LLM Scraper is a TypeScript library that extracts structured data from web pages using LLMs with Zod schemas, Playwright, and streaming support.

AI Web Scraping

+1 more

Rod

High-Level DevTools Driver for Go

Library or tool · Self-hosted

Rod is a Go library for web automation and scraping that drives browsers via the DevTools Protocol with a simple, chainable API.

Browser Automation

DynamoDB

AWS managed serverless NoSQL database

Infrastructure · API / SaaS

Amazon DynamoDB is a fully managed, serverless NoSQL database with single-digit millisecond performance for high-throughput scraping workloads.

NoSQL Databases

BullMQ

Message Queue and Background Jobs for Node.js

Library or tool · Self-hosted

BullMQ is a task queue for background jobs. Its JavaScript/TypeScript package supports Redis; version 6.3.11 also includes an optional PostgreSQL backend. Backend support must be checked for each language binding.

Task Queues

Apache Iceberg

Open Table Format for Huge Analytic Datasets

Infrastructure · Self-hosted

Apache Iceberg is a high-performance open table format enabling multi-engine analytics with schema evolution, time travel, and hidden partitioning.

Lakehouse & Table Formats

Delta Lake

Open-Source Lakehouse Storage Framework

Infrastructure · Self-hosted

Delta Lake is an open-source storage framework that brings ACID transactions, schema evolution, and time travel to data lakes at petabyte scale.

Lakehouse & Table Formats

LanceDB

Serverless vector database, no server management

Infrastructure · Self-hosted

LanceDB is a serverless vector database built in Rust that runs embedded or in the cloud with native DataFrame integration for AI workloads.

Vector Databases

Mage AI

AI-Powered Data Pipeline Builder

Library or tool · Self-hosted

Mage AI is a data engineering platform that lets you build, deploy, and run scalable data pipelines with AI assistance across Python, SQL, and R.

Workflow Orchestration

Scrapling

Adaptive web scraping for the modern web

Library or tool · Framework / Self-hosted

Stars: 87.9K
Code license: BSD-3-Clause

Scrapling is an adaptive Python scraping framework that learns from website changes, handling everything from single requests to full-scale crawls.

Scraping Frameworks

Cassandra

Distributed NoSQL with high availability at scale

Infrastructure · Self-hosted

Apache Cassandra is a distributed NoSQL database that handles massive amounts of scraped data with high availability and linear scalability.

NoSQL Databases

WebdriverIO

Next-Gen Browser and Mobile Automation for Node.js

Library or tool · Self-hosted

WebdriverIO is an all-in-one Node.js automation framework for web and mobile testing with smart selectors, component testing, and extensive plugins.

Browser Automation

+1 more

TestCafe

WebDriver-Free End-to-End Browser Testing

Library or tool · Self-hosted

TestCafe is a free, open-source end-to-end testing framework that works across browsers without WebDriver, with intuitive syntax and built-in waiting.

Browser Automation

RQ (Redis Queue)

Simple Job Queues for Python

Library or tool · Self-hosted

RQ is a lightweight Python library for queueing jobs and processing them in the background with Redis-backed workers.

Task Queues

dbt

Transform Data in Your Warehouse with SQL

Library or tool · Self-hosted

dbt empowers data teams to transform, test, and document data directly in the warehouse using SQL, with version control and governance built in.

Data Transformation & Validation

Great Expectations

Data Quality Testing and Validation Platform

Library or tool · Self-hosted

Great Expectations helps data teams validate, document, and monitor data quality across pipelines with automated testing and observability.

Data Transformation & Validation

StarRocks

Sub-second analytics for enterprise-scale data

Infrastructure · Self-hosted

StarRocks is an open-source analytical database that delivers sub-second query latency for complex joins and aggregations at enterprise scale.

Analytics Databases

Manticore Search

Open-source search database with SQL and HTTP APIs

Infrastructure · Self-hosted

Manticore Search is an open-source search database for full-text search and analytical queries through SQL and HTTP APIs.

Search Engines

OpenSearch

Open-source search and analytics suite

Infrastructure · Self-hosted

OpenSearch is an Apache 2.0-licensed search and analytics suite that lets you ingest, search, visualize, and analyze scraped data at scale.

Search Engines

Fly.io

Deploy containers globally on edge servers

Infrastructure · API / SaaS

Fly.io lets you deploy containerized scrapers and services on edge servers worldwide with persistent storage and instant scaling.

Cloud Compute

Patchright

Undetected Playwright fork for bot evasion

Library or tool · Self-hosted

Patchright is a patched fork of Playwright that evades bot detection with a Playwright-style API and documented compatibility limits.

Browser Automation

+1 more

Meltano

CLI-First ELT Engine Built on Singer

Library or tool · Self-hosted

Meltano is an open-source, CLI-first ELT platform with 600+ connectors, GitOps workflows, and native dbt integration for data pipeline management.

ETL Tools

Scrapoxy

End-of-life proxy orchestration project

Library or tool · Retired · Self-hosted

Scrapoxy is an open-source proxy orchestration project that reached End of Life as announced on scrapoxy.io, so it is historical context rather than a new-deployment choice.

Proxy Services

BeeAI Framework

AI agents and multi-agent workflows

Library or tool · Framework / Self-hosted

Stars: 3.4K
Code license: Apache-2.0

Python and TypeScript framework for building AI agents and multi-agent workflows with model providers, memory, tools and MCP integrations.

AI Agent Tooling

Nutch

Highly extensible and scalable web crawler

Library or tool · Self-hosted

Apache Nutch is a highly extensible, production-ready web crawler built on Hadoop for large-scale batch crawling and data acquisition tasks.

Large-Scale & Distributed Crawling

Heritrix

Web-scale archival web crawler

Library or tool · Self-hosted

Heritrix is the Internet Archive's open-source, extensible, archival-quality web crawler designed for large-scale web preservation and data collection.

Large-Scale & Distributed Crawling

Nodriver

Undetected browser automation without Selenium

Library or tool · Library / Self-hosted

Stars: 4.8K
Code license: AGPL-3.0

Nodriver is the successor to undetected-chromedriver, providing fast CDP-based browser automation with built-in anti-detection and no Selenium dependency.

Browser Automation

+1 more

parse5

HTML5 spec-compliant parser for Node.js

Library or tool · Library / Self-hosted

Stars: 3.9K
Code license: MIT

parse5 is a spec-compliant HTML5 parser for Node.js, parsing HTML exactly as modern browsers do.

HTML Parsers

Algolia

Managed search and AI retrieval platform

Infrastructure · API / SaaS

Algolia is a managed search platform that delivers fast, relevant search results over scraped and structured content for web applications.

Search Engines

Inngest

Durable Functions for Workflows and Queues

Library or tool · Self-hosted

Inngest replaces traditional queues with durable step functions, enabling reliable multi-step workflows on serverless, servers, or the edge.

Workflow Orchestration

htmlparser2

The fast and forgiving HTML/XML parser

Library or tool · Library / Self-hosted

Stars: 4.8K
Code license: MIT

htmlparser2 is a streaming Node.js HTML and XML parser, offering a streaming, event-driven API with a forgiving approach to malformed markup.

HTML Parsers

dlt (data load tool)

Python Library for Data Loading

Library or tool · Self-hosted

dlt is a lightweight, open-source Python library for loading data from any source into well-structured datasets with automatic schema inference.

Data Transformation & Validation

+1 more

Mechanicalsoup

Automate website interaction with Python

Library or tool · Library / Self-hosted

Stars: 4.9K
Code license: MIT

MechanicalSoup is a Python library for automating website interaction, handling cookies, redirects, forms, and links without a browser.

Scraping Frameworks

Camoufox

Anti-detect browser with fingerprint injection

Library or tool · Self-hosted

Stars: 12.6K
Code license: MPL-2.0

Camoufox is an open-source anti-detect browser based on Firefox with robust fingerprint injection to evade bot detection systems.

Browser Automation

+1 more

Scrapy Redis

Redis-based distributed components for Scrapy

Library or tool · Self-hosted

Scrapy-Redis provides Redis-backed components for Scrapy, enabling distributed crawling with shared request queues and item pipelines.

Large-Scale & Distributed Crawling

Huey

Lightweight Python Task Queue

Library or tool · Self-hosted

Huey is a lightweight Python task queue with a clean API supporting Redis, SQLite, or in-memory storage for background job processing.

Task Queues

Apache Hudi

Lakehouse Platform for Incremental Processing

Infrastructure · Self-hosted

Apache Hudi is an open data lakehouse platform that enables efficient record-level upserts, incremental processing, and ACID transactions on data lakes.

Lakehouse & Table Formats

LLM-Scraper

Structured Web Extraction with LLMs

Library or tool · Library / Self-hosted

Stars: 6.9K
Code license: MIT

LLM Scraper is a TypeScript library that extracts structured data from web pages using LLMs with Zod schemas, Playwright, and streaming support.

AI Web Scraping

+1 more

Rod

High-Level DevTools Driver for Go

Library or tool · Self-hosted

Rod is a Go library for web automation and scraping that drives browsers via the DevTools Protocol with a simple, chainable API.

Browser Automation

DynamoDB

AWS managed serverless NoSQL database

Infrastructure · API / SaaS

Amazon DynamoDB is a fully managed, serverless NoSQL database with single-digit millisecond performance for high-throughput scraping workloads.

NoSQL Databases

BullMQ

Message Queue and Background Jobs for Node.js

Library or tool · Self-hosted

Task Queues

Apache Iceberg

Open Table Format for Huge Analytic Datasets

Infrastructure · Self-hosted

Apache Iceberg is a high-performance open table format enabling multi-engine analytics with schema evolution, time travel, and hidden partitioning.

Lakehouse & Table Formats

Delta Lake

Open-Source Lakehouse Storage Framework

Infrastructure · Self-hosted

Delta Lake is an open-source storage framework that brings ACID transactions, schema evolution, and time travel to data lakes at petabyte scale.

Lakehouse & Table Formats

LanceDB

Serverless vector database, no server management

Infrastructure · Self-hosted

LanceDB is a serverless vector database built in Rust that runs embedded or in the cloud with native DataFrame integration for AI workloads.

Vector Databases

Mage AI

AI-Powered Data Pipeline Builder

Library or tool · Self-hosted

Mage AI is a data engineering platform that lets you build, deploy, and run scalable data pipelines with AI assistance across Python, SQL, and R.

Workflow Orchestration

Scrapling

Adaptive web scraping for the modern web

Library or tool · Framework / Self-hosted

Stars: 87.9K
Code license: BSD-3-Clause

Scrapling is an adaptive Python scraping framework that learns from website changes, handling everything from single requests to full-scale crawls.

Scraping Frameworks

Cassandra

Distributed NoSQL with high availability at scale

Infrastructure · Self-hosted

Apache Cassandra is a distributed NoSQL database that handles massive amounts of scraped data with high availability and linear scalability.

NoSQL Databases

WebdriverIO

Next-Gen Browser and Mobile Automation for Node.js

Library or tool · Self-hosted

WebdriverIO is an all-in-one Node.js automation framework for web and mobile testing with smart selectors, component testing, and extensive plugins.

Browser Automation

+1 more

TestCafe

WebDriver-Free End-to-End Browser Testing

Library or tool · Self-hosted

TestCafe is a free, open-source end-to-end testing framework that works across browsers without WebDriver, with intuitive syntax and built-in waiting.

Browser Automation

RQ (Redis Queue)

Simple Job Queues for Python

Library or tool · Self-hosted

RQ is a lightweight Python library for queueing jobs and processing them in the background with Redis-backed workers.

Task Queues

dbt

Transform Data in Your Warehouse with SQL

Library or tool · Self-hosted

dbt empowers data teams to transform, test, and document data directly in the warehouse using SQL, with version control and governance built in.

Data Transformation & Validation

Great Expectations

Data Quality Testing and Validation Platform

Library or tool · Self-hosted

Great Expectations helps data teams validate, document, and monitor data quality across pipelines with automated testing and observability.

Data Transformation & Validation

StarRocks

Sub-second analytics for enterprise-scale data

Infrastructure · Self-hosted

StarRocks is an open-source analytical database that delivers sub-second query latency for complex joins and aggregations at enterprise scale.

Analytics Databases

Manticore Search

Open-source search database with SQL and HTTP APIs

Infrastructure · Self-hosted

Manticore Search is an open-source search database for full-text search and analytical queries through SQL and HTTP APIs.

Search Engines

OpenSearch

Open-source search and analytics suite

Infrastructure · Self-hosted

OpenSearch is an Apache 2.0-licensed search and analytics suite that lets you ingest, search, visualize, and analyze scraped data at scale.

Search Engines