All tools

Fly.io

Deploy containers globally on edge servers

Stars
1.6K
Forks
0

Fly.io lets you deploy containerized scrapers and services on edge servers worldwide with persistent storage and instant scaling.

Patchright

Undetected Playwright fork for bot evasion

Stars
2.2K
Forks
88
Last commit
5 months ago

Patchright is a patched fork of Playwright that evades bot detection while maintaining full API compatibility for seamless migration.

Meltano

CLI-First ELT Engine Built on Singer

Stars
2.3K
Forks
0

Meltano is an open-source, CLI-first ELT platform with 600+ connectors, GitOps workflows, and native dbt integration for data pipeline management.

Scrapoxy

End-of-life proxy orchestration project

Stars
2.4K
Forks
0

Scrapoxy is an open-source proxy orchestration project that reached End of Life as announced on scrapoxy.io, so it is historical context rather than a new-deployment choice.

BeeAI Framework

Production-Ready AI Agent Framework

Stars
3.1K
Forks
400
Last commit
5 months ago

BeeAI Framework is an open-source platform for building production-ready multi-agent systems in Python and TypeScript with MCP and workflow support.

Nutch

Highly extensible and scalable web crawler

Stars
3.1K
Forks
1.3K
Last commit
5 months ago

Apache Nutch is a highly extensible, production-ready web crawler built on Hadoop for large-scale batch crawling and data acquisition tasks.

Heritrix

Web-scale archival web crawler

Stars
3.2K
Forks
782
Last commit
6 months ago

Heritrix is the Internet Archive's open-source, extensible, archival-quality web crawler designed for large-scale web preservation and data collection.

Nodriver

Undetected browser automation without Selenium

Stars
3.5K
Forks
341
Last commit
8 months ago

Nodriver is the successor to undetected-chromedriver, providing fast CDP-based browser automation with built-in anti-detection and no Selenium dependency.

parse5

HTML5 spec-compliant parser for Node.js

Stars
3.9K
Forks
251
Last commit
5 months ago

parse5 is the fastest spec-compliant HTML5 parser for Node.js, parsing HTML exactly as modern browsers do.

Algolia

Managed search and AI retrieval platform

Stars
4K
Forks
557
Last commit
5 months ago

Algolia is a managed search platform that delivers fast, relevant search results over scraped and structured content for web applications.

Inngest

Durable Functions for Workflows and Queues

Stars
4.7K
Forks
240
Last commit
5 months ago

Inngest replaces traditional queues with durable step functions, enabling reliable multi-step workflows on serverless, servers, or the edge.

htmlparser2

The fast and forgiving HTML/XML parser

Stars
4.8K
Forks
394
Last commit
5 months ago

htmlparser2 is the fastest Node.js HTML and XML parser, offering a streaming, event-driven API with a forgiving approach to malformed markup.

dlt (data load tool)

Python Library for Data Loading

Stars
4.8K
Forks
0

dlt is a lightweight, open-source Python library for loading data from any source into well-structured datasets with automatic schema inference.

Mechanicalsoup

Automate website interaction with Python

Stars
4.8K
Forks
385
Last commit
6 months ago

MechanicalSoup is a Python library for automating website interaction, handling cookies, redirects, forms, and links without a browser.

Camoufox

Anti-detect browser with fingerprint injection

Stars
4.9K
Forks
459
Last commit
5 months ago

Camoufox is an open-source anti-detect browser based on Firefox with robust fingerprint injection to evade bot detection systems.

Scrapy Redis

Redis-based distributed components for Scrapy

Stars
5.6K
Forks
1.6K
Last commit
2 years ago

Scrapy-Redis provides Redis-backed components for Scrapy, enabling distributed crawling with shared request queues and item pipelines.

Huey

Lightweight Python Task Queue

Stars
5.9K
Forks
392
Last commit
6 months ago

Huey is a lightweight Python task queue with a clean API supporting Redis, SQLite, or in-memory storage for background job processing.

Apache Hudi

Lakehouse Platform for Incremental Processing

Stars
6.1K
Forks
0

Apache Hudi is an open data lakehouse platform that enables efficient record-level upserts, incremental processing, and ACID transactions on data lakes.

LLM-Scraper

Turn Any Webpage into Structured Data with LLMs

Stars
6.2K
Forks
369
Last commit
7 months ago

LLM Scraper is a TypeScript library that extracts structured data from any webpage using LLMs with Zod schemas, Playwright, and streaming support.

Rod

High-Level DevTools Driver for Go

Stars
6.6K
Forks
437
Last commit
9 months ago

Rod is a Go library for web automation and scraping that drives browsers via the DevTools Protocol with a simple, chainable API.

DynamoDB

AWS managed serverless NoSQL database

Stars
7.6K
Forks
1.5K
Last commit
7 months ago

Amazon DynamoDB is a fully managed, serverless NoSQL database with single-digit millisecond performance for high-throughput scraping workloads.

BullMQ

Message Queue and Background Jobs for Node.js

Stars
8.3K
Forks
547
Last commit
5 months ago

BullMQ is an open-source Redis-based message queue trusted by thousands of companies processing billions of jobs daily across Node.js, Python, and more.

Apache Iceberg

Open Table Format for Huge Analytic Datasets

Stars
8.5K
Forks
0

Apache Iceberg is a high-performance open table format enabling multi-engine analytics with schema evolution, time travel, and hidden partitioning.

Delta Lake

Open-Source Lakehouse Storage Framework

Stars
8.6K
Forks
0

Delta Lake is an open-source storage framework that brings ACID transactions, schema evolution, and time travel to data lakes at petabyte scale.

LanceDB

Serverless vector database, no server management

Stars
8.6K
Forks
0

LanceDB is a serverless vector database built in Rust that runs embedded or in the cloud with native DataFrame integration for AI workloads.

Mage AI

AI-Powered Data Pipeline Builder

Stars
8.6K
Forks
905
Last commit
5 months ago

Mage AI is a data engineering platform that lets you build, deploy, and run scalable data pipelines with AI assistance across Python, SQL, and R.

Scrapling

Adaptive web scraping for the modern web

Stars
8.8K
Forks
537
Last commit
5 months ago

Scrapling is an adaptive Python scraping framework that learns from website changes, handling everything from single requests to full-scale crawls.

Cassandra

Distributed NoSQL with high availability at scale

Stars
9.6K
Forks
3.8K
Last commit
5 months ago

Apache Cassandra is a distributed NoSQL database that handles massive amounts of scraped data with high availability and linear scalability.

WebdriverIO

Next-Gen Browser and Mobile Automation for Node.js

Stars
9.7K
Forks
2.7K
Last commit
5 months ago

WebdriverIO is an all-in-one Node.js automation framework for web and mobile testing with smart selectors, component testing, and extensive plugins.

TestCafe

WebDriver-Free End-to-End Browser Testing

Stars
9.9K
Forks
677
Last commit
5 months ago

TestCafe is a free, open-source end-to-end testing framework that works across browsers without WebDriver, with intuitive syntax and built-in waiting.

RQ (Redis Queue)

Simple Job Queues for Python

Stars
10.6K
Forks
1.5K
Last commit
5 months ago

RQ is a lightweight Python library for queueing jobs and processing them in the background with Redis-backed workers.

dbt

Transform Data in Your Warehouse with SQL

Stars
10.9K
Forks
0

dbt empowers data teams to transform, test, and document data directly in the warehouse using SQL, with version control and governance built in.

Great Expectations

Data Quality Testing and Validation Platform

Stars
11.1K
Forks
0

Great Expectations helps data teams validate, document, and monitor data quality across pipelines with automated testing and observability.

StarRocks

Sub-second analytics for enterprise-scale data

Stars
11.3K
Forks
2.3K
Last commit
5 months ago

StarRocks is an open-source analytical database that delivers sub-second query latency for complex joins and aggregations at enterprise scale.

Manticore Search

Fast open-source search database, 2.8x faster

Stars
11.6K
Forks
622
Last commit
5 months ago

Manticore Search is a fast, open-source search database that delivers 2.8x faster performance than Elasticsearch for full-text and analytical queries.

OpenSearch

Open-source search and analytics suite

Stars
12.3K
Forks
2.4K
Last commit
5 months ago

OpenSearch is an Apache 2.0-licensed search and analytics suite that lets you ingest, search, visualize, and analyze scraped data at scale.