Web scraping tools directory

Apache Airflow

Programmatically Author, Schedule, and Monitor Workflows

Stars
47.1K
Forks
16.3K
Last recorded push
8 months ago

Apache Airflow is an open-source platform for programmatically authoring, scheduling, and monitoring data pipelines and batch workflows.

ClickHouse

Real-time analytics database for petabyte-scale data

Stars
50.3K
Forks
8K
Last recorded push
8 months ago

ClickHouse is a fast, open-source columnar database that runs analytical queries on billions of scraped records in milliseconds.

rclone

Move and sync scraped datasets across cloud storage

Stars
60.3K
Forks
0

Rclone is a command-line data movement tool that syncs, copies, and mounts files across cloud storage providers for moving scraped datasets between systems.

Meilisearch

Lightning-fast search engine through configured exporters or custom item pipelines

Stars
59.5K
Forks
2.3K
Last recorded push
8 months ago

Meilisearch is a fast, typo-tolerant search engine that indexes scraped data and delivers instant search-as-you-type results.

MinIO

Archived community S3-compatible object store

Stars
61.3K
Forks
6.9K
Last recorded push
9 months ago

MinIO is a self-hosted, S3-compatible object store for files, datasets, and media, with distributed storage and erasure coding. Its repository is archived.

Redis

In-memory data store for caching and queues

Stars
76.7K
Forks
24.5K
Last recorded push
8 months ago

Redis is a blazing-fast in-memory data store used to cache scraped results, manage job queues, and power real-time data pipelines.

Elasticsearch

Distributed RESTful search and analytics engine

Stars
78.2K
Forks
25.8K
Last recorded push
8 months ago

Elasticsearch is a distributed search and analytics engine that lets you index, query, and analyze large volumes of scraped data in real time.

Browser Use

Open-Source AI Browser Automation Platform

Stars
117.7K
Forks
9.1K
Last recorded push
8 months ago

Browser Use provides open-source libraries for AI-assisted browser automation. Browser Use Cloud is a separate hosted service.

Firecrawl

Web Data API for AI Applications

Stars
190.5K
Forks
5.8K
Last recorded push
8 months ago

Firecrawl is an open-source web crawling and scraping API that turns entire websites into clean, LLM-ready markdown or structured data for AI applications.

Playwright

Cross-Browser Automation for Modern Web Apps

Stars
97.5K
Forks
5K
Last recorded push
8 months ago

Playwright is an open-source framework for reliable end-to-end testing and browser automation across Chromium, Firefox, and WebKit with a single API.

Puppeteer

JavaScript API for Chrome and Firefox Automation

Stars
95.7K
Forks
9.4K
Last recorded push
8 months ago

Puppeteer is a JavaScript library that provides a high-level API to control Chrome or Firefox for web scraping, testing, and browser automation tasks.

n8n

AI Workflow Automation for Technical Teams

Stars
207K
Forks
54K
Last recorded push
8 months ago

n8n is a workflow automation platform that combines AI capabilities with business process automation, offering code flexibility with no-code speed.

Find your next web scraping tool.

Discover and compare scraping APIs, no-code tools and open-source libraries.

Apache Airflow

ClickHouse

rclone

Meilisearch

MinIO

Redis

Elasticsearch

Browser Use

Firecrawl

Playwright

Puppeteer

n8n

Apache Airflow

ClickHouse

rclone

Meilisearch

MinIO

Redis

Elasticsearch

Browser Use

Firecrawl

Playwright

Puppeteer

n8n