All tools

rclone

Move and sync scraped datasets across cloud storage

Stars
55.1K
Forks
0

Rclone is a command-line data movement tool that syncs, copies, and mounts files across cloud storage providers for moving scraped datasets between systems.

Meilisearch

Lightning-fast search engine with zero config

Stars
55.5K
Forks
2.3K
Last commit
5 months ago

Meilisearch is a fast, typo-tolerant search engine that indexes scraped data and delivers instant search-as-you-type results.

Crawl4AI

Open-Source LLM-Friendly Web Crawler

Stars
58.9K
Forks
6K
Last commit
5 months ago

Crawl4AI is an open-source web crawler and scraper built for LLMs, AI agents, and data pipelines with blazing-fast async crawling and AI-ready output.

Scrapy

Fast, high-level web crawling framework for Python

Stars
59.5K
Forks
11.2K
Last commit
5 months ago

Scrapy is an open-source Python framework for building fast, scalable web crawlers that extract structured data from websites efficiently.

MinIO

S3-compatible object storage you can self-host

Stars
59.9K
Forks
6.9K
Last commit
6 months ago

MinIO is a high-performance, S3-compatible object store for storing scraped files, datasets, and media at exascale.

Redis

In-memory data store for caching and queues

Stars
72.6K
Forks
24.5K
Last commit
5 months ago

Redis is a blazing-fast in-memory data store used to cache scraped results, manage job queues, and power real-time data pipelines.

Elasticsearch

Distributed RESTful search and analytics engine

Stars
75.9K
Forks
25.8K
Last commit
5 months ago

Elasticsearch is a distributed search and analytics engine that lets you index, query, and analyze large volumes of scraped data in real time.

Browser Use

Open-Source AI Browser Automation Platform

Stars
76.4K
Forks
9.1K
Last commit
5 months ago

Browser Use is an open-source platform that lets AI agents automate web tasks using natural language, with stealth browsers and custom models for scale.

Firecrawl

Web Data API for AI Applications

Stars
77K
Forks
5.8K
Last commit
5 months ago

Firecrawl is an open-source web crawling and scraping API that turns entire websites into clean, LLM-ready markdown or structured data for AI applications.

Playwright

Cross-Browser Automation for Modern Web Apps

Stars
81.6K
Forks
5K
Last commit
5 months ago

Playwright is an open-source framework for reliable end-to-end testing and browser automation across Chromium, Firefox, and WebKit with a single API.

Puppeteer

JavaScript API for Chrome and Firefox Automation

Stars
93.4K
Forks
9.4K
Last commit
5 months ago

Puppeteer is a JavaScript library that provides a high-level API to control Chrome or Firefox for web scraping, testing, and browser automation tasks.

n8n

AI Workflow Automation for Technical Teams

Stars
170.9K
Forks
54K
Last commit
5 months ago

n8n is a workflow automation platform that combines AI capabilities with business process automation, offering code flexibility with no-code speed.

Every tool in the directory - scraping, anti-blocking, data processing, databases, infrastructure, and utilities.