All tools

Every tool in the directory - scraping, anti-blocking, data processing, databases, infrastructure, and utilities.

rclone

Move and sync scraped datasets across cloud storage

Infrastructure · CLI / Self-hosted

Stars: 60.3K
Code license: MIT

Rclone is a command-line data movement tool that syncs, copies, and mounts files across cloud storage providers for moving scraped datasets between systems.

Data Movement

Meilisearch

Lightning-fast search engine through configured exporters or custom item pipelines

Infrastructure · Self-hosted

Meilisearch is a fast, typo-tolerant search engine that indexes scraped data and delivers instant search-as-you-type results.

Search Engines

Crawl4AI

Open-Source LLM-Friendly Web Crawler

Library or tool · Framework / Self-hosted

Stars: 85.6K
Code license: Apache-2.0

Crawl4AI is an open-source Python crawler with asynchronous crawling, Markdown output and structured extraction for LLM and data workflows.

AI Web Scraping

+1 more

Scrapy

Fast, high-level web crawling framework for Python

Library or tool · Framework / Self-hosted

Stars: 64.8K
Code license: BSD-3-Clause

Scrapy is an open-source Python framework for building fast, scalable web crawlers that extract structured data from websites efficiently.

Scraping Frameworks

MinIO

Archived community S3-compatible object store

Infrastructure · Self-hosted

MinIO is a self-hosted, S3-compatible object store for files, datasets, and media, with distributed storage and erasure coding. Its repository is archived.

Object Storage

Redis

In-memory data store for caching and queues

Infrastructure · Self-hosted

Redis is a blazing-fast in-memory data store used to cache scraped results, manage job queues, and power real-time data pipelines.

Caching Databases

Elasticsearch

Distributed RESTful search and analytics engine

Infrastructure · Self-hosted

Stars: 78.3K
Code license: AGPL-3.0-only OR SSPL-1.0 OR Elastic-2.0 (default source headers)

Elasticsearch is a distributed search and analytics engine that lets you index, query, and analyze large volumes of scraped data in real time.

Search Engines

Browser Use

Open-Source AI Browser Automation Platform

Library or tool · Self-hosted

Browser Use provides open-source libraries for AI-assisted browser automation. Browser Use Cloud is a separate hosted service.

AI Web Scraping

+1 more

Firecrawl

Web Data API for AI Applications

Platform or service · API / Self-hosted

Firecrawl is an open-source web crawling and scraping API that turns entire websites into clean, LLM-ready markdown or structured data for AI applications.

AI Web Scraping

+1 more

Playwright

Cross-Browser Automation for Modern Web Apps

Library or tool · Library / Self-hosted

Stars: 97.8K
Code license: Apache-2.0

Playwright is an open-source framework for reliable end-to-end testing and browser automation across Chromium, Firefox, and WebKit with a single API.

Browser Automation

+1 more

Puppeteer

JavaScript API for Chrome and Firefox Automation

Library or tool · Library / Self-hosted

Stars: 95.7K
Code license: Apache-2.0

Puppeteer is a JavaScript library that provides a high-level API to control Chrome or Firefox for web scraping, testing, and browser automation tasks.

Browser Automation

+1 more

n8n

AI Workflow Automation for Technical Teams

Platform or service · API / SaaS / Self-hosted

n8n is a workflow automation platform that combines AI capabilities with business process automation, offering code flexibility with no-code speed.

Workflow Orchestration

rclone

Move and sync scraped datasets across cloud storage

Infrastructure · CLI / Self-hosted

Stars: 60.3K
Code license: MIT

Rclone is a command-line data movement tool that syncs, copies, and mounts files across cloud storage providers for moving scraped datasets between systems.

Data Movement

Meilisearch

Lightning-fast search engine through configured exporters or custom item pipelines

Infrastructure · Self-hosted

Meilisearch is a fast, typo-tolerant search engine that indexes scraped data and delivers instant search-as-you-type results.

Search Engines

Crawl4AI

Open-Source LLM-Friendly Web Crawler

Library or tool · Framework / Self-hosted

Stars: 85.6K
Code license: Apache-2.0

Crawl4AI is an open-source Python crawler with asynchronous crawling, Markdown output and structured extraction for LLM and data workflows.

AI Web Scraping

+1 more

Scrapy

Fast, high-level web crawling framework for Python

Library or tool · Framework / Self-hosted

Stars: 64.8K
Code license: BSD-3-Clause

Scrapy is an open-source Python framework for building fast, scalable web crawlers that extract structured data from websites efficiently.

Scraping Frameworks

MinIO

Archived community S3-compatible object store

Infrastructure · Self-hosted

MinIO is a self-hosted, S3-compatible object store for files, datasets, and media, with distributed storage and erasure coding. Its repository is archived.

Object Storage

Redis

In-memory data store for caching and queues

Infrastructure · Self-hosted

Redis is a blazing-fast in-memory data store used to cache scraped results, manage job queues, and power real-time data pipelines.

Caching Databases

Elasticsearch

Distributed RESTful search and analytics engine

Infrastructure · Self-hosted

Stars: 78.3K
Code license: AGPL-3.0-only OR SSPL-1.0 OR Elastic-2.0 (default source headers)

Elasticsearch is a distributed search and analytics engine that lets you index, query, and analyze large volumes of scraped data in real time.

Search Engines

Browser Use

Open-Source AI Browser Automation Platform

Library or tool · Self-hosted

Browser Use provides open-source libraries for AI-assisted browser automation. Browser Use Cloud is a separate hosted service.

AI Web Scraping

+1 more

Firecrawl

Web Data API for AI Applications

Platform or service · API / Self-hosted

Firecrawl is an open-source web crawling and scraping API that turns entire websites into clean, LLM-ready markdown or structured data for AI applications.

AI Web Scraping

+1 more

Playwright

Cross-Browser Automation for Modern Web Apps

Library or tool · Library / Self-hosted

Stars: 97.8K
Code license: Apache-2.0

Playwright is an open-source framework for reliable end-to-end testing and browser automation across Chromium, Firefox, and WebKit with a single API.

Browser Automation

+1 more

Puppeteer

JavaScript API for Chrome and Firefox Automation

Library or tool · Library / Self-hosted

Stars: 95.7K
Code license: Apache-2.0

Puppeteer is a JavaScript library that provides a high-level API to control Chrome or Firefox for web scraping, testing, and browser automation tasks.

Browser Automation

+1 more

n8n

AI Workflow Automation for Technical Teams

Platform or service · API / SaaS / Self-hosted

n8n is a workflow automation platform that combines AI capabilities with business process automation, offering code flexibility with no-code speed.

Workflow Orchestration