Web scraping tools directory

Cloudflare Workers

Run serverless code at the edge worldwide

Cloud Compute

Cloudflare Workers lets you run JavaScript and TypeScript at the edge with ultra-low latency for lightweight scrapers, proxies, and APIs.

ScraperAPI

Scale data collection with a simple API

Stars
2
Forks
1
Last recorded push
2 years ago

ScraperAPI handles proxy rotation, browsers, and CAPTCHAs so developers can scrape any public page with a single API call at scale.

ScrapingBee

Web scraping API with proxy and browser handling

Stars
132
Forks
4
Last recorded push
8 months ago

ScrapingBee is a web scraping API that handles headless browsers, rotates proxies, and offers AI-powered extraction so you can focus on your data.

CapSolver

AI-powered automatic CAPTCHA solving service

Stars
74
Forks
10
Last recorded push
last year

CapSolver is an AI-powered CAPTCHA solver that automatically bypasses reCAPTCHA, Cloudflare, AWS WAF, and other challenges for automation.

Oxylabs

Premium proxy service for data at scale

Stars
105
Forks
5
Last recorded push
9 months ago

Oxylabs offers 175M+ residential and datacenter proxies with web scraping APIs and browser automation tools for large-scale public data extraction.

SerpAPI

Real-Time Google Search Results API

Stars
176
Forks
11
Last recorded push
10 months ago

SerpAPI is a real-time API that delivers structured Google search results, handling proxies, captchas, and rich data parsing automatically.

BeautifulSoup

Python library for parsing HTML and XML

Stars
226
Forks
57
Last recorded push
3 years ago

Beautiful Soup is a Python library for pulling data out of HTML and XML files, providing Pythonic idioms for navigating and searching parse trees.

Modal

Serverless Python platform for AI and data teams

Stars
523
Forks
0

Modal is a serverless Python platform that runs scraping, AI, and data workloads at scale with configurable containers and autoscaling.

2Captcha

CAPTCHA solving service with API integration

Stars
803
Forks
126
Last recorded push
8 months ago

2Captcha is a CAPTCHA solving service that bypasses reCAPTCHA, Cloudflare Turnstile, hCaptcha, and other challenges via a simple API.

Zendriver

Async-first undetected browser automation

Stars
1.5K
Forks
75
Last recorded push
10 months ago

Zendriver is a fast, async-first undetected browser automation framework based on nodriver with Docker support for scalable scraping.

Frontera

Scalable crawl frontier framework

Stars
1.3K
Forks
217
Last recorded push
last year

Frontera is a Python crawl frontier framework for managing when and what to crawl, enabling web crawlers of any scale with Scrapy integration.

Fly.io

Deploy containers globally on edge servers

Stars
1.7K
Forks
0

Fly.io lets you deploy containerized scrapers and services on edge servers worldwide with persistent storage and instant scaling.

Meltano

CLI-First ELT Engine Built on Singer

Stars
2.7K
Forks
0

Meltano is an open-source, CLI-first ELT platform with 600+ connectors, GitOps workflows, and native dbt integration for data pipeline management.

Scrapoxy

End-of-life proxy orchestration project

Stars
2.4K
Forks
0

Scrapoxy is an open-source proxy orchestration project that reached End of Life as announced on scrapoxy.io, so it is historical context rather than a new-deployment choice.

Nutch

Highly extensible and scalable web crawler

Stars
3.3K
Forks
1.3K
Last recorded push
8 months ago

Apache Nutch is a highly extensible, production-ready web crawler built on Hadoop for large-scale batch crawling and data acquisition tasks.

Algolia

Managed search and AI retrieval platform

Stars
4.1K
Forks
557
Last recorded push
8 months ago

Algolia is a managed search platform that delivers fast, relevant search results over scraped and structured content for web applications.

Inngest

Durable Functions for Workflows and Queues

Stars
5.9K
Forks
240
Last recorded push
8 months ago

Inngest replaces traditional queues with durable step functions, enabling reliable multi-step workflows on serverless, servers, or the edge.

htmlparser2

The fast and forgiving HTML/XML parser

Stars
4.8K
Forks
394
Last recorded push
8 months ago

htmlparser2 is a streaming Node.js HTML and XML parser, offering a streaming, event-driven API with a forgiving approach to malformed markup.

dlt (data load tool)

Python Library for Data Loading

Stars
6K
Forks
0

dlt is a lightweight, open-source Python library for loading data from any source into well-structured datasets with automatic schema inference.

Mechanicalsoup

Automate website interaction with Python

Stars
4.9K
Forks
385
Last recorded push
9 months ago

MechanicalSoup is a Python library for automating website interaction, handling cookies, redirects, forms, and links without a browser.

Huey

Lightweight Python Task Queue

Stars
6K
Forks
392
Last recorded push
9 months ago

Huey is a lightweight Python task queue with a clean API supporting Redis, SQLite, or in-memory storage for background job processing.

Apache Hudi

Lakehouse Platform for Incremental Processing

Stars
6.3K
Forks
0

Apache Hudi is an open data lakehouse platform that enables efficient record-level upserts, incremental processing, and ACID transactions on data lakes.

DynamoDB

AWS managed serverless NoSQL database

Stars
7.6K
Forks
1.5K
Last recorded push
10 months ago

Amazon DynamoDB is a fully managed, serverless NoSQL database with single-digit millisecond performance for high-throughput scraping workloads.

BullMQ

Message Queue and Background Jobs for Node.js

Stars
9.5K
Forks
547
Last recorded push
8 months ago

BullMQ is a task queue for background jobs. Its JavaScript/TypeScript package supports Redis; version 6.3.11 also includes an optional PostgreSQL backend. Backend support must be checked for each language binding.

Apache Iceberg

Open Table Format for Huge Analytic Datasets

Stars
9.3K
Forks
0

Apache Iceberg is a high-performance open table format enabling multi-engine analytics with schema evolution, time travel, and hidden partitioning.

Delta Lake

Open-Source Lakehouse Storage Framework

Stars
9K
Forks
0

Delta Lake is an open-source storage framework that brings ACID transactions, schema evolution, and time travel to data lakes at petabyte scale.

LanceDB

Serverless vector database, no server management

Stars
11.6K
Forks
0

LanceDB is a serverless vector database built in Rust that runs embedded or in the cloud with native DataFrame integration for AI workloads.

Mage AI

AI-Powered Data Pipeline Builder

Stars
8.8K
Forks
905
Last recorded push
8 months ago

Mage AI is a data engineering platform that lets you build, deploy, and run scalable data pipelines with AI assistance across Python, SQL, and R.

Scrapling

Adaptive web scraping for the modern web

Stars
86.8K
Forks
537
Last recorded push
8 months ago

Scrapling is an adaptive Python scraping framework that learns from website changes, handling everything from single requests to full-scale crawls.

Cassandra

Distributed NoSQL with high availability at scale

Stars
10.1K
Forks
3.8K
Last recorded push
8 months ago

Apache Cassandra is a distributed NoSQL database that handles massive amounts of scraped data with high availability and linear scalability.

WebdriverIO

Next-Gen Browser and Mobile Automation for Node.js

Stars
9.8K
Forks
2.7K
Last recorded push
8 months ago

WebdriverIO is an all-in-one Node.js automation framework for web and mobile testing with smart selectors, component testing, and extensive plugins.

TestCafe

WebDriver-Free End-to-End Browser Testing

Stars
9.9K
Forks
677
Last recorded push
8 months ago

TestCafe is a free, open-source end-to-end testing framework that works across browsers without WebDriver, with intuitive syntax and built-in waiting.

RQ (Redis Queue)

Simple Job Queues for Python

Stars
10.7K
Forks
1.5K
Last recorded push
8 months ago

RQ is a lightweight Python library for queueing jobs and processing them in the background with Redis-backed workers.

dbt

Transform Data in Your Warehouse with SQL

Stars
14K
Forks
0

dbt empowers data teams to transform, test, and document data directly in the warehouse using SQL, with version control and governance built in.

Great Expectations

Data Quality Testing and Validation Platform

Stars
11.9K
Forks
0

Great Expectations helps data teams validate, document, and monitor data quality across pipelines with automated testing and observability.

StarRocks

Sub-second analytics for enterprise-scale data

Stars
12.2K
Forks
2.3K
Last recorded push
8 months ago

StarRocks is an open-source analytical database that delivers sub-second query latency for complex joins and aggregations at enterprise scale.