Best Open Source Web Scraping Tools

A Python library to interact with ScrapingBee's API for headless browsers and proxy rotation

Official DataForSEO python client

Bright Data's python SDK, use it to call bright data's scrape and search tools. bypass any Bot-detection or Captcha and extract data from any website in seconds.

Python 3 package for easy integration with the API of capsolver captcha solving service to solver recaptcha, geetest and solve any other captchas. Capsolver.

a Python client library for SerpApi.

The Apify SDK for Python is the official library for creating Apify Actors in Python. It provides useful features like actor lifecycle management, local storage emulation, and actor event handling.

git mirror for Beautiful Soup 4.3.2

Python 3 package for easy integration with the API of 2captcha captcha solving service to bypass recaptcha, сloudflare turnstile, funcaptcha, geetest and solve any other captchas.

Async-first undetected browser automation based on nodriver. Clean API with Docker support for scalable scraping.

This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.

A scalable frontier for web crawlers

Undetected Playwright fork. Patches runtime to evade bot detection while maintaining full Playwright API compatibility.

Code-first ELT engine built on Singer. GitOps workflows, 300+ connectors, integrates with dbt.

Open-source super proxy aggregator. Manage multiple proxy providers in one place. Built-in fingerprinting and rotation.

Build production-ready AI agents in Python and TypeScript. Linux Foundation project.

Apache Nutch is an extensible and scalable web crawler

Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.

Successor to undetected-chromedriver. CDP-based browser automation with built-in anti-detection. No Selenium dependency.

HTML5 spec compliant parser

Managed search, great for e-commerce

The leading workflow orchestration platform. Run stateful step functions and AI workflows on serverless, servers, or the edge.

Low-level, fast, forgiving HTML parser

Lightweight Python library for data loading. Auto schema inference, 5000+ sources supported.

A Python library for automating interaction with websites.

Anti-detect Firefox browser with fingerprint injection & anti-bot evasion. Playwright-compatible.

Redis-based components for Scrapy.

Tiny but feature-rich, Redis or SQLite

Lakehouse platform for upserts and incremental processing. Efficient record-level updates.

TypeScript library for structured data extraction

DevTools Protocol, handles JavaScript

AWS managed, auto-scaling NoSQL

BullMQ - Message Queue and Batch processing for NodeJS, Python, Elixir and PHP based on Redis

Open table format for huge analytics tables. Multi-engine (Spark, Trino, Flink). Used by Netflix, Airbnb.

Lakehouse storage framework. ACID transactions, schema evolution, time travel. Databricks standard.