Best Python Web Scraping Tools

Algolia

Managed search and AI retrieval platform

Stars
4K
Forks
557
Last commit
5 months ago

Algolia is a managed search platform that delivers fast, relevant search results over scraped and structured content for web applications.

dlt (data load tool)

Python Library for Data Loading

Stars
4.8K
Forks
0

dlt is a lightweight, open-source Python library for loading data from any source into well-structured datasets with automatic schema inference.

Mechanicalsoup

Automate website interaction with Python

Stars
4.8K
Forks
385
Last commit
6 months ago

MechanicalSoup is a Python library for automating website interaction, handling cookies, redirects, forms, and links without a browser.

Camoufox

Anti-detect browser with fingerprint injection

Stars
4.9K
Forks
459
Last commit
5 months ago

Camoufox is an open-source anti-detect browser based on Firefox with robust fingerprint injection to evade bot detection systems.

Scrapy Redis

Redis-based distributed components for Scrapy

Stars
5.6K
Forks
1.6K
Last commit
2 years ago

Scrapy-Redis provides Redis-backed components for Scrapy, enabling distributed crawling with shared request queues and item pipelines.

Huey

Lightweight Python Task Queue

Stars
5.9K
Forks
392
Last commit
6 months ago

Huey is a lightweight Python task queue with a clean API supporting Redis, SQLite, or in-memory storage for background job processing.

Apache Hudi

Lakehouse Platform for Incremental Processing

Stars
6.1K
Forks
0

Apache Hudi is an open data lakehouse platform that enables efficient record-level upserts, incremental processing, and ACID transactions on data lakes.

BullMQ

Message Queue and Background Jobs for Node.js

Stars
8.3K
Forks
547
Last commit
5 months ago

BullMQ is an open-source Redis-based message queue trusted by thousands of companies processing billions of jobs daily across Node.js, Python, and more.

Apache Iceberg

Open Table Format for Huge Analytic Datasets

Stars
8.5K
Forks
0

Apache Iceberg is a high-performance open table format enabling multi-engine analytics with schema evolution, time travel, and hidden partitioning.

Delta Lake

Open-Source Lakehouse Storage Framework

Stars
8.6K
Forks
0

Delta Lake is an open-source storage framework that brings ACID transactions, schema evolution, and time travel to data lakes at petabyte scale.

LanceDB

Serverless vector database, no server management

Stars
8.6K
Forks
0

LanceDB is a serverless vector database built in Rust that runs embedded or in the cloud with native DataFrame integration for AI workloads.

Mage AI

AI-Powered Data Pipeline Builder

Stars
8.6K
Forks
905
Last commit
5 months ago

Mage AI is a data engineering platform that lets you build, deploy, and run scalable data pipelines with AI assistance across Python, SQL, and R.

Scrapling

Adaptive web scraping for the modern web

Stars
8.8K
Forks
537
Last commit
5 months ago

Scrapling is an adaptive Python scraping framework that learns from website changes, handling everything from single requests to full-scale crawls.

Cassandra

Distributed NoSQL with high availability at scale

Stars
9.6K
Forks
3.8K
Last commit
5 months ago

Apache Cassandra is a distributed NoSQL database that handles massive amounts of scraped data with high availability and linear scalability.

RQ (Redis Queue)

Simple Job Queues for Python

Stars
10.6K
Forks
1.5K
Last commit
5 months ago

RQ is a lightweight Python library for queueing jobs and processing them in the background with Redis-backed workers.

dbt

Transform Data in Your Warehouse with SQL

Stars
10.9K
Forks
0

dbt empowers data teams to transform, test, and document data directly in the warehouse using SQL, with version control and governance built in.

Great Expectations

Data Quality Testing and Validation Platform

Stars
11.1K
Forks
0

Great Expectations helps data teams validate, document, and monitor data quality across pipelines with automated testing and observability.

OpenSearch

Open-source search and analytics suite

Stars
12.3K
Forks
2.4K
Last commit
5 months ago

OpenSearch is an Apache 2.0-licensed search and analytics suite that lets you ingest, search, visualize, and analyze scraped data at scale.

KeyDB

Multithreaded Redis fork with higher throughput

Stars
12.4K
Forks
647
Last commit
2 years ago

KeyDB is a multithreaded Redis-compatible database delivering over 1M ops/sec per node for high-throughput caching and queuing.

Apache Druid

Real-time OLAP database for event-driven data

Stars
13.9K
Forks
0

Apache Druid is a real-time OLAP database that delivers sub-second queries on streaming and batch data for analytics at scale.

Dagster

Unified Platform for AI and Data Pipelines

Stars
14.8K
Forks
2K
Last commit
5 months ago

Dagster is a data orchestration platform that helps teams build, schedule, and monitor reliable data pipelines with integrated lineage and observability.

Weaviate

AI-native vector database with built-in ML models

Stars
15.5K
Forks
0

Weaviate is an AI-native vector database with built-in ML models for hybrid search, RAG, and semantic retrieval over scraped content.

QuestDB

Time-series database with ultra-low latency

Stars
16.5K
Forks
0

QuestDB is an open-source time-series database delivering ultra-low latency ingestion and queries for scraping metrics and monitoring data.

Temporal

Durable Execution for Invincible Applications

Stars
17.9K
Forks
1.3K
Last commit
5 months ago

Temporal is a durable execution platform that makes distributed applications fault-tolerant by automatically capturing state and recovering from failures.

Luigi

Python Batch Pipeline Framework by Spotify

Stars
18.6K
Forks
2.4K
Last commit
5 months ago

Luigi is a Python package by Spotify for building complex pipelines of batch jobs with dependency resolution, workflow management, and Hadoop support.

Skyvern

AI Browser Automation with Computer Vision

Stars
20.2K
Forks
1.8K
Last commit
5 months ago

Skyvern automates browser-based workflows using LLMs and computer vision, replacing brittle scripts with AI that adapts to any webpage automatically.

Airbyte

Open-Source Data Integration Platform

Stars
20.5K
Forks
0

Airbyte is an open-source ELT platform with 600+ connectors that moves data from APIs, databases, and files to warehouses and data lakes at scale.

Crawlee

Build reliable web scrapers. Fast.

Stars
21.3K
Forks
1.2K
Last commit
5 months ago

Crawlee is a JavaScript and Python web scraping library that handles blocking, proxies, and browsers so you can build reliable crawlers fast.

Prefect

Workflow Orchestration for Python Data Pipelines

Stars
21.4K
Forks
2.1K
Last commit
5 months ago

Prefect is an open-source Python workflow orchestration framework for building resilient data pipelines with full observability and zero rewrites.

ScrapeGraphAI

AI-Powered Web Scraping API for Agents

Stars
22.4K
Forks
1.9K
Last commit
5 months ago

ScrapeGraphAI extracts structured data from any website using AI, handling proxies, rendering, and adapting to site changes automatically.

Typesense

Typo-tolerant search with sub-50ms responses

Stars
25.1K
Forks
849
Last commit
5 months ago

Typesense is a fast, typo-tolerant search engine that delivers instant search-as-you-type results over your scraped and structured data.

Chroma

AI-native embedding database with simple API

Stars
25.7K
Forks
0

Chroma is an open-source embedding database with a simple API for storing and querying vectors, ideal for AI search over scraped content.

Pydantic

Data Validation Using Python Type Hints

Stars
26.5K
Forks
0

Pydantic is the most widely used Python data validation library, offering fast schema enforcement with type hints and a Rust-powered core engine.

Celery

Distributed Task Queue for Python

Stars
27.9K
Forks
4.9K
Last commit
5 months ago

Celery is a simple, flexible, and reliable distributed task queue for processing vast amounts of messages with real-time and scheduled execution.

MongoDB

Flexible document database for any data shape

Stars
28K
Forks
5.7K
Last commit
5 months ago

MongoDB is a flexible document database that stores scraped data as JSON-like documents, making it easy to ingest varied web content.

Qdrant

High-performance vector search engine in Rust

Stars
28.4K
Forks
0

Qdrant is a high-performance vector search engine built in Rust for production-grade semantic search and RAG applications.