Python Web Scraping Tools

Ultimate Sitemap Parser

Auto-Discover And Parse Sitemaps In Python

Stars
257
Forks
74
Last recorded push
8 months ago

Python library that finds a site's sitemaps from robots.txt and common paths, recurses sitemap indexes, and streams pages with low memory.

Requests-HTML

Python Requests With HTML Parsing

Stars
13.8K
Forks
999
Last recorded push
2 years ago

Python library combining HTTP requests, HTML parsing, CSS selectors, and optional JavaScript rendering for scraping workflows.

pyquery

jQuery-Style HTML Parsing For Python

Stars
2.4K
Forks
184
Last recorded push
7 months ago

Python library that applies a jQuery-style API to HTML and XML documents for selection, traversal, and extraction.

Helium

Readable Selenium Automation For Python

Stars
8.3K
Forks
513
Last recorded push
3 months ago

Python browser automation library that wraps Selenium with a higher-level API for clicking, typing, and finding visible elements.

AutoScraper

Example-Based HTML Extraction For Python

Stars
8K
Forks
764
Last recorded push
last year

Python library that learns simple scraping rules from example values and extracts similar data from HTML pages.

Pyppeteer

Unmaintained Python Port of Puppeteer

Stars
3.9K
Forks
347
Last recorded push
2 years ago

Python port of Puppeteer for Chrome and Chromium automation. Its maintainers mark the repository as unmaintained and suggest Playwright Python.

Parsel

Extract Web Data Cleanly With Python

Stars
1.4K
Forks
162
Last recorded push
3 months ago

A Python library for extracting data from HTML, XML, and JSON with CSS, XPath, JMESPath, and regex.

selectolax

Fast HTML Parsing for Python Workflows

Stars
1.7K
Forks
91
Last recorded push
4 months ago

Parse HTML with CSS selectors in Python using a fast, MIT-licensed library built for large-scale extraction.

SeleniumBase

Reliable Browser Automation For Python

Stars
13.1K
Forks
1.6K
Last recorded push
3 months ago

An open-source Python framework for browser automation, testing, and practical anti-detection scraping with UC Mode and CDP Mode.

curl_cffi

Browser-Like Requests Without Full Browsers

Stars
6.7K
Forks
501
Last recorded push
3 months ago

Python HTTP client for scraping with browser-like TLS, JA3, and HTTP/2 fingerprint impersonation without running a full browser.

Unstructured

Clean Documents For AI Pipelines

Stars
15.6K
Forks
1.3K
Last recorded push
3 months ago

An open-source Python document ETL tool that turns PDFs, HTML, images, and more into structured elements for RAG and LLM workflows.

lxml

Fast XML And HTML Parsing For Python

Stars
3.1K
Forks
619
Last recorded push
3 months ago

A fast Python library for parsing XML and HTML, often used as a foundation for scraping and data tools.

Docling

Turn Documents Into AI-Ready Data

Stars
68.7K
Forks
4.4K
Last recorded push
3 months ago

Open-source Python software that converts PDFs, DOCX, PPTX, and images into structured data for GenAI and RAG pipelines.

Zyte API

All-in-One Web Scraping API with AI Extraction

Scraping FrameworksSERP APIs

Zyte API is a full-stack web scraping API that combines anti-bot bypassing, browser rendering, and AI-powered data extraction in a single endpoint.

Exa AI

Semantic Web Search API Powered by AI

SERP APIs

Exa AI is a modern search engine and web search API that uses semantic embeddings to deliver high-quality, real-time web data for AI applications.

Tavily

Real-Time Search API for AI Agents and RAG

SERP APIs

Tavily is a real-time search engine for AI agents and RAG workflows, delivering fast web search and content extraction through a single secure API.

Brave Search API

Independent Web Search API with 35B+ Pages

SERP APIs

Brave Search API provides enterprise-grade access to an independent index of 35+ billion web pages with real-time results optimized for AI and RAG pipelines.

AWS Athena

Serverless SQL Queries on Amazon S3

ETL Tools

Amazon Athena is a serverless, interactive query service that lets you analyze petabyte-scale data in S3 using standard SQL with pay-per-query pricing.

AWS Glue

Serverless Data Integration Service by AWS

ETL Tools

AWS Glue is a fully managed, serverless ETL service that discovers, prepares, and integrates data from over 100 sources into lakes and warehouses.

Webshare

Fast and affordable proxy servers

Proxy Services

Webshare provides self-serve proxy servers across many locations, offering datacenter and residential proxy options for scraping workflows.

AWS Lambda

Serverless compute for event-driven pipelines

Cloud Compute

AWS Lambda runs code without managing servers, ideal for event-driven scraping pipelines triggered by schedules, queues, or API calls.

ScrapingBee

Web scraping API with proxy and browser handling

Stars
132
Forks
4
Last recorded push
8 months ago

ScrapingBee is a web scraping API that handles headless browsers, rotates proxies, and offers AI-powered extraction so you can focus on your data.

DataForSEO

Comprehensive API Stack for SEO and Marketing Data

Stars
51
Forks
12
Last recorded push
8 months ago

DataForSEO provides SEO and digital marketing data via API, serving as a trusted data partner for 750+ software companies and agencies worldwide.

CapSolver

AI-powered automatic CAPTCHA solving service

Stars
74
Forks
10
Last recorded push
last year

CapSolver is an AI-powered CAPTCHA solver that automatically bypasses reCAPTCHA, Cloudflare, AWS WAF, and other challenges for automation.

Oxylabs

Premium proxy service for data at scale

Stars
105
Forks
5
Last recorded push
9 months ago

Oxylabs offers 175M+ residential and datacenter proxies with web scraping APIs and browser automation tools for large-scale public data extraction.

SerpAPI

Real-Time Google Search Results API

Stars
176
Forks
11
Last recorded push
10 months ago

SerpAPI is a real-time API that delivers structured Google search results, handling proxies, captchas, and rich data parsing automatically.

Apify

Full-stack web scraping and data extraction platform

Stars
180
Forks
21
Last recorded push
8 months ago

Apify is a cloud platform for web scraping, browser automation, and data extraction with an Actor marketplace and developer SDKs.

BeautifulSoup

Python library for parsing HTML and XML

Stars
226
Forks
57
Last recorded push
3 years ago

Beautiful Soup is a Python library for pulling data out of HTML and XML files, providing Pythonic idioms for navigating and searching parse trees.

Modal

Serverless Python platform for AI and data teams

Stars
523
Forks
0

Modal is a serverless Python platform that runs scraping, AI, and data workloads at scale with configurable containers and autoscaling.

2Captcha

CAPTCHA solving service with API integration

Stars
803
Forks
126
Last recorded push
8 months ago

2Captcha is a CAPTCHA solving service that bypasses reCAPTCHA, Cloudflare Turnstile, hCaptcha, and other challenges via a simple API.

Zendriver

Async-first undetected browser automation

Stars
1.5K
Forks
75
Last recorded push
10 months ago

Zendriver is a fast, async-first undetected browser automation framework based on nodriver with Docker support for scalable scraping.

Frontera

Scalable crawl frontier framework

Stars
1.3K
Forks
217
Last recorded push
last year

Frontera is a Python crawl frontier framework for managing when and what to crawl, enabling web crawlers of any scale with Scrapy integration.

Patchright

Undetected Playwright fork for bot evasion

Stars
4.8K
Forks
88
Last recorded push
8 months ago

Patchright is a patched fork of Playwright that evades bot detection with a Playwright-style API and documented compatibility limits.

Meltano

CLI-First ELT Engine Built on Singer

Stars
2.7K
Forks
0

Meltano is an open-source, CLI-first ELT platform with 600+ connectors, GitOps workflows, and native dbt integration for data pipeline management.

BeeAI Framework

Production-Ready AI Agent Framework

Stars
3.4K
Forks
400
Last recorded push
8 months ago

BeeAI Framework is an open-source platform for building production-ready multi-agent systems in Python and TypeScript with MCP and workflow support.

Nodriver

Undetected browser automation without Selenium

Stars
4.8K
Forks
341
Last recorded push
11 months ago

Nodriver is the successor to undetected-chromedriver, providing fast CDP-based browser automation with built-in anti-detection and no Selenium dependency.

Ultimate Sitemap Parser

Auto-Discover And Parse Sitemaps In Python

Stars
257
Forks
74
Last recorded push
8 months ago

Python library that finds a site's sitemaps from robots.txt and common paths, recurses sitemap indexes, and streams pages with low memory.

Requests-HTML

Python Requests With HTML Parsing

Stars
13.8K
Forks
999
Last recorded push
2 years ago

Python library combining HTTP requests, HTML parsing, CSS selectors, and optional JavaScript rendering for scraping workflows.

pyquery

jQuery-Style HTML Parsing For Python

Stars
2.4K
Forks
184
Last recorded push
7 months ago

Python library that applies a jQuery-style API to HTML and XML documents for selection, traversal, and extraction.

Helium

Readable Selenium Automation For Python

Stars
8.3K
Forks
513
Last recorded push
3 months ago

Python browser automation library that wraps Selenium with a higher-level API for clicking, typing, and finding visible elements.

AutoScraper

Example-Based HTML Extraction For Python

Stars
8K
Forks
764
Last recorded push
last year

Python library that learns simple scraping rules from example values and extracts similar data from HTML pages.

Pyppeteer

Unmaintained Python Port of Puppeteer

Stars
3.9K
Forks
347
Last recorded push
2 years ago

Python port of Puppeteer for Chrome and Chromium automation. Its maintainers mark the repository as unmaintained and suggest Playwright Python.

Parsel

Extract Web Data Cleanly With Python

Stars
1.4K
Forks
162
Last recorded push
3 months ago

A Python library for extracting data from HTML, XML, and JSON with CSS, XPath, JMESPath, and regex.

selectolax

Fast HTML Parsing for Python Workflows

Stars
1.7K
Forks
91
Last recorded push
4 months ago

Parse HTML with CSS selectors in Python using a fast, MIT-licensed library built for large-scale extraction.

SeleniumBase

Reliable Browser Automation For Python

Stars
13.1K
Forks
1.6K
Last recorded push
3 months ago

An open-source Python framework for browser automation, testing, and practical anti-detection scraping with UC Mode and CDP Mode.

curl_cffi

Browser-Like Requests Without Full Browsers

Stars
6.7K
Forks
501
Last recorded push
3 months ago

Python HTTP client for scraping with browser-like TLS, JA3, and HTTP/2 fingerprint impersonation without running a full browser.

Unstructured

Clean Documents For AI Pipelines

Stars
15.6K
Forks
1.3K
Last recorded push
3 months ago

An open-source Python document ETL tool that turns PDFs, HTML, images, and more into structured elements for RAG and LLM workflows.

lxml

Fast XML And HTML Parsing For Python

Stars
3.1K
Forks
619
Last recorded push
3 months ago

A fast Python library for parsing XML and HTML, often used as a foundation for scraping and data tools.

Docling

Turn Documents Into AI-Ready Data

Stars
68.7K
Forks
4.4K
Last recorded push
3 months ago

Open-source Python software that converts PDFs, DOCX, PPTX, and images into structured data for GenAI and RAG pipelines.

Zyte API

All-in-One Web Scraping API with AI Extraction

Scraping FrameworksSERP APIs

Zyte API is a full-stack web scraping API that combines anti-bot bypassing, browser rendering, and AI-powered data extraction in a single endpoint.

Exa AI

Semantic Web Search API Powered by AI

SERP APIs

Exa AI is a modern search engine and web search API that uses semantic embeddings to deliver high-quality, real-time web data for AI applications.

Tavily

Real-Time Search API for AI Agents and RAG

SERP APIs

Tavily is a real-time search engine for AI agents and RAG workflows, delivering fast web search and content extraction through a single secure API.

Brave Search API

Independent Web Search API with 35B+ Pages

SERP APIs

Brave Search API provides enterprise-grade access to an independent index of 35+ billion web pages with real-time results optimized for AI and RAG pipelines.

AWS Athena

Serverless SQL Queries on Amazon S3

ETL Tools

Amazon Athena is a serverless, interactive query service that lets you analyze petabyte-scale data in S3 using standard SQL with pay-per-query pricing.

AWS Glue

Serverless Data Integration Service by AWS

ETL Tools

AWS Glue is a fully managed, serverless ETL service that discovers, prepares, and integrates data from over 100 sources into lakes and warehouses.

Webshare

Fast and affordable proxy servers

Proxy Services

Webshare provides self-serve proxy servers across many locations, offering datacenter and residential proxy options for scraping workflows.

AWS Lambda

Serverless compute for event-driven pipelines

Cloud Compute

AWS Lambda runs code without managing servers, ideal for event-driven scraping pipelines triggered by schedules, queues, or API calls.

ScrapingBee

Web scraping API with proxy and browser handling

Stars
132
Forks
4
Last recorded push
8 months ago

ScrapingBee is a web scraping API that handles headless browsers, rotates proxies, and offers AI-powered extraction so you can focus on your data.

DataForSEO

Comprehensive API Stack for SEO and Marketing Data

Stars
51
Forks
12
Last recorded push
8 months ago

DataForSEO provides SEO and digital marketing data via API, serving as a trusted data partner for 750+ software companies and agencies worldwide.

CapSolver

AI-powered automatic CAPTCHA solving service

Stars
74
Forks
10
Last recorded push
last year

CapSolver is an AI-powered CAPTCHA solver that automatically bypasses reCAPTCHA, Cloudflare, AWS WAF, and other challenges for automation.

Oxylabs

Premium proxy service for data at scale

Stars
105
Forks
5
Last recorded push
9 months ago

Oxylabs offers 175M+ residential and datacenter proxies with web scraping APIs and browser automation tools for large-scale public data extraction.

SerpAPI

Real-Time Google Search Results API

Stars
176
Forks
11
Last recorded push
10 months ago

SerpAPI is a real-time API that delivers structured Google search results, handling proxies, captchas, and rich data parsing automatically.

Apify

Full-stack web scraping and data extraction platform

Stars
180
Forks
21
Last recorded push
8 months ago

Apify is a cloud platform for web scraping, browser automation, and data extraction with an Actor marketplace and developer SDKs.

BeautifulSoup

Python library for parsing HTML and XML

Stars
226
Forks
57
Last recorded push
3 years ago

Beautiful Soup is a Python library for pulling data out of HTML and XML files, providing Pythonic idioms for navigating and searching parse trees.

Modal

Serverless Python platform for AI and data teams

Stars
523
Forks
0

Modal is a serverless Python platform that runs scraping, AI, and data workloads at scale with configurable containers and autoscaling.

2Captcha

CAPTCHA solving service with API integration

Stars
803
Forks
126
Last recorded push
8 months ago

2Captcha is a CAPTCHA solving service that bypasses reCAPTCHA, Cloudflare Turnstile, hCaptcha, and other challenges via a simple API.

Zendriver

Async-first undetected browser automation

Stars
1.5K
Forks
75
Last recorded push
10 months ago

Zendriver is a fast, async-first undetected browser automation framework based on nodriver with Docker support for scalable scraping.

Frontera

Scalable crawl frontier framework

Stars
1.3K
Forks
217
Last recorded push
last year

Frontera is a Python crawl frontier framework for managing when and what to crawl, enabling web crawlers of any scale with Scrapy integration.

Patchright

Undetected Playwright fork for bot evasion

Stars
4.8K
Forks
88
Last recorded push
8 months ago

Patchright is a patched fork of Playwright that evades bot detection with a Playwright-style API and documented compatibility limits.

Meltano

CLI-First ELT Engine Built on Singer

Stars
2.7K
Forks
0

Meltano is an open-source, CLI-first ELT platform with 600+ connectors, GitOps workflows, and native dbt integration for data pipeline management.

BeeAI Framework

Production-Ready AI Agent Framework

Stars
3.4K
Forks
400
Last recorded push
8 months ago

BeeAI Framework is an open-source platform for building production-ready multi-agent systems in Python and TypeScript with MCP and workflow support.

Nodriver

Undetected browser automation without Selenium

Stars
4.8K
Forks
341
Last recorded push
11 months ago

Nodriver is the successor to undetected-chromedriver, providing fast CDP-based browser automation with built-in anti-detection and no Selenium dependency.

Discover web scraping tools and libraries for Python. Compare features, community support, and documentation.