Web scraping tools

Quick answer

Route by stack and operating model. Scrapy fits Python teams that want self-hosted crawlers, Crawlee fits Node and TypeScript teams, and Playwright fits jobs where JavaScript rendering is the hard part. Use Firecrawl or another scraping API when fast integration and AI-ready output matter, and use Octoparse or ParseHub when a no-code workflow is the better handoff.

Sitebulb

SEO Audits With Sitemap Discovery And Exports

Sitemap Scrapers

Desktop and cloud SEO auditing tool that discovers XML sitemaps, accepts a sitemap index as a crawl source, and exports URLs to CSV or Google Sheets.

Screaming Frog SEO Spider

Desktop SEO Crawler With Sitemap Auditing

Sitemap Scrapers

Desktop SEO crawler that can crawl linked XML sitemaps, download a sitemap in List Mode to crawl only its URLs, and export flat URL lists to CSV.

Ben Balter sitemap-parser

Simple Sitemaps.org XML Parsing For Ruby

Stars
33
Forks
47
Last commit
last year

Ruby parser for sitemaps.org-compliant XML sitemaps; a recurse option flattens nested indexes into one URL list returned by to_a.

TurnerSoftware SitemapTools

Discover And Parse A Domain's Sitemaps In C#

Stars
38
Forks
10
Last commit
2 years ago

C# library whose GetAllSitemapsForDomainAsync() takes a domain and discovers plus parses all its sitemaps, with lenient parsing for messy real-world files.

VIPnytt SitemapParser

Recursive Sitemap Parsing For PHP

Stars
78
Forks
33
Last commit
10 months ago

PHP library with parse() and parseRecursive() APIs; parseRecursive on a robots.txt URL walks every nested sitemap for a complete URL list.

Crawler-Commons

The De Facto Java Sitemap Parsing Library

Stars
259
Forks
91
Last commit
2 months ago

Java library with a SAX-based sitemap parser for sitemaps, indexes, extensions, and feeds; you supply the bytes and it parses and normalizes them.

gopher-parse-sitemap

Zero-Dependency Streaming Sitemap Parser In Go

Stars
40
Forks
19
Last commit
3 years ago

Streaming Go parser for huge XML sitemaps with minimal memory and no external dependencies, consuming entries on the fly via callbacks.

go-sitemap-parser

Resolve Robots, Indexes, And Sitemaps In Go

Stars
6
Forks
1
Last commit
3 months ago

Go library whose Parse() accepts a robots.txt, sitemap index, or plain sitemap URL and resolves the rest, with concurrent fetching and depth limits.

sitemap-stream-parser

Stream Very Large Sitemaps With Low Memory

Stars
39
Forks
19
Last commit
5 years ago

Streaming Node.js XML sitemap parser for very large, deeply nested sitemaps, with a callback API and a helper that finds sitemaps in robots.txt.

Sitemapper

Parse Sitemaps And Indexes In Node.js

Stars
135
Forks
83
Last commit
3 months ago

Node.js library that parses a sitemap URL including sitemap indexes with many child sitemaps, with concurrency, retries, timeout, and proxy options.

Ultimate Sitemap Parser

Auto-Discover And Parse Sitemaps In Python

Stars
242
Forks
74
Last commit
7 months ago

Python library that finds a site's sitemaps from robots.txt and common paths, recurses sitemap indexes, and streams pages with low memory.

Crawlbase

Web data infrastructure for scraping at scale

AI Web Scraping

+2 more

Web data platform for crawling, extracting, proxying, and storing scraped data through APIs, managed crawlers, cloud storage, and MCP integrations.

Requests-HTML

Python Requests With HTML Parsing

Stars
13.8K
Forks
999
Last commit
2 years ago

Python library combining HTTP requests, HTML parsing, CSS selectors, and optional JavaScript rendering for scraping workflows.

pyquery

jQuery-Style HTML Parsing For Python

Stars
2.4K
Forks
184
Last commit
6 months ago

Python library that applies a jQuery-style API to HTML and XML documents for selection, traversal, and extraction.

Helium

Readable Selenium Automation For Python

Stars
8.3K
Forks
513
Last commit
2 months ago

Python browser automation library that wraps Selenium with a higher-level API for clicking, typing, and finding visible elements.

AutoScraper

Example-Based HTML Extraction For Python

Stars
7.5K
Forks
764
Last commit
last year

Python library that learns simple scraping rules from example values and extracts similar data from HTML pages.

Pyppeteer

Python Control For Headless Chromium

Stars
4K
Forks
347
Last commit
2 years ago

Python library for controlling Chrome and Chromium through a Puppeteer-like API for screenshots, automation, and scraping.

Chat4Data

Chat-Based Web Data Extraction

AI Web ScrapingWeb Scraping

Chrome-based AI scraping tool that lets users extract page data through conversational instructions and export structured results.

Thunderbit

AI Web Scraping From Browser Pages

AI Web ScrapingWeb Scraping

No-code AI web scraping tool for turning pages, PDFs, and images into structured tables from the browser.

Kadoa

Self-Healing AI Web Data Pipelines

AI Web ScrapingWeb Scraping

AI web scraping platform for building self-healing extraction agents that monitor pages and collect structured data at scale.

got-scraping

Browser-Like Requests For Reliable Scraping

Stars
760
Forks
60
Last commit
3 months ago

A Node.js HTTP client for scraping that sends browser-like requests with realistic headers and TLS behavior.

Parsel

Extract Web Data Cleanly With Python

Stars
1.3K
Forks
162
Last commit
2 months ago

A Python library for extracting data from HTML, XML, and JSON with CSS, XPath, JMESPath, and regex.

jsoup

Parse Real-World HTML In Java

Stars
11.4K
Forks
2.3K
Last commit
2 months ago

A Java library for parsing real-world HTML into a standards-based DOM with CSS selectors and DOM traversal.

selectolax

Fast HTML Parsing for Python Workflows

Stars
1.6K
Forks
91
Last commit
3 months ago

Parse HTML with CSS selectors in Python using a fast, MIT-licensed library built for large-scale extraction.

SeleniumBase

Reliable Browser Automation For Python

Stars
12.8K
Forks
1.6K
Last commit
2 months ago

An open-source Python framework for browser automation, testing, and practical anti-detection scraping with UC Mode and CDP Mode.

Diffbot

Turn Web Pages Into Usable Data

AI Web Scraping

An AI extraction platform that turns web pages into structured data and a knowledge graph for web data workflows.

lxml

Fast XML And HTML Parsing For Python

Stars
3K
Forks
619
Last commit
2 months ago

A fast Python library for parsing XML and HTML, often used as a foundation for scraping and data tools.

Scrapfly

AI-Ready Web Data Extraction

Scraping APIs

A commercial scraping platform that helps teams collect structured web data, automate cloud browsers, crawl sites, handle anti-bot blocks, and use proxies.

ZenRows

AI-Ready Scraping With Anti-Bot Bypass

Scraping APIs

A commercial scraping API for bypassing bot defenses, rendering JavaScript, using residential proxies, and returning AI-ready results.

Stagehand

Automate Web Workflows With AI

Stars
23.3K
Forks
1.6K
Last commit
2 months ago

Open-source TypeScript framework for building AI-assisted browser automations that combine plain-language steps with Playwright code.

Browserbase

Reliable Cloud Browsers for AI Agents

Headless Browsers

Run hosted browser sessions for AI automation and scraping with search, fetch, stealth, observability, and replay built in.

Browse AI

No-code AI scraping and monitoring for any site

AI Web Scraping

Extract and monitor data from any website without code, turn pages into spreadsheets or APIs, and connect to thousands of apps.

ParseHub

Free visual scraper for dynamic, interactive sites

Web Scraping

Extract data from dynamic websites by clicking elements, handle AJAX, forms, and dropdowns, then export to JSON, Excel, or API.

Octoparse

Turn web pages into structured data, no code

Web Scraping

Build scrapers visually and turn web pages into structured data in minutes, with scheduling and an optional managed data service.

Perplexity Sonar

Generative AI Search API with Real-Time Research

SERP APIs

Perplexity Sonar is a generative AI search API that provides real-time web research with citations, customizable sources, and multiple performance tiers.

ValueSERP

Low-Cost Google SERP API for Data Collection

SERP APIs

ValueSERP provides real-time, reliable Google search data at the best value with rates as low as $0.50 per 1,000 searches and minimal maintenance.

Sitebulb

SEO Audits With Sitemap Discovery And Exports

Sitemap Scrapers

Desktop and cloud SEO auditing tool that discovers XML sitemaps, accepts a sitemap index as a crawl source, and exports URLs to CSV or Google Sheets.

Screaming Frog SEO Spider

Desktop SEO Crawler With Sitemap Auditing

Sitemap Scrapers

Desktop SEO crawler that can crawl linked XML sitemaps, download a sitemap in List Mode to crawl only its URLs, and export flat URL lists to CSV.

Ben Balter sitemap-parser

Simple Sitemaps.org XML Parsing For Ruby

Stars
33
Forks
47
Last commit
last year

Ruby parser for sitemaps.org-compliant XML sitemaps; a recurse option flattens nested indexes into one URL list returned by to_a.

TurnerSoftware SitemapTools

Discover And Parse A Domain's Sitemaps In C#

Stars
38
Forks
10
Last commit
2 years ago

C# library whose GetAllSitemapsForDomainAsync() takes a domain and discovers plus parses all its sitemaps, with lenient parsing for messy real-world files.

VIPnytt SitemapParser

Recursive Sitemap Parsing For PHP

Stars
78
Forks
33
Last commit
10 months ago

PHP library with parse() and parseRecursive() APIs; parseRecursive on a robots.txt URL walks every nested sitemap for a complete URL list.

Crawler-Commons

The De Facto Java Sitemap Parsing Library

Stars
259
Forks
91
Last commit
2 months ago

Java library with a SAX-based sitemap parser for sitemaps, indexes, extensions, and feeds; you supply the bytes and it parses and normalizes them.

gopher-parse-sitemap

Zero-Dependency Streaming Sitemap Parser In Go

Stars
40
Forks
19
Last commit
3 years ago

Streaming Go parser for huge XML sitemaps with minimal memory and no external dependencies, consuming entries on the fly via callbacks.

go-sitemap-parser

Resolve Robots, Indexes, And Sitemaps In Go

Stars
6
Forks
1
Last commit
3 months ago

Go library whose Parse() accepts a robots.txt, sitemap index, or plain sitemap URL and resolves the rest, with concurrent fetching and depth limits.

sitemap-stream-parser

Stream Very Large Sitemaps With Low Memory

Stars
39
Forks
19
Last commit
5 years ago

Streaming Node.js XML sitemap parser for very large, deeply nested sitemaps, with a callback API and a helper that finds sitemaps in robots.txt.

Sitemapper

Parse Sitemaps And Indexes In Node.js

Stars
135
Forks
83
Last commit
3 months ago

Node.js library that parses a sitemap URL including sitemap indexes with many child sitemaps, with concurrency, retries, timeout, and proxy options.

Ultimate Sitemap Parser

Auto-Discover And Parse Sitemaps In Python

Stars
242
Forks
74
Last commit
7 months ago

Python library that finds a site's sitemaps from robots.txt and common paths, recurses sitemap indexes, and streams pages with low memory.

Crawlbase

Web data infrastructure for scraping at scale

AI Web Scraping

+2 more

Web data platform for crawling, extracting, proxying, and storing scraped data through APIs, managed crawlers, cloud storage, and MCP integrations.

Requests-HTML

Python Requests With HTML Parsing

Stars
13.8K
Forks
999
Last commit
2 years ago

Python library combining HTTP requests, HTML parsing, CSS selectors, and optional JavaScript rendering for scraping workflows.

pyquery

jQuery-Style HTML Parsing For Python

Stars
2.4K
Forks
184
Last commit
6 months ago

Python library that applies a jQuery-style API to HTML and XML documents for selection, traversal, and extraction.

Helium

Readable Selenium Automation For Python

Stars
8.3K
Forks
513
Last commit
2 months ago

Python browser automation library that wraps Selenium with a higher-level API for clicking, typing, and finding visible elements.

AutoScraper

Example-Based HTML Extraction For Python

Stars
7.5K
Forks
764
Last commit
last year

Python library that learns simple scraping rules from example values and extracts similar data from HTML pages.

Pyppeteer

Python Control For Headless Chromium

Stars
4K
Forks
347
Last commit
2 years ago

Python library for controlling Chrome and Chromium through a Puppeteer-like API for screenshots, automation, and scraping.

Chat4Data

Chat-Based Web Data Extraction

AI Web ScrapingWeb Scraping

Chrome-based AI scraping tool that lets users extract page data through conversational instructions and export structured results.

Thunderbit

AI Web Scraping From Browser Pages

AI Web ScrapingWeb Scraping

No-code AI web scraping tool for turning pages, PDFs, and images into structured tables from the browser.

Kadoa

Self-Healing AI Web Data Pipelines

AI Web ScrapingWeb Scraping

AI web scraping platform for building self-healing extraction agents that monitor pages and collect structured data at scale.

got-scraping

Browser-Like Requests For Reliable Scraping

Stars
760
Forks
60
Last commit
3 months ago

A Node.js HTTP client for scraping that sends browser-like requests with realistic headers and TLS behavior.

Parsel

Extract Web Data Cleanly With Python

Stars
1.3K
Forks
162
Last commit
2 months ago

A Python library for extracting data from HTML, XML, and JSON with CSS, XPath, JMESPath, and regex.

jsoup

Parse Real-World HTML In Java

Stars
11.4K
Forks
2.3K
Last commit
2 months ago

A Java library for parsing real-world HTML into a standards-based DOM with CSS selectors and DOM traversal.

selectolax

Fast HTML Parsing for Python Workflows

Stars
1.6K
Forks
91
Last commit
3 months ago

Parse HTML with CSS selectors in Python using a fast, MIT-licensed library built for large-scale extraction.

SeleniumBase

Reliable Browser Automation For Python

Stars
12.8K
Forks
1.6K
Last commit
2 months ago

An open-source Python framework for browser automation, testing, and practical anti-detection scraping with UC Mode and CDP Mode.

Diffbot

Turn Web Pages Into Usable Data

AI Web Scraping

An AI extraction platform that turns web pages into structured data and a knowledge graph for web data workflows.

lxml

Fast XML And HTML Parsing For Python

Stars
3K
Forks
619
Last commit
2 months ago

A fast Python library for parsing XML and HTML, often used as a foundation for scraping and data tools.

Scrapfly

AI-Ready Web Data Extraction

Scraping APIs

A commercial scraping platform that helps teams collect structured web data, automate cloud browsers, crawl sites, handle anti-bot blocks, and use proxies.

ZenRows

AI-Ready Scraping With Anti-Bot Bypass

Scraping APIs

A commercial scraping API for bypassing bot defenses, rendering JavaScript, using residential proxies, and returning AI-ready results.

Stagehand

Automate Web Workflows With AI

Stars
23.3K
Forks
1.6K
Last commit
2 months ago

Open-source TypeScript framework for building AI-assisted browser automations that combine plain-language steps with Playwright code.

Browserbase

Reliable Cloud Browsers for AI Agents

Headless Browsers

Run hosted browser sessions for AI automation and scraping with search, fetch, stealth, observability, and replay built in.

Browse AI

No-code AI scraping and monitoring for any site

AI Web Scraping

Extract and monitor data from any website without code, turn pages into spreadsheets or APIs, and connect to thousands of apps.

ParseHub

Free visual scraper for dynamic, interactive sites

Web Scraping

Extract data from dynamic websites by clicking elements, handle AJAX, forms, and dropdowns, then export to JSON, Excel, or API.

Octoparse

Turn web pages into structured data, no code

Web Scraping

Build scrapers visually and turn web pages into structured data in minutes, with scheduling and an optional managed data service.

Perplexity Sonar

Generative AI Search API with Real-Time Research

SERP APIs

Perplexity Sonar is a generative AI search API that provides real-time web research with citations, customizable sources, and multiple performance tiers.

ValueSERP

Low-Cost Google SERP API for Data Collection

SERP APIs

ValueSERP provides real-time, reliable Google search data at the best value with rates as low as $0.50 per 1,000 searches and minimal maintenance.

Browse 8 subcategories of Web Scraping tools and find the perfect solution for your needs.

Quick answer

Related guides

Top picks in Web Scraping

Related categories