Best Free Web Scraping Tools

Ben Balter sitemap-parser

Simple Sitemaps.org XML Parsing For Ruby

Stars
33
Forks
47
Last commit
last year

Ruby parser for sitemaps.org-compliant XML sitemaps; a recurse option flattens nested indexes into one URL list returned by to_a.

TurnerSoftware SitemapTools

Discover And Parse A Domain's Sitemaps In C#

Stars
38
Forks
10
Last commit
2 years ago

C# library whose GetAllSitemapsForDomainAsync() takes a domain and discovers plus parses all its sitemaps, with lenient parsing for messy real-world files.

VIPnytt SitemapParser

Recursive Sitemap Parsing For PHP

Stars
78
Forks
33
Last commit
9 months ago

PHP library with parse() and parseRecursive() APIs; parseRecursive on a robots.txt URL walks every nested sitemap for a complete URL list.

Crawler-Commons

The De Facto Java Sitemap Parsing Library

Stars
259
Forks
91
Last commit
15 days ago

Java library with a SAX-based sitemap parser for sitemaps, indexes, extensions, and feeds; you supply the bytes and it parses and normalizes them.

gopher-parse-sitemap

Zero-Dependency Streaming Sitemap Parser In Go

Stars
40
Forks
19
Last commit
3 years ago

Streaming Go parser for huge XML sitemaps with minimal memory and no external dependencies, consuming entries on the fly via callbacks.

go-sitemap-parser

Resolve Robots, Indexes, And Sitemaps In Go

Stars
6
Forks
1
Last commit
2 months ago

Go library whose Parse() accepts a robots.txt, sitemap index, or plain sitemap URL and resolves the rest, with concurrent fetching and depth limits.

sitemap-stream-parser

Stream Very Large Sitemaps With Low Memory

Stars
39
Forks
19
Last commit
4 years ago

Streaming Node.js XML sitemap parser for very large, deeply nested sitemaps, with a callback API and a helper that finds sitemaps in robots.txt.

Sitemapper

Parse Sitemaps And Indexes In Node.js

Stars
135
Forks
83
Last commit
last month

Node.js library that parses a sitemap URL including sitemap indexes with many child sitemaps, with concurrency, retries, timeout, and proxy options.

Ultimate Sitemap Parser

Auto-Discover And Parse Sitemaps In Python

Stars
242
Forks
74
Last commit
5 months ago

Python library that finds a site's sitemaps from robots.txt and common paths, recurses sitemap indexes, and streams pages with low memory.

Requests-HTML

Python Requests With HTML Parsing

Stars
13.8K
Forks
999
Last commit
2 years ago

Python library combining HTTP requests, HTML parsing, CSS selectors, and optional JavaScript rendering for scraping workflows.

pyquery

jQuery-Style HTML Parsing For Python

Stars
2.4K
Forks
184
Last commit
4 months ago

Python library that applies a jQuery-style API to HTML and XML documents for selection, traversal, and extraction.

Helium

Readable Selenium Automation For Python

Stars
8.3K
Forks
513
Last commit
17 days ago

Python browser automation library that wraps Selenium with a higher-level API for clicking, typing, and finding visible elements.

AutoScraper

Example-Based HTML Extraction For Python

Stars
7.5K
Forks
764
Last commit
last year

Python library that learns simple scraping rules from example values and extracts similar data from HTML pages.

Pyppeteer

Python Control For Headless Chromium

Stars
4K
Forks
347
Last commit
2 years ago

Python library for controlling Chrome and Chromium through a Puppeteer-like API for screenshots, automation, and scraping.

pgvector

Add Vector Search To PostgreSQL

Stars
21.9K
Forks
1.2K
Last commit
23 days ago

Open-source PostgreSQL extension for vector similarity search that keeps embeddings and scraped data in the same database.

jsoup

Parse Real-World HTML In Java

Stars
11.4K
Forks
2.3K
Last commit
22 days ago

A Java library for parsing real-world HTML into a standards-based DOM with CSS selectors and DOM traversal.

lxml

Fast XML And HTML Parsing For Python

Stars
3K
Forks
619
Last commit
22 days ago

A fast Python library for parsing XML and HTML, often used as a foundation for scraping and data tools.

ScraperAPI

Scale data collection with a simple API

Stars
2
Forks
1
Last commit
2 years ago

ScraperAPI handles proxy rotation, browsers, and CAPTCHAs so developers can scrape any public page with a single API call at scale.

ScrapingBee

Web scraping API with proxy and browser handling

Stars
29
Forks
4
Last commit
5 months ago

ScrapingBee is a web scraping API that handles headless browsers, rotates proxies, and offers AI-powered extraction so you can focus on your data.

DataForSEO

Comprehensive API Stack for SEO and Marketing Data

Stars
39
Forks
12
Last commit
5 months ago

DataForSEO provides SEO and digital marketing data via API, serving as a trusted data partner for 750+ software companies and agencies worldwide.

CapSolver

AI-powered automatic CAPTCHA solving service

Stars
63
Forks
10
Last commit
last year

CapSolver is an AI-powered CAPTCHA solver that automatically bypasses reCAPTCHA, Cloudflare, AWS WAF, and other challenges for automation.

Oxylabs

Premium proxy service for data at scale

Stars
93
Forks
5
Last commit
6 months ago

Oxylabs offers 175M+ residential and datacenter proxies with web scraping APIs and browser automation tools for large-scale public data extraction.

SerpAPI

Real-Time Google Search Results API

Stars
116
Forks
11
Last commit
7 months ago

SerpAPI is a real-time API that delivers structured Google search results, handling proxies, captchas, and rich data parsing automatically.

Apify

Full-stack web scraping and data extraction platform

Stars
161
Forks
21
Last commit
5 months ago

Apify is a cloud platform for web scraping, browser automation, and data extraction with 19,000+ ready-made tools and code templates.

BeautifulSoup

Python library for parsing HTML and XML

Stars
204
Forks
57
Last commit
3 years ago

Beautiful Soup is a Python library for pulling data out of HTML and XML files, providing Pythonic idioms for navigating and searching parse trees.

2Captcha

CAPTCHA solving service with API integration

Stars
714
Forks
126
Last commit
5 months ago

2Captcha is a CAPTCHA solving service that bypasses reCAPTCHA, Cloudflare Turnstile, hCaptcha, and other challenges via a simple API.

Zendriver

Async-first undetected browser automation

Stars
1.1K
Forks
75
Last commit
7 months ago

Zendriver is a fast, async-first undetected browser automation framework based on nodriver with Docker support for scalable scraping.

Scrapy Cluster

Distributed on-demand scraping with Scrapy

Stars
1.2K
Forks
322
Last commit
2 years ago

Scrapy Cluster uses Redis and Kafka to create a distributed, on-demand Scrapy crawling cluster for coordinated large-scale web scraping.

Frontera

Scalable crawl frontier framework

Stars
1.3K
Forks
217
Last commit
last year

Frontera is a Python crawl frontier framework for managing when and what to crawl, enabling web crawlers of any scale with Scrapy integration.

Patchright

Undetected Playwright fork for bot evasion

Stars
2.2K
Forks
88
Last commit
5 months ago

Patchright is a patched fork of Playwright that evades bot detection while maintaining full API compatibility for seamless migration.

Meltano

CLI-First ELT Engine Built on Singer

Stars
2.3K
Forks
0

Meltano is an open-source, CLI-first ELT platform with 600+ connectors, GitOps workflows, and native dbt integration for data pipeline management.

Scrapoxy

End-of-life proxy orchestration project

Stars
2.4K
Forks
0

Scrapoxy is an open-source proxy orchestration project that reached End of Life as announced on scrapoxy.io, so it is historical context rather than a new-deployment choice.

BeeAI Framework

Production-Ready AI Agent Framework

Stars
3.1K
Forks
400
Last commit
5 months ago

BeeAI Framework is an open-source platform for building production-ready multi-agent systems in Python and TypeScript with MCP and workflow support.

Nutch

Highly extensible and scalable web crawler

Stars
3.1K
Forks
1.3K
Last commit
5 months ago

Apache Nutch is a highly extensible, production-ready web crawler built on Hadoop for large-scale batch crawling and data acquisition tasks.

Heritrix

Web-scale archival web crawler

Stars
3.2K
Forks
782
Last commit
6 months ago

Heritrix is the Internet Archive's open-source, extensible, archival-quality web crawler designed for large-scale web preservation and data collection.

Nodriver

Undetected browser automation without Selenium

Stars
3.5K
Forks
341
Last commit
8 months ago

Nodriver is the successor to undetected-chromedriver, providing fast CDP-based browser automation with built-in anti-detection and no Selenium dependency.