What is a sitemap scraper and why use one?

A sitemap scraper is a tool that reads a site's XML sitemaps to produce the list of URLs the site publishes. Teams use it for discovery, auditing, and seeding scraping pipelines before they spend budget on a broader crawl.

How do I extract all URLs from a sitemap?

Read robots.txt Sitemap entries, try common paths like /sitemap.xml and /sitemap_index.xml, recurse sitemap index files down to child sitemaps, then dedupe the collected URLs before exporting them.

How is a sitemap crawler different from a full site crawler?

A sitemap scraper reads only the URLs the site declares, which is fast and cheap. A full crawl follows links and can find pages the sitemap omits, plus orphan pages that a sitemap only workflow may miss, but it costs more to run.

How should sitemap index files and .xml.gz files be handled?

A sitemap index points at child sitemaps that you fetch and parse in turn. Many sitemaps are gzip-compressed as .xml.gz files, so the tool must decompress the response before parsing the XML.

How do tools auto-discover sitemaps for a domain?

Most tools check robots.txt first, then try common paths. Some also brute-force known URL patterns or read HTML meta hints, and the DuckDB sitemap community extension is a niche SQL option for querying sitemaps directly.

Best Sitemap Scrapers and Parsers

Tools in this guide

Published profiles that match this guide.

Browse all Sitemaps tools

Sitebulb

SEO Audits With Sitemap Discovery And Exports

Sitemap Scrapers

Desktop and cloud SEO auditing tool that discovers XML sitemaps, accepts a sitemap index as a crawl source, and exports URLs to CSV or Google Sheets.

Screaming Frog SEO Spider

Desktop SEO Crawler With Sitemap Auditing

Sitemap Scrapers

Desktop SEO crawler that can crawl linked XML sitemaps, download a sitemap in List Mode to crawl only its URLs, and export flat URL lists to CSV.

Ben Balter sitemap-parser

Simple Sitemaps.org XML Parsing For Ruby

Stars
33
Forks
47
Last commit
last year

Ruby parser for sitemaps.org-compliant XML sitemaps; a recurse option flattens nested indexes into one URL list returned by to_a.

TurnerSoftware SitemapTools

Discover And Parse A Domain's Sitemaps In C#

Stars
38
Forks
10
Last commit
2 years ago

C# library whose GetAllSitemapsForDomainAsync() takes a domain and discovers plus parses all its sitemaps, with lenient parsing for messy real-world files.

VIPnytt SitemapParser

Recursive Sitemap Parsing For PHP

Stars
78
Forks
33
Last commit
9 months ago

PHP library with parse() and parseRecursive() APIs; parseRecursive on a robots.txt URL walks every nested sitemap for a complete URL list.

Crawler-Commons

The De Facto Java Sitemap Parsing Library

Stars
259
Forks
91
Last commit
15 days ago

Java library with a SAX-based sitemap parser for sitemaps, indexes, extensions, and feeds; you supply the bytes and it parses and normalizes them.

gopher-parse-sitemap

Zero-Dependency Streaming Sitemap Parser In Go

Stars
40
Forks
19
Last commit
3 years ago

Streaming Go parser for huge XML sitemaps with minimal memory and no external dependencies, consuming entries on the fly via callbacks.

go-sitemap-parser

Resolve Robots, Indexes, And Sitemaps In Go

Stars
6
Forks
1
Last commit
2 months ago

Go library whose Parse() accepts a robots.txt, sitemap index, or plain sitemap URL and resolves the rest, with concurrent fetching and depth limits.

sitemap-stream-parser

Stream Very Large Sitemaps With Low Memory

Stars
39
Forks
19
Last commit
4 years ago

Streaming Node.js XML sitemap parser for very large, deeply nested sitemaps, with a callback API and a helper that finds sitemaps in robots.txt.

Sitemapper

Parse Sitemaps And Indexes In Node.js

Stars
135
Forks
83
Last commit
last month

Node.js library that parses a sitemap URL including sitemap indexes with many child sitemaps, with concurrency, retries, timeout, and proxy options.

Ultimate Sitemap Parser

Auto-Discover And Parse Sitemaps In Python

Stars
242
Forks
74
Last commit
5 months ago

Python library that finds a site's sitemaps from robots.txt and common paths, recurses sitemap indexes, and streams pages with low memory.

Crawlee

Build reliable web scrapers. Fast.

Stars
21.3K
Forks
1.2K
Last commit
5 months ago

Crawlee is a JavaScript and Python web scraping library that handles blocking, proxies, and browsers so you can build reliable crawlers fast.

How to choose

Best for

Teams that need a complete URL list before a crawl.
SEO and site audits comparing declared URLs against the live site.
Discovery pipelines that seed a scraper from a domain's sitemaps.

What to compare

Whether it auto-discovers sitemaps from robots.txt and common paths.
Whether it recurses sitemap index files and handles gzipped .xml.gz files.
Memory behavior on very large sitemaps, especially streaming instead of loading all URLs at once.
Output format and how easily URLs export to CSV or JSON.

Before you commit

Check whether the tool fetches pages itself or expects you to supply the bytes.
Check the license and language ecosystem before adopting it.

Best Sitemap Scrapers and Parsers

Browse sitemap scrapers and parsers that crawl XML sitemaps and sitemap indexes to extract URLs for discovery, auditing, and scraping pipelines.

Quick answer