Ultimate Sitemap Parser

Ultimate Sitemap Parser is an open-source Python library for discovering and parsing website sitemaps. Give it a homepage URL and it locates the sitemaps, walks nested indexes, and yields pages without loading everything into memory.

Key Features:

• Auto-Discovery - Finds sitemaps via robots.txt and common locations from just a homepage URL.
• Recursive Indexes - Traverses sitemap index files down to their child sitemaps.
• Format Coverage - Parses XML sitemaps, Google News and Image extensions, plain text, RSS, and Atom.
• Low Memory - A lazily-loaded all_pages() generator keeps large sitemaps memory-light.
• Custom Clients - Supports a custom web client for retries, headers, or proxies.

Choose Ultimate Sitemap Parser when a Python project needs to turn a domain into a URL list discovered from the available sitemaps.