Find a site's page URLs without crawling every page. Discover sitemaps from robots.txt, follow sitemap indexes and export a clean URL list with last-modified dates.
Exports retain the page URL, source sitemap and available lastmod, changefreq and priority values, along with discovery method and depth. Compressed gzip sitemaps are supported. No account, API key or paid scraping service is required.
The report highlights duplicate URLs, invalid XML, failed child sitemaps, unusual last-modified dates and URLs on other hosts. Useful results remain available when part of an index fails, with those errors included in the report.
A run collects up to 5,000 URLs from at most 30 documents and four levels of nested indexes. If a limit is reached, the result is marked partial. This tool reads sitemap documents; it does not crawl every listed page, check each page's HTTP status or determine Google indexing.
For larger or scheduled workflows, browse sitemap parsers and scrapers. To inspect robots.txt rules for AI crawlers, use the AI crawler checker.