Crawler-Commons

Crawler-Commons is an open-source, actively maintained Java library and a library for sitemap parsing on the JVM. Its SAX-based parser handles sitemaps, indexes, extensions, and syndication feeds.

Key Features:

• SAX Parser - SiteMapParser handles sitemaps, sitemap indexes, extensions, and feed formats.
• Bring Your Own Fetch - Since version 0.8 there is no built-in HTTP fetcher; you supply the bytes.
• Normalization - Parses and normalizes the URLs it extracts.
• Robots Utilities - Also provides robots.txt parsing for crawler infrastructure.
• Actively Maintained - Provides sitemap and robots.txt utilities for JVM crawler stacks.

Choose Crawler-Commons when a Java crawler needs robust sitemap and robots.txt parsing and controls its own fetching.