Heritrix

Heritrix is an extensible, archival-quality web crawler built by the Internet Archive for large-scale web preservation and data collection.

Key Features:

• Web-Scale Crawling — Designed to crawl billions of pages for web archival projects
• WARC Output — Produces standard WARC files for long-term web preservation
• Fine-Grained Control — Extensive configuration for crawl scope, politeness, and prioritization
• Web UI — Browser-based interface for monitoring and controlling active crawls
• Docker Support — Run Heritrix in containers for reproducible crawl environments
• Extensible — Plugin system for custom processors, extractors, and modules

Whether you're preserving web content for research, building large-scale web archives, or conducting comprehensive domain crawls, Heritrix provides production-grade crawling trusted by the Internet Archive.