Heritrix

PublishedJun 12, 2026Last updatedOct 16, 2026

Heritrix is the Internet Archive's open-source, extensible, archival-quality web crawler designed for large-scale web preservation and data collection.

Visit Heritrix

Heritrix is an extensible, archival-quality web crawler built by the Internet Archive for large-scale web preservation and data collection.

Key Features:

Web-Scale Crawling - Designed to crawl billions of pages for web archival projects
WARC Output - Produces standard WARC files for long-term web preservation
Fine-Grained Control - Extensive configuration for crawl scope, politeness, and prioritization
Web UI - Browser-based interface for monitoring and controlling active crawls
Docker Support - Run Heritrix in containers for reproducible crawl environments
Extensible - Plugin system for custom processors, extractors, and modules

Whether you're preserving web content for research, building large-scale web archives, or conducting comprehensive domain crawls, Heritrix provides production-grade crawling trusted by the Internet Archive.

Web scraping workflow

How to use Heritrix in a web scraping pipeline

Heritrix is an archival web crawler. Its useful output for a scraping system is a retained collection of fetched material that can be processed later, rather than a ready-made table of product prices or article fields.

A practical workflow

Configure a crawl job with seed URLs, scope rules, limits and a crawl policy suited to the collection you want to preserve.
Run the job and retain its WARC output together with crawl logs and job metadata. Review failures and exclusions instead of assuming every discovered URL was captured.
Process the retained archive with a separate parser or extraction job, then load the resulting records into a database, search index or analytical dataset.

When it fits

It fits preservation, repeatable offline extraction and workflows where keeping the original response matters as much as extracting today's fields. Multiple downstream extractors can work from the same retained capture.

Boundaries

A WARC archive is not a structured business dataset. Interactive application behavior and browser-only content need separate consideration. Storage capacity, crawl scope and revisit policy remain operational decisions, and a retained response may still require content-type-specific parsing or document extraction.

Categories:

Large-Scale & Distributed Crawling

Features:

Distributed CrawlingRate LimitingSelf-HostedOpen SourceWARC OutputWeb UIPlugin System

Tags:

open-source