Nutch

Apache Nutch is a highly extensible, production-ready web crawler that leverages Apache Hadoop for scalable, distributed batch crawling operations.

Key Features:

• Hadoop Integration - Built on Hadoop for distributed crawling across large clusters
• Pluggable Architecture - Extensible parsing, indexing, and scoring via plugins
• Scalable - Handles web-scale crawling with horizontal scaling across nodes
• Search Integration - Native integration with Apache Solr and Elasticsearch for indexing
• Mature - Battle-tested in production environments for over two decades
• Fine-Grained Config - Configurable URL filtering, fetch scheduling, and politeness

Whether you're building search engines, creating web archives, or running large-scale data acquisition, Apache Nutch provides a proven, extensible crawler for distributed environments.