Nutch

Apache Nutch is a highly extensible, production-ready web crawler that leverages Apache Hadoop for scalable, distributed batch crawling operations.

Key Features:

• Hadoop Integration — Built on Hadoop for distributed crawling across large clusters
• Pluggable Architecture — Extensible parsing, indexing, and scoring via plugins
• Scalable — Handles web-scale crawling with horizontal scaling across nodes
• Search Integration — Native integration with Apache Solr and Elasticsearch for indexing
• Mature — Battle-tested in production environments for over two decades
• Fine-Grained Config — Configurable URL filtering, fetch scheduling, and politeness

Whether you're building search engines, creating web archives, or running large-scale data acquisition, Apache Nutch provides a proven, extensible crawler for distributed environments.