Large-Scale & Distributed Crawling
A curated collection of the best crawlers, shared scheduling and crawl-frontier tools for large-scale web collection. Multi-server support depends on the implementation.
Quick answer
Start with seed URLs or a crawl frontier, schedule requests, and collect pages across a large crawl. Scrapy-Redis adds shared queues and deduplication to Scrapy, Nutch supports batch crawling, Heritrix produces web archives, and Frontera manages the crawl frontier. Plain Scrapy has no built-in multi-server crawl mechanism; find it under Scraping Frameworks. Scrapy Cluster is retained as an archived project, not a default choice for a new deployment.
Frequently asked questions
Is there an open-source web crawler?
Yes. Apache Nutch is an Apache 2.0 licensed crawler designed around Hadoop data structures and batch processing. It supports extensions for fetching, parsing and indexing. Running it still requires infrastructure and a crawl configuration suited to your workload.
Sources (checked ): Apache Nutch overview; Apache Nutch repository and license
What is indexing vs. crawling?
Crawling discovers URLs and retrieves their content. Indexing analyzes retrieved content and stores a representation for search. Google documents these as separate stages and does not guarantee that every discovered page will be crawled or indexed. A crawler does not automatically provide a search index.
Sources (checked ): Google Search: crawling and indexing
