Scrapy Cluster

Scrapy Cluster is a distributed, on-demand web scraping system that combines Scrapy with Redis and Kafka for coordinated, large-scale crawling operations.

Key Features:

• Kafka Integration - API-driven crawl requests via Kafka for real-time, on-demand scraping
• Redis Coordination - Distributed request management and deduplication across nodes
• Horizontal Scaling - Add crawler nodes dynamically to increase throughput
• Kafka Monitor - REST API for submitting and managing crawl jobs programmatically
• Modular Design - Pluggable components for customizing crawl behavior and data flow

Whether you're building on-demand scraping services, scaling data collection infrastructure, or coordinating crawlers across a cluster, Scrapy Cluster provides a proven architecture for distributed Scrapy deployments.