Scrapy Cluster

Scrapy Cluster is a distributed, on-demand web scraping system that combines Scrapy with Redis and Kafka for coordinated, large-scale crawling operations.

Key Features:

• Kafka Integration — API-driven crawl requests via Kafka for real-time, on-demand scraping
• Redis Coordination — Distributed request management and deduplication across nodes
• Horizontal Scaling — Add crawler nodes dynamically to increase throughput
• Kafka Monitor — REST API for submitting and managing crawl jobs programmatically
• Modular Design — Pluggable components for customizing crawl behavior and data flow

Whether you're building on-demand scraping services, scaling data collection infrastructure, or coordinating crawlers across a cluster, Scrapy Cluster provides a proven architecture for distributed Scrapy deployments.