Scrapy Cluster
Scrapy Cluster uses Redis and Kafka to create a distributed, on-demand Scrapy crawling cluster for coordinated large-scale web scraping. Its repository is archived.
Scrapy Cluster is a distributed, on-demand web scraping system that combines Scrapy with Redis and Kafka for coordinated, large-scale crawling operations.
Key Features:
- Kafka Integration - API-driven crawl requests via Kafka for real-time, on-demand scraping
- Redis Coordination - Distributed request management and deduplication across nodes
- Horizontal Scaling - Add crawler nodes dynamically to increase throughput
- Kafka Monitor - REST API for submitting and managing crawl jobs programmatically
- Modular Design - Pluggable components for customizing crawl behavior and data flow
Whether you're building on-demand scraping services, scaling data collection infrastructure, or coordinating crawlers across a cluster, Scrapy Cluster provides a proven architecture for distributed Scrapy deployments.
Maintenance: The repository was archived on 2026-08-04 and is read-only. Check its older dependency requirements before using it in a new deployment.
Web scraping workflow
How to use Scrapy Cluster in a web scraping pipeline
Scrapy Cluster documents a distributed Scrapy architecture with Kafka inputs and outputs and Redis-coordinated requests. Its upstream repository is archived, so this pattern is relevant to understanding or migrating an existing installation rather than recommending a new deployment.
A practical workflow
- Identify the job messages entering Kafka and the Redis state used to coordinate the waiting spider instances. Record job identifiers and the expected output topics.
- Trace how a submitted seed expands into crawl requests across workers, including the existing scope and throttling configuration.
- Preserve the output contract and crawl-state requirements when moving collection jobs to a maintained stack, and compare results using a bounded set of representative jobs.
When it fits
It fits maintenance and migration of an inherited deployment whose producers and consumers already depend on these message flows. The architecture can also inform a design review without requiring the archived implementation to be installed.
Boundaries
The repository is read-only, and its documented dependency stack is historical. Do not assume compatibility with current Scrapy, Python or infrastructure releases. An existing Kafka message is not proof of a completed or correctly parsed collection job.
Growing a website? Track its keywords, Google rankings, Search Console insights, and share of voice with bisibility. Free during beta.
Our productRepository
istresearch/scrapy-cluster
Stars
1.2KForks
317Last commit
2021-04-07License
MITLanguage
Python
Metrics recorded 2026-10-16
Repository details
Last push: 2023-11-07
License source: Repository license file
Sources
The repository was archived on 2026-08-04 and is read-only. Check its older dependency requirements before using it in a new deployment. Status checked on 2026-10-01.