Task Queues
A curated collection of the best distributed task queue systems for managing background jobs and parallel processing workloads.
Quick answer
Send a crawl or processing job to a queue, let a worker execute it, and collect the result. Celery fits Python workloads, RQ fits simple Redis-backed Python jobs, and BullMQ fits Node.js workers using Redis. Choose Workflow Orchestration when the decision is about coordinating a multi-step process and its durable state.
Frequently asked questions
What is a task queue?
A task queue sends units of work to separate workers. For example, Celery uses a message broker to deliver tasks that worker processes execute outside the application’s main request flow.
Sources (checked ): Celery introduction
What is the difference between a message queue and a task queue?
A message queue transports messages between components. A task queue adds a model for executing work: in Celery, messages describe tasks and workers run them. A broker and a task execution framework therefore have different roles.
Sources (checked ): Celery introduction