Luigi
Luigi is a Python package by Spotify for building complex pipelines of batch jobs with dependency resolution, workflow management, and Hadoop support.
Luigi is a lightweight, reliable Python framework that helps you build complex pipelines of batch jobs with automatic dependency resolution. Created by Spotify, Luigi handles the plumbing of long-running data processes so you can focus on the tasks themselves.
Key Features:
- Dependency Resolution - Automatically manages task dependencies and execution order
- Workflow Visualization - Built-in UI for monitoring pipeline progress and failures
- Hadoop Integration - Native support for HDFS, Hive, Pig, and Spark jobs
- Atomic File Operations - Ensures data pipeline integrity with atomic file system abstractions
- Flexible Tasks - Chain any task type from database dumps to ML training to web scraping
Whether you're a data engineer, analytics team, or batch processing specialist, Luigi provides a simple framework to orchestrate reliable data pipelines.
Web scraping workflow
How to use Luigi in a web scraping pipeline
Luigi connects batch tasks through their dependencies and output targets. In web scraping, it can express the relationship between a collected snapshot, a parsed export and the table produced from that export.
A practical workflow
- Create a task whose code invokes a separate scraper and writes a dated capture manifest.
- Make a parsing task require that manifest, then produce a validated file as its output target.
- Let a loading task depend on the validated file and publish a completion marker only after the destination write succeeds.
When it fits
Use Luigi for file-oriented batch collection where downstream jobs should wait for specific upstream outputs. A clear target naming scheme helps distinguish today's crawl from an older artifact that happens to exist.
Boundaries
Luigi stitches together the tasks you implement; it does not replace the HTTP client, parser or storage system. Define completion carefully: an existing file is not necessarily fresh or correct. Use versioned targets, avoid partial output publication and make destination writes safe when a task is rerun.
Growing a website? Track its keywords, Google rankings, Search Console insights, and share of voice with bisibility. Free during beta.
Our productRepository
spotify/luigi
Stars
18.8KForks
2.5KLast commit
2026-10-07License
Apache-2.0Language
Python
Metrics recorded 2026-10-16
Repository details
Last push: 2026-10-14
License source: Repository license file
Sources
Verified against the spotify/luigi repository and the PyPI package.