[opensourcedex]

luigi is a free, open source scheduling & event management project written in Python and released under Apache-2.0. It has 18,775 GitHub stars, 2,463 forks and 175 open issues, and was last pushed 2 months ago. On this registry it ranks #2 of 23 tracked projects in Scheduling & Event Management, with 5 head-to-head comparisons available. It gained 1 stars over the last 3 tracked days.

What is luigi?

Luigi is a Python module for building complex pipelines of batch jobs, handling dependency resolution, workflow management, visualization, and failure handling, aimed at data engineers and teams who need to stitch together long-running tasks such as Hadoop, Hive, Pig, Spark, or local Python jobs into repeatable workflows.

What it is

Luigi is an open-source Python package, maintained by Spotify under the Apache-2.0 licence, that addresses the plumbing typically associated with long-running batch processes. It lets you chain many tasks together, automate them, and cope with the failures that will inevitably occur along the way. The tasks themselves can be almost anything, but in practice they are long-running operations such as Hadoop jobs, dumps of data to and from databases, or machine learning algorithms. Luigi resolves the dependencies between those tasks, tracks which have completed and which have not, and exposes the whole workflow through a web interface for searching and filtering.

The concrete problem it solves is orchestration rather than data processing. It is not a replacement for lower-level tools such as Hive, Pig, or Cascading; instead it sits above them and stitches their work together, where each task can be a Hive query, a Hadoop job written in Java, a Spark job in Scala or Python, a Python snippet, or a database table dump. It replaces hand-rolled glue code and cron-driven chains with an explicit dependency graph, so that pipelines comprising thousands of tasks and running for days or weeks can be managed without losing track of state. Its file system abstractions for HDFS and local files ensure that file system operations are atomic, which means a pipeline will not crash in a state containing partial data.

Key capabilities

  • Dependency resolution across task graphs, so each task declares what it depends on and Luigi determines execution order.
  • Built-in Hadoop support, including task templates for running Python mapreduce jobs, Hive jobs, and Pig jobs.
  • File system abstractions for HDFS and local files that keep all file system operations atomic and avoid partial data.
  • Web visualiser server that lets you search and filter among all tasks, with a dependency graph view where completed tasks appear green and pending tasks yellow.
  • Command line integration, making pipelines scriptable and callable from other tooling.
  • Failure handling for long-running batch processes, since failures are treated as expected rather than exceptional.
  • Optional TOML-based configuration, installed through the luigi[toml] extra.

Who uses it and how

  • Data engineering teams running production pipelines where most tasks are Hadoop jobs alongside some tasks that run locally and build up data files.
  • Teams chaining heterogeneous work: a Hive query, a Hadoop job in Java, a Spark job in Scala or Python, a Python snippet, and a database dump can all sit in one dependency graph.
  • Operations that need long-running pipelines of thousands of tasks taking days or weeks, where dependency tracking matters more than raw task throughput.
  • Groups that need visibility into workflow progress, using the visualiser to inspect the dependency graph and locate unfinished or failing nodes.
  • Anyone already running Hive, Pig, or Cascading who wants an orchestration layer above those tools rather than inside them.

Getting started

Install the latest stable release from PyPI with pip install luigi, or run pip install luigi[toml] to add TOML-based configuration support. Bleeding-edge code can be installed with pip install git+https://github.com/spotify/luigi.git.

How it compares

Conceptually Luigi is similar to GNU Make, where tasks have dependencies on other tasks, and it shares features with Oozie and Azkaban; one major difference is that Luigi is not built specifically for Hadoop and is easy to extend with other kinds of work. It is explicitly not a replacement for lower-level data processing packages such as Hive, Pig, or Cascading, and should be understood as the layer that stitches those tools together rather than a substitute for them.

When to use it — and when not to

Luigi suits teams that already run batch processing tools and need a scheduler and dependency manager above them, and who can operate the Luigi server and its workflow daemon as long-lived infrastructure. It is a poor fit for anyone looking for a data processing engine in its own right, since it deliberately does not replace Hive, Pig, or Cascading, nor is it designed solely around Hadoop. Luigi is tested against Python 3.10 through 3.14, so older Python runtimes are not supported.

project readme (upstream, from github) — read inline

.. figure:: https://raw.githubusercontent.com/spotify/luigi/master/doc/luigi.png :alt: Luigi Logo :align: center

.. image:: https://img.shields.io/endpoint.svg?url=https%3A%2F%2Factions-badge.atrox.dev%2Fspotify%2Fluigi%2Fbadge&label=build&logo=none&%3Fref%3Dmaster&style=flat :target: https://actions-badge.atrox.dev/spotify/luigi/goto?ref=master

.. image:: https://img.shields.io/codecov/c/github/spotify/luigi/master.svg?style=flat :target: https://codecov.io/gh/spotify/luigi?branch=master

.. image:: https://img.shields.io/pypi/v/luigi.svg?style=flat :target: https://pypi.python.org/pypi/luigi

.. image:: https://img.shields.io/pypi/l/luigi.svg?style=flat :target: https://pypi.python.org/pypi/luigi

.. image:: https://readthedocs.org/projects/luigi/badge/?version=stable :target: https://luigi.readthedocs.io/en/stable/?badge=stable :alt: Documentation Status

Luigi is a Python (3.10, 3.11, 3.12, 3.13, 3.14 tested) package that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization, handling failures, command line integration, and much more.

Getting Started

Run pip install luigi to install the latest stable version from PyPI . Documentation for the latest release _ is hosted on readthedocs.

Run pip install luigi[toml] to install Luigi with TOML-based configs __ support.

For the bleeding edge code, pip install git+https://github.com/spotify/luigi.git. Bleeding edge documentation __ is also available.

Background

The purpose of Luigi is to address all the plumbing typically associated with long-running batch processes. You want to chain many tasks, automate them, and failures will happen. These tasks can be anything, but are typically long running things like Hadoop _ jobs, dumping data to/from databases, running machine learning algorithms, or anything else.

There are other software packages that focus on lower level aspects of data processing, like Hive , Pig , or Cascading . Luigi is not a framework to replace these. Instead it helps you stitch many tasks together, where each task can be a Hive query , a Hadoop job in Java , a Spark job in Scala or Python , a Python snippet, dumping a table _ from a database, or anything else. It's easy to build up long-running pipelines that comprise thousands of tasks and take days or weeks to complete. Luigi takes care of a lot of the workflow management so that you can focus on the tasks themselves and their dependencies.

You can build pretty much any task you want, but Luigi also comes with a toolbox of several common task templates that you use. It includes support for running Python mapreduce jobs _ in Hadoop, as well as Hive , and Pig , jobs. It also comes with file system abstractions for HDFS _, and local files that ensures all file system operations are atomic. This is important because it means your data pipeline will not crash in a state containing partial data.

Visualiser page

The Luigi server comes with a web interface too, so you can search and filter among all your tasks.

.. figure:: https://raw.githubusercontent.com/spotify/luigi/master/doc/visualiser_front_page.png :alt: Visualiser page

Dependency graph example

Just to give you an idea of what Luigi does, this is a screen shot from something we are running in production. Using Luigi's visualiser, we get a nice visual overview of the dependency graph of the workflow. Each node represents a task which has to be run. Green tasks are already completed whereas yellow tasks are yet to be run. Most of these tasks are Hadoop jobs, but there are also some things that run locally and build up data files.

.. figure:: https://raw.githubusercontent.com/spotify/luigi/master/doc/user_recs.png :alt: Dependency graph

Philosophy

Conceptually, Luigi is similar to GNU Make _ where you have certain tasks and these tasks in turn may have dependencies on other tasks. There are also some similarities to Oozie _ and Azkaban _. One major difference is that Luigi is not just built specifically for Hadoop, and it's easy to extend it with other kinds of tasks.

Everything in Luigi is in Python. Instead of XML configuration or similar external data files, the dependency graph is specified within Python. This makes it easy to build up complex dependency graphs of tasks, where the dependencies can involve date algebra or recursive references to other versions of the same task. However, the workflow can trigger things not in Python, such as running Pig scripts _ or scp'ing files _.

Who uses Luigi?

We use Luigi internally at Spotify _ to run thousands of tasks every day, organized in complex dependency graphs. Most of these tasks are Hadoop jobs. Luigi provides an infrastructure that powers all kinds of stuff including recommendations, toplists, A/B test analysis, external reports, internal dashboards, etc.

Since Luigi is open source and without any registration walls, the exact number of Luigi users is unknown. But based on the number of unique contributors, we expect hundreds of enterprises to use it. Some users have written blog posts or held presentations about Luigi:

  • Spotify _ (presentation, 2014) __
  • Foursquare _ (presentation, 2013) __
  • Mortar Data (Datadog) _ (documentation / tutorial) __
  • Stripe _ (presentation, 2014) __
  • Buffer _ (blog, 2014) __
  • SeatGeek _ (blog, 2015) __
  • Treasure Data _ (blog, 2015) __
  • Growth Intelligence _ (presentation, 2015) __
  • AdRoll _ (blog, 2015) __
  • 17zuoye (presentation, 2015) __
  • Custobar _ (presentation, 2016) __
  • Blendle _ (presentation) __
  • TrustYou _ (presentation, 2015) __
  • Groupon _ / OrderUp _ (alternative implementation) __
  • Red Hat - Marketing Operations _ (blog, 2017) __
  • GetNinjas _ (blog, 2017) __
  • voyages-sncf.com _ (presentation, 2017) __
  • Open Targets _ (blog, 2017) __
  • `Leipzi

readme truncated — read the full docs on github

Frequently asked questions

Is luigi free to use?

luigi is open source under the Apache-2.0 licence. There is no licence fee and no seat count — you can self-host it or, where the project offers one, pay a vendor for a managed version instead.

What does luigi do?

Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also

What is luigi written in?

luigi is primarily written in Python. Its source is publicly available at https://github.com/spotify/luigi, and it has 18,775 GitHub stars.