Dagster is a cloud-native data pipeline orchestrator written in Python and released under the Apache-2.0 licence, built for data engineers, data scientists, and machine learning teams who declare the tables, data sets, models, and reports their pipelines should produce and need those assets tested, observable, and kept up to date from local development through production.

What it is

Dagster is an orchestration platform for the development, production, and observation of data assets. It lives in the Python ecosystem, is distributed on PyPI, and officially supports Python 3.9 through Python 3.14. The programming model is declarative: an engineer writes ordinary Python functions and marks them with the @dg.asset decorator, describing the artifacts they want to exist rather than the sequence of jobs that happen to create them. Dagster then determines when to run those functions and keeps the resulting assets current. The README example shows this with three assets — country_populations returning a pandas DataFrame, continent_change_model returning a scikit-learn LinearRegression, and continent_stats depending on both — loaded into the project's web UI as a graph.

The concrete problem it addresses is the gap between running tasks and owning outputs. Pipelines can be written purely as collections of tasks, with no first-class notion of the assets those tasks produce, which makes it difficult to know what exists, whether it is fresh, and where it came from. Dagster puts the asset at the centre of the model instead, so lineage, metadata, cataloguing, and diagnostics attach to named objects such as tables, data sets, machine learning models, and reports. It is designed to be used at every stage of the data development lifecycle, spanning local development, unit tests, integration tests, staging environments, and production.

Key capabilities

• Declarative asset definitions using the @dg.asset decorator, with dependency wiring expressed through typed Python function arguments as shown by continent_stats consuming both country_populations and continent_change_model.
• A web UI that renders the asset graph defined in Python.
• Integrated lineage and observability, plus a unified control plane for centralising metadata, diagnostics, and cataloguing in one tool.
• Testability built into the development lifecycle, supporting unit tests, integration tests, and CI/CD practices that surface data quality issues and catch bugs early.
• A robust multi-tenant, multi-tool orchestration engine intended to scale both technically and organisationally.
• A growing library of integrations for popular data tools, with deployment onto the operator's own infrastructure.
• Command-line and server components shipped as separate packages, installed together via dagster, dagster-webserver, and dagster-dg-cli.

Who uses it and how

• Teams running the full development lifecycle, using the same asset definitions across local development, unit tests, integration tests, staging, and production rather than maintaining separate pipeline code per environment.
• Organisations adopting CI/CD practices for data, building reusable components and flagging data quality problems before they reach downstream consumers.
• Multi-tenant, multi-tool environments where several teams and several external systems share one orchestration engine and one metadata control plane.
• Data teams that already run established data tools and want to integrate rather than replace them, deploying to infrastructure they control.
• A community of data practitioners who share knowledge through the project's Slack workspace, GitHub Discussions, newsletter, blog, and other channels.

Getting started

Dagster installs from PyPI with uv add dagster dagster-webserver dagster-dg-cli. The project points new users at the documentation site and its hands-on ETL pipeline tutorial and quickstart guide.

How it compares

The available facts do not name other orchestration products, whether commercial or open source, and provide no list of paid tools that Dagster replaces. It stands alone in this registry on the evidence given: an Apache-2.0 licensed Python project distributed on PyPI, with deployment left to the operator's own infrastructure rather than tied to a vendor account.

When to use it — and when not to

Dagster fits teams working in Python who want assets, lineage, and metadata managed in one place and are willing to run the orchestrator and its web server on their own infrastructure. The README does not document the backing services a self-hoster must operate — no database, object storage, or mail configuration is described — so the operational footprint has to be established from the documentation rather than the repository front page. The project is Python-only, and with 2,589 open issues and no release information in the supplied facts, teams outside the Python ecosystem or wanting a small, statically stable dependency should look carefully before committing.

import dagster as dg
import pandas as pd

from sklearn.linear_model import LinearRegression


@dg.asset
def country_populations() -> pd.DataFrame:
    df = pd.read_html("https://tinyurl.com/mry64ebh")[0]
    df.columns = ["country", "pop2022", "pop2023", "change", "continent", "region"]
    df["change"] = df["change"].str.rstrip("%").astype("float")
    return df


@dg.asset
def continent_change_model(country_populations: pd.DataFrame) -> LinearRegression:
    data = country_populations.dropna(subset=["change"])
    return LinearRegression().fit(pd.get_dummies(data[["continent"]]), data["change"])


@dg.asset
def continent_stats(
    country_populations: pd.DataFrame, continent_change_model: LinearRegression
) -> pd.DataFrame:
    result = country_populations.groupby("continent").sum()
    result["pop_change_factor"] = continent_change_model.coef_
    return result

uv add dagster dagster-webserver dagster-dg-cli