Taier is an Apache-2.0 licensed, Java-based distributed dispatching system that gives big-data teams one place to submit, schedule, operate, maintain, and visualize the tasks and indicators of an ETL pipeline.

What it is

Taier is a one-stop big data development platform built for submission, scheduling, operation and maintenance, and indicator information display. It runs in the Hadoop, Hive, Spark, and Flink ecosystem, tieing those engines and their many SQL dialects behind a single scheduling layer. The project describes itself as a distributed dispatching system focused on submitting and scheduling different kinds of tasks, with the stated aim of reducing the cost of ETL, making complex dependencies between tasks clear, and cutting the labor cost of submitting, scheduling, and operating those tasks. An IDE-style development surface is part of the platform, so task authoring and task scheduling live in the same tool rather than in two.

The concrete problem it solves is the gap between an orchestration front end and the underlying big-data platform: users should not have to reason about the complex dependencies between tasks or the architecture below them. Taier absorbs that dependency graph and the cluster plumbing so work shifts to business logic. In the ecosystem of schedulers the topics list places it alongside Azkaban-style job scheduling and cron-driven job submission, and it is the layer that replaces hand-rolled cron entries or a separate workflow tool glued onto a Hadoop cluster. Isolation by tenants and clusters plus Kerberos support let several teams share one control plane without sharing cluster access.

Key capabilities

• Visual configuration for DAGs, so upstream and downstream task dependencies are edited as a graph rather than as text.
• Multiple task modes, including a guide mode and a script mode, for the same underlying task types.
• Batch and stream tasks in one scheduler, with support for integrating Flink Standalone and for various different versions of Hadoop.
• Broad task type coverage: Work Flow, Data Sync, Data Acquisition, Flink, Flink SQL, Shell, Python, Spark SQL, Hive SQL, DataX, and SQL targets including OceanBase, ClickHouse, Doris, TiDB, MySQL, Vertica, Postgre, SqlServer, Greenplum, MaxCompute, and GaussDB.
• Plugin development, so teams can write and add their own task plugins instead of waiting on upstream support.
• Isolation by tenants and clusters, with Kerberos authentication, task resource restriction, and different versions for the same task.
• Real-time monitoring of cluster resources and real-time presentation of data indicators, plus user-defined parameters per task and easy distributed extension.

Who uses it and how

• Data platform teams running Hadoop, Hive, Spark, or Flink who need one scheduler in front of a mixed engine estate instead of one tool per engine.
• ETL teams carrying complex inter-task dependencies that have outgrown manual ordering and want the graph expressed visually.
• Organizations with several teams on shared clusters, using tenant and cluster isolation plus Kerberos so access stays separated.
• Streaming operators who need Flink Standalone jobs and batch jobs submitted and observed from the same console, with cluster resource monitoring alongside.
• Teams that need a non-intrusive control plane, since the platform is described as completely safe and non-intrusive to the cluster's environment.

Getting started

The documented quick start is Docker Compose: download docker-compose.yml from the repository and run docker-compose up -d. A hosted online preview is also published at http://taier.dtstack.cn/, and task development is walked through in the project documentation at https://dtstack.github.io/Taier/.

How it compares

The repository's topics list names Azkaban and chunjun as adjacent technology, so Taier sits in the same conversation as Azkaban for workflow scheduling and as the Chunjun/DataX family for data movement, while adding an IDE and indicator display on top. The distinguishing claim in the facts is that it is non-intrusive to the cluster and supports isolation by tenants and clusters with Kerberos, which matters when several tenants share one Hadoop or Flink estate. No list of paid products it replaces is provided, so no licence or cost comparison against commercial schedulers is made here.

When to use it — and when not to

A self-hoster must already operate the big-data side — Hadoop, Hive, Spark, or Flink, with Kerberos where authentication is enabled — and must run the platform itself via the supplied docker-compose.yml; there is no managed cloud option listed. Anyone without an existing Hadoop-ecosystem cluster, or a team wanting a single-language script runner with no scheduler, should look elsewhere. The honest weaknesses are that the repository carries 75 open issues, the README excerpt defers most operational detail to the external documentation site, and the sibling README is the Chinese-language document, so English-only readers will lean on the hosted docs.

$ wget https://raw.githubusercontent.com/DTStack/Taier/master/docker-compose.yml
$ docker-compose up -d