[opensourcedex]

conduit is a free, open source data engineering & integration project written in Go and released under Apache-2.0. It has 611 GitHub stars, 63 forks and 123 open issues, and was last pushed 14 hours ago. On this registry it ranks #37 of 39 tracked projects in Data Engineering & Integration, with 5 head-to-head comparisons available.

What is conduit?

Conduit is an Apache-2.0 licenced data streaming tool written in Go that moves records between production data stores in real time, and it is aimed at data engineers who want Kafka Connect-style pipelines without a JVM in the stack.

What it is

Conduit is an open-source data integration tool created and open-sourced by Meroxa. It lives in the Go ecosystem and in the data engineering and integration space, and it is distributed under the Apache-2.0 licence. Pipelines in Conduit are assembled from simple building blocks, each running in its own goroutine and connected to the next through Go channels, which is what makes a Conduit pipeline performant on multi-core machines. The project ships with common connectors, processors, and observability data out of the box, and it exposes an HTTP API alongside its command line.

The concrete problem Conduit solves is the operational weight of Kafka Connect. Conduit is positioned directly as a Kafka Connect replacement, and the specific thing it replaces is the JVM-based Kafka Connect runtime that many teams otherwise have to run, tune, and monitor alongside their brokers. Conduit connectors are plugins rather than JVM classes: they talk to Conduit over a gRPC interface, so a plugin can be written in any language as long as it conforms to that interface. That removes the requirement that connector authors work in the Java ecosystem.

Key capabilities

  • conduit quickstart scaffolds and runs a demo pipeline in one command, with sample records reaching the console within seconds, state held in memory, and nothing written to the working directory.
  • Record order is guaranteed: the order of received records does not change as they pass through a pipeline.
  • Consistency is handled by propagating acknowledgements back to the start of the pipeline only when a record has been successfully processed on all destinations.
  • Pipelines are composed of building blocks that each run in their own goroutine and are wired together with Go channels, which the README calls out as the reason Conduit pipelines are performant on multi-core machines.
  • Connectors are plugins that communicate with Conduit over a gRPC interface, so plugins can be written in any language that conforms to the required interface.
  • Conduit ships with common connectors, processors, and observability data out of the box rather than requiring them to be sourced separately.
  • Conduit can be embedded directly in a Go application, and it publishes an HTTP API with separate API documentation.

Who uses it and how

  • Teams running Kafka-based data pipelines that want to drop the JVM runtime, using the kafka and kafkaconnect topics as the integration surface.
  • Data engineers building ETL and data-pipeline workloads who need real-time streaming rather than batch movement.
  • Go shops that want Conduit inside their own application rather than as a separate service, using the embedding path documented in the README.
  • Connector authors who want to write plugins in a language other than Java, targeting the gRPC plugin interface.
  • Operators running pipelines on multi-core hosts, where the goroutine-and-channel execution model is the reason given for Conduit's throughput.

Getting started

The documentation site leads with the install script, curl https://conduitdata.io/install.sh | bash, which detects the platform and installs through the native package manager for it. Alternatives are brew install conduit on macOS, or downloading a pre-built binary from the latest release and running ./conduit run.

How it compares

Kafka Connect is the tool Conduit is explicitly built to replace, and the contrast the facts support is runtime and licensing: Conduit is Apache-2.0 and written in Go, so it needs no JVM, and its connectors are gRPC plugins that can be written in any language rather than Java classes. Beyond that, the facts do not name other comparable tools, so Conduit stands on its own in this registry.

When to use it — and when not to

A self-hoster takes on running the Conduit process itself, configuring it, and setting up its storage, and on macOS the release binaries are only ad-hoc signed, not Developer ID signed and not notarized, so a binary downloaded through a browser carries the com.apple.quarantine attribute and Gatekeeper kills it with no error message until xattr -d com.apple.quarantine ./conduit is run. Teams that need a fully managed, hosted pipeline service should look elsewhere, since the facts describe only a self-hosted binary, an install script, and a Go library. The project is active but carries a visible backlog of 123 open issues, and the README excerpt does not spell out the operational requirements of the storage layer.

project readme (upstream, from github) — read inline

Conduit

Logo

Data Integration for Production Data Stores. :dizzy:

scarf pixel License Test Go Report Card Discord Twitter Go Reference Conduit docs API docs

Overview

Conduit is a data streaming tool written in Go. It aims to provide the best user experience for building and running real-time data pipelines. Conduit comes with common connectors, processors and observability data out of the box.

Conduit pipelines are built out of simple building blocks which run in their own goroutines and are connected using Go channels. This makes Conduit pipelines incredibly performant on multi-core machines. Conduit guarantees the order of received records won't change, it also takes care of consistency by propagating acknowledgements to the start of the pipeline only when a record is successfully processed on all destinations.

Conduit connectors are plugins that communicate with Conduit via a gRPC interface. This means that plugins can be written in any language as long as they conform to the required interface.

Conduit was created and open-sourced by Meroxa.

Quick start

The fastest way to see Conduit working — scaffold and run a demo pipeline with a single command:

conduit quickstart

Sample records flow to your console within seconds. State is in-memory and nothing is written to your working directory; press Ctrl-C to stop. For a full walkthrough, see .

Installation guide

Install script (recommended)

The install script detects your platform and installs Conduit through the native package manager for it — Homebrew on macOS, dpkg or rpm on Linux:

curl https://conduitdata.io/install.sh | bash

This is the route the documentation site leads with, and on macOS it is the one to prefer: it goes through Homebrew, so it avoids the quarantine problem described under Download binary and run.

Homebrew

Make sure you have homebrew installed on your machine, then run:

brew update
brew install conduit

Download binary and run

Download a pre-built binary from the latest release and run it:

./conduit run

On macOS, the release binaries are ad-hoc signed — they are not Developer ID signed and they are not notarized. A binary downloaded through a browser therefore carries the com.apple.quarantine attribute, and Gatekeeper refuses to run it with no error message at all: the process is killed and you get an empty terminal. Clear the attribute after downloading and before you run the binary for the first time:

xattr -d com.apple.quarantine ./conduit

curl and wget do not set the quarantine attribute, so a binary fetched that way runs as-is. Homebrew and the install script are unaffected — on macOS, prefer either of those.

Once you see that the service is running, the configured pipeline should start processing records automatically. You can also interact with the Conduit API directly, we recommend navigating to http://localhost:8080/openapi and exploring the HTTP API through Swagger UI.

Conduit can be configured through command line parameters. To view the full list of available options, run ./conduit run --help or see configuring Conduit.

Debian

Download the right .deb file for your machine architecture from the latest release, then run the command below, substituting the version and architecture of the file you downloaded for and:

dpkg -i conduit_<version>_Linux_<arch>.deb

RPM

Download the right .rpm file for your machine architecture from the latest release, then run the command below, substituting the version and architecture of the file you downloaded for and:

rpm -i conduit_<version>_Linux_<arch>.rpm

Build from source

Requirements:

git clone [email protected]:ConduitIO/conduit.git
cd conduit
make
./conduit run

Docker

Our Docker images are hosted on GitHub's Container Registry. To run the latest Conduit version, you should run the following command:

docker run -p 8080:8080 conduit.docker.scarf.sh/conduitio/conduit:latest

Preflight checks

Before running Conduit for the first time (or after changing configuration), run conduit doctor to check whether your environment is ready:

conduit doctor

It runs a set of offline, non-destructive checks — config resolution and validation, database reachability, API address availability, plugin directories, the built-in plugin registry, and whether a running engine is reachable — and reports pass/warn/fail for each, grouped by category. It does not start a Runtime; this is distinct from the running server's /readyz and /healthz endpoints, which answer "is the running engine serving?" instead of "would conduit run succeed here?".

Useful flags: --json for machine-readable output, --check (repeatable) to run only specific checks, --deep to additionally verify standalone connector plugin binaries in an isolated subprocess, --require-server to fail instead of warn when no Conduit server is reachable, and -q/--quiet to only print warnings, failures, and the summary. See conduit doctor --help for the full list of checks and exit codes, and the design doc for the rationale.

Configuring Conduit

Conduit accepts CLI flags, environment variables and a configuration file to configure its behavior. Each CLI flag has a corresponding environment variable and a corresponding field in the configuration file. Conduit uses the value for each configuration option based on the following priorities:

  • CLI flags (highest priority) - if a CLI flag is provided it will always be respected, regardless of the environment variable or configuration file. To see a full list of available flags run conduit run --help.

  • Environment variables (lower priority) - an environment variable is only used if no CLI flag is provided for the same option. Environment variables have the prefix CONDUIT and contain underscores instead of dots and hyphens (e.g. the flag -db.postgres.connection-string corresponds to CONDUIT_DB_POSTGRES_CONNECTION_STRING).

  • Configuration file (lowest priority) - By default, Conduit loads a configuration file named conduit.yaml located in the same directory as the Conduit binary. You can customize the directory path to this file using the CLI flag --config.path. The configuration file is optional, as any value specified within it can be overridden by an environment variable or a CLI flag.

    The file must be a YAML document, and keys can be hierarchically structured using .. For example:

    db:
  type: postgres # corresponds to flag -db.type and env variable CONDUIT_DB_TYPE
  postgres:
    connection-string: postgres://localhost:5432/conduitdb # -db.postgres.connection-string or CONDUIT_DB_POSTGRES_CONNECTION_STRING

    To generate a configuration file with default values, use: conduit init --path .

This parsing confi

readme truncated — read the full docs on github

Frequently asked questions

Is conduit free to use?

conduit is open source under the Apache-2.0 licence. There is no licence fee and no seat count — you can self-host it or, where the project offers one, pay a vendor for a managed version instead.

What does conduit do?

Conduit streams data between data stores. Kafka Connect replacement. No JVM required.

What is conduit written in?

conduit is primarily written in Go. Its source is publicly available at https://github.com/ConduitIO/conduit, and it has 611 GitHub stars.