[opensourcedex]

go-streams is a free, open source data engineering & integration project written in Go and released under MIT. It has 2,172 GitHub stars, 174 forks and 11 open issues, and was last pushed 8 months ago. On this registry it ranks #23 of 39 tracked projects in Data Engineering & Integration, with 5 head-to-head comparisons available.

What is go-streams?

What it is

go-streams is a lightweight stream processing library for Go. It lives in the Go data engineering and analytics ecosystem, and it lets programmers define declarative data pipelines from composable sources, flows, and sinks. The core module has no external dependencies, so the basic pipeline model does not require a broker, cloud service, or third-party runtime.

The project solves a concrete problem: many Go applications need to move and transform data continuously, but the standard library provides only low-level primitives such as channels, files, and readers or writers. go-streams supplies a pipeline abstraction where a source emits data, flows transform or route it, and sinks consume it. This supports ETL, data-pipeline, and stream-processing tasks inside the same language as the surrounding application.

Key capabilities

  • It provides Source, Flow, and Sink components for pipeline entry, processing, and termination.
  • It includes Map, FlatMap, Filter, Fold, Reduce, and PassThrough transformation flows.
  • It supports Split, FanOut, RoundRobin, Merge, ZipWith, and Flatten routing flows.
  • It offers Batch, SlidingWindow, TumblingWindow, and SessionWindow windowing flows.
  • It includes Throttler and AdaptiveThrottler for rate control, with CPU and memory usage.
  • It provides Go channel, file, io.Reader, io.Writer, os.Stdout, and Discard connectors.
  • It offers separate modules for Aerospike, Apache Kafka, Apache Pulsar, AWS S3, and Azure Blob Storage.

Who uses it and how

  • Go developers use it to define pipelines declaratively instead of writing custom channel-handling code.
  • Teams building ETL or data-integration jobs use it to connect sources, apply transformations, and write results to sinks.
  • Projects exchanging data with Kafka, Pulsar, Aerospike, Redis, or NATS Streaming use it as a Go-side pipeline layer.
  • Developers use os.Stdout and Discard sinks for local development and debugging.
  • Pipeline authors use Keyed flows to group elements by key and process related data in parallel.

Getting started

The provided facts point to the Go package page and show connector support in separate modules with their own dependencies. The excerpt does not list Docker images, hosted options, or explicit install commands.

When to use it — and when not to

Use go-streams when a Go application needs a lightweight pipeline abstraction for stream processing, ETL, or data routing. Avoid it when a hosted pipeline service is required, because the supplied facts mention no hosted option. Self-hosters must still operate external systems used by connectors, such as Kafka, Pulsar, Aerospike, Redis, NATS Streaming, AWS S3, or Azure Blob Storage.

project readme (upstream, from github) — read inline

go-streams

Build PkgGoDev Go Report Card codecov

go-streams provides a lightweight and efficient stream processing framework for Go. Its concise DSL allows for easy definition of declarative data pipelines using composable sources, flows, and sinks.

pipeline-architecture-example

Wiki
In computing, a pipeline, also known as a data pipeline, is a set of data processing elements connected in series, where the output of one element is the input of the next one. The elements of a pipeline are often executed in parallel or in time-sliced fashion. Some amount of buffer storage is often inserted between elements.

Overview

The core module has no external dependencies and provides three key components for constructing stream processing pipelines:

  • Source: The entry point of a pipeline, emitting data into the stream. (One open output)
  • Flow: A processing unit, transforming data as it moves through the pipeline. (One open input, one open output)
  • Sink: The termination point of a pipeline, consuming processed data and often acting as a subscriber. (One open input)

Flows

The flow package provides a collection of Flow implementations for common stream processing operations. These building blocks can be used to transform and manipulate data within pipelines.

  • Map: Transforms each element in the stream.
  • FlatMap: Transforms each element into a stream of slices of zero or more elements.
  • Filter: Selects elements from the stream based on a condition.
  • Fold: Combines elements of the stream with the last folded value and emits the new value. Requires an initial value.
  • Reduce: Combines elements of the stream with the last reduced value and emits the new value. Does not require an initial value.
  • PassThrough: Passes elements through unchanged.
  • Split1: Divides the stream into two streams based on a boolean predicate.
  • FanOut1: Duplicates the stream to multiple outputs for parallel processing.
  • RoundRobin1: Distributes elements evenly across multiple outputs.
  • Merge1: Combines multiple streams into a single stream.
  • ZipWith1: Combines elements from multiple streams using a function.
  • Flatten1: Flattens a stream of slices of elements into a stream of elements.
  • Batch: Breaks a stream of elements into batches based on size or timing.
  • Throttler: Limits the rate at which elements are processed.
  • AdaptiveThrottler: Limits the rate at which elements are processed based on the current system resource utilization (CPU and memory usage).
  • SlidingWindow: Creates overlapping windows of elements.
  • TumblingWindow: Creates non-overlapping, fixed-size windows of elements.
  • SessionWindow: Creates windows based on periods of activity and inactivity.
  • Keyed: Groups elements by key for parallel processing of related data.

1 Utility Flows

Connectors

Standard Source and Sink implementations are located in the extension package.

  • Go channel inbound and outbound connector
  • File inbound and outbound connector
  • Standard Go io.Reader Source and io.Writer Sink connectors
  • os.Stdout and Discard Sink connectors (useful for development and debugging)

The following connectors are available as separate modules and have their own dependencies.

Usage Examples

See the examples directory for practical code samples demonstrating how to build complete stream processing pipelines, covering various use cases and integration scenarios.

License

Licensed under the MIT License.

Frequently asked questions

Is go-streams free to use?

go-streams is open source under the MIT licence. There is no licence fee and no seat count — you can self-host it or, where the project offers one, pay a vendor for a managed version instead.

What does go-streams do?

A lightweight stream processing library for Go

What is go-streams written in?

go-streams is primarily written in Go. Its source is publicly available at https://github.com/reugn/go-streams, and it has 2,172 GitHub stars.