[opensourcedex]

fluvio is a free, open source data engineering & integration project written in Rust and released under Apache-2.0. It has 5,256 GitHub stars, 530 forks and 139 open issues, and was last pushed 18 days ago. On this registry it ranks #12 of 39 tracked projects in Data Engineering & Integration, with 5 head-to-head comparisons available. It gained 1 stars over the last 3 tracked days.

What is fluvio?

What it is

Fluvio is a distributed data streaming engine written in Rust, distributed under the Apache-2.0 license and developed in the fluvio-community repository. It is described as composable, declarative and stateful: a streaming runtime that pairs with Stateful DataFlow, a distributed stream processing framework, so that a single system covers both the transport of event streams and the transformation of that data while it is in motion. The project sits in the Rust and cloud-native ecosystem and is published on crates.io, with documentation hosted at fluvio.io.

The concrete problem it addresses is the gap between collecting data in motion and acting on it. Rather than moving records into a separate processing tier, Fluvio combines a distributed streaming engine with a stateful dataflow layer, which means developers can build pipelines that ingest, transform and route events for data-intensive applications that must respond in real time. The topic list frames this as data analytics, data integration, data pipelines, event-driven architecture, distributed systems and serverless workloads, all of which depend on streams being available and processable as a single, unified paradigm.

Key capabilities

  • Distributed data streaming engine implemented in Rust, with the runtime designed as a lean core rather than a bundled platform.
  • Stateful DataFlow stream processing framework, allowing stateful transformations over streams rather than stateless filtering alone.
  • Composable and declarative model, so streaming and stream processing share one paradigm instead of two disconnected tools.
  • Command-line interface covering cluster lifecycle, topic creation, production and consumption.
  • Fluvio Version Manager (fvm), which installs and manages Fluvio CLI versions, including the latest stable channel, dev builds and pinned releases.
  • Version selection through environment variables such as FVM_VERSION and FLUVIO_VERSION at install time.
  • Rust client crate published on crates.io with generated documentation on docs.rs.

Who uses it and how

  • Developers building real-time, event-driven applications that need data collected and transformed as it arrives.
  • Teams running cloud-native and distributed-systems workloads where streaming is part of a larger pipeline.
  • Data engineers assembling data integration and data pipeline workflows across systems, using topics as the hand-off point.
  • Serverless and stateful application authors who need processing logic to retain state across events.
  • Local development workflows: a single-machine cluster is started from the CLI, topics are created, and data is produced and consumed from separate terminals to exercise a pipeline end to end.

Getting started

Installation goes through the Fluvio Version Manager, fetched with a curl script from the fluvio-community/fluvio repository; fvm installs the Fluvio CLI into $HOME/.fluvio with binaries in $HOME/.fluvio/bin. A local cluster

project readme (upstream, from github) — read inline

Fluvio

Composable, Declarative, Stateful Data Streaming System

CI Status CD Status fluvio Crates.io version Fluvio Rust documentation Fluvio dependency status Fluvio Discord


An animated visual of fluvio distributed streaming runtime


Fluvio is a lean and mean distributed data streaming engine written in Rust. Combined with Stateful DataFlow distributed stream processing framework, Fluvio provides a unified composable distributed streaming and stream processing paradigm for developers.

Quick Start - Get started with Fluvio in 2 minutes or less!

Step 1. Download Fluvio Version Manager:

Fluvio is installed via the Fluvio Version Manager, shortened to fvm.

To install fvm, run the following command:

TEMPORARY NOTE: While we are transitioning to a new fluvio-community hosted fluvio build and release, you should install the dev version of FVM:

curl -fsS https://raw.githubusercontent.com/fluvio-community/fluvio/master/install.sh | FVM_VERSION=dev bash

After we get the fluvio-community based released sorted again, it will return to:

curl -fsS https://raw.githubusercontent.com/fluvio-community/fluvio/master/install.sh | bash

As part of the initial setup, fvm will also install the Fluvio CLI available in the stable channel as of the moment of installation.

Fluvio is stored in $HOME/.fluvio, with the executable binaries stored in $HOME/.fluvio/bin.

For the best compatibliity on Windows, InfinyOn recommends WSL2

Step 2. Start a cluster:

Start cluster on you local machine with the following command:

fluvio cluster start

Step 3. Create Topic:

The following command will create a topic called hello-fluvio:

fluvio topic create hello-fluvio

Step 4. Produce to Topic, Consume From Topic:

Produce data to your topic. Run the command first and then type some messages:

fluvio produce hello-fluvio
> hello fluvio
Ok!
> test message
Ok!

Consume data from the topic, Run the following command in a different terminal:

fluvio consume hello-fluvio -B -d

Just like that! You have a local cluster running.

Using Pre-Built Fluvio Versions

You may want to use other Fluvio versions than the latest stable release. You can do so by specifying the version using environment variables.

Install Latest Dev Version

To install the latest development build of fvm and fluvio:

curl -fsS https://raw.githubusercontent.com/fluvio-community/fluvio/master/install.sh | FVM_VERSION=dev bash

Install Specific Fluvio Version

curl -fsS https://raw.githubusercontent.com/fluvio-community/fluvio/master/install.sh | FLUVIO_VERSION=x.y.z bash

Install Specific FVM Version

curl -fsS https://raw.githubusercontent.com/fluvio-community/fluvio/master/install.sh | FVM_VERSION=v0.18.1 bash
Check Fluvio Core Documentation

Fluvio documentation will provide additional context on how to use the Fluvio clusters, CLI, clients, a development kits.

Check Stateful DataFlow Documentation

Stateful DataFlow designed to handle complex data processing workflows, allowing for customization and scalability through various programming languages and system primitives.

Learn how to build custom connectors

Fluvio can connect to practically any system that you can think of.

  • For first party systems, fluvio clients can integrate with the edge system or application to source data.
  • For third party systems fluvio connectors connect at the protocol level and collects data into fluvio topics.

Out of the box Fluvio has native http, webhook, mqtt, kafka inbound connectors. In terms of outbound connectors out of the box Fluvio supports http, SQL, kafka, and experimental builds of DuckDB, Redis, S3, Graphite etc.

Using Connector Development Kit, its intuitive to build connectors to any system fast.

Check out the docs and let us know if you need help building any connector.

Learn how to build custom smart modules

Fluvio applies wasm based stream processing and data transformations. We call these reusable transformation functions smart modules. Reusable Smart modules are built using Smart Module Development Kit and can be distributed using InfinyOn Cloud hub.

Clients

Language Specifc API docs:

Community Maintained:

Contributing

If you'd like to contribute to the project, please read our Contributing guide.

Community

Many fluvio users and developers have made projects to share with the community. Here a a few listed below:

Projects Using Fluvio

Community Connectors

Community Development Resources

More projects and utilities are available in the Fluvio Community Github Org

Contributors are awesome

Made with contrib.rocks.

Community

Many fluvio users and developers have made projects to share with the community. Here a a few listed below:

Projects Using Fluvio

License

This project is licensed under the Apache license.

Frequently asked questions

Is fluvio free to use?

fluvio is open source under the Apache-2.0 licence. There is no licence fee and no seat count — you can self-host it or, where the project offers one, pay a vendor for a managed version instead.

What does fluvio do?

🦀 event stream processing for developers to collect and transform data in motion to power responsive data intensive applications.

What is fluvio written in?

fluvio is primarily written in Rust. Its source is publicly available at https://github.com/fluvio-community/fluvio, and it has 5,256 GitHub stars.