Open Source Data Engineering & Integration Tools

airflow ★ 47K

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

Airbyte ★ 22K

Open-source data integration for modern teams

Cube ★ 21K

Unify data models and metrics across your entire stack

shardingsphere ★ 21K

Empowering Data Intelligence with Distributed SQL for Sharding, Scalability, and Security Across All Databases.

debezium ★ 13K

Change data capture for a variety of databases. Please log issues at https://github.com/debezium/dbz/issues.

CocoIndex ★ 12K

Ultra-fast data transformation for AI with lineage

seatunnel ★ 9.8K

SeaTunnel is a multimodal, high-performance, distributed, massive data integration tool.

steampipe ★ 8.0K

Zero-ETL, infinite possibilities. Live query APIs, code & more with SQL. No DB required.

realtime ★ 7.7K

Broadcast, Presence, and Postgres Changes via WebSockets

CloudQuery ★ 6.6K

Sync and transform data from any source to any destination

flink-cdc ★ 6.5K

Flink CDC is a streaming data integration tool

Lightdash ★ 6.2K

Open-source BI for modern data teams

fluvio ★ 5.3K

🦀 event stream processing for developers to collect and transform data in motion to power responsive data intensive applications.

whodb ★ 5.0K

Where data access meets operational intelligence

docetl ★ 4.2K

A system for agentic LLM-powered data processing and ETL

aws-sdk-pandas ★ 4.1K

pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager,

chunjun ★ 4.1K

A data integration framework

sqlmesh ★ 3.3K

Scalable and efficient data transformation framework - backwards compatible with dbt.

peerdb ★ 3.3K

Fast, Simple and a cost effective tool to replicate data from Postgres to Data Warehouses, Queues and Storage

devlake ★ 3.2K

Apache DevLake is an open-source dev data platform to ingest, analyze, and visualize the fragmented data from DevOps tools, extracting insights for engineering

Open Wearables ★ 2.7K

Convert wearable data into actionable health insights with open algorithms

API ★ 2.5K

API for Current cases and more stuff about COVID-19 and Influenza

Elementary Data ★ 2.4K

Data observability for modern data teams

go-streams ★ 2.2K

A lightweight stream processing library for Go

doit ★ 2.1K

CLI task management & automation tool

CherryUSB ★ 2.1K

CherryUSB is a tiny and beautiful, high performance and portable USB host and device stack for embedded system with USB IP

helix-foundry ★ 2.0K

Connect your company's data, let an agent draft an ontology over it, and load it into HelixDB as one graph you can explore and ask questions of. Hosted at found

multiwoven ★ 1.7K

🔥🔥🔥 Open source Reverse ETL - alternative to hightouch and census.

Seal-Report ★ 1.6K

Database Reporting Tool and Tasks (.Net)

riko ★ 1.6K

A Python stream processing engine modeled after Yahoo! Pipes

olake ★ 1.6K

OLake - Fastest Databases, Kafka & S3 Replication to Apache Iceberg with Table optimization (Called OLake Fusion). ⚡ Efficient, quick and scalable data ingestio

duckle ★ 1.5K

Open-source ETL/ELT you deploy on your own servers or cloud. Built on DuckDB: no-code/low-code visual pipelines or SQL, 385 components, dbt, CDC, data quality,

OpenContracts ★ 1.5K

The open document intelligence platform for builders and hackers - DMS for the agentic world

hop ★ 1.5K

Hop Orchestration Platform

Addax ★ 1.5K

Actively maintained fork of Alibaba DataX — a fast, versatile ETL tool for RDBMS/NoSQL data transfer

pgsync ★ 1.4K

Postgres, MySQL, or MariaDB to Elasticsearch/OpenSearch sync

squid-sdk ★ 1.3K

TypeScript ETL toolkit for indexing Ethereum, Solana, and Substrate data, sourced from SQD Network.

drasi-platform ★ 1.3K

The Data Change Processing platform

zerocode ★ 1.0K

zerocode-tdd is a community-developed, free, open-source, outcome-driven automated testing framework for Data Pipelines, ETL, REST API, Kafka(Data Streams), Dat

sling-cli ★ 919

Sling is a CLI tool that extracts data from a source storage/database and loads it in a target storage/database.

seatunnel-web ★ 888

SeaTunnel is a distributed, high-performance data integration platform for the synchronization and transformation of massive data (offline & real-time).

wexflow ★ 844

Workflow Automation Engine

Etl.Net ★ 813

Mass processing data with a complete ETL for .net developers

conduit ★ 618

Conduit streams data between data stores. Kafka Connect replacement. No JVM required.

pudl ★ 613

The Public Utility Data Liberation Project provides analysis-ready energy system data to climate advocates, researchers, policymakers, and journalists.

beginner_de_project ★ 611

Beginner data engineering project - batch edition

redun ★ 608

Yet another redundant workflow engine

sycamore ★ 605

🍁 Sycamore is an LLM-powered search and analytics platform for unstructured data.

automate-dv ★ 598

A free to use dbt package for creating and loading Data Vault 2.0 compliant Data Warehouses (powered by dbt, an open source data engineering tool, registered t

metorikku ★ 589

A simplified, lightweight ETL Framework based on Apache Spark

genblaze ★ 552

Genblaze is an open source Python SDK for orchestrating generative AI media pipelines across video, audio, and image providers with built in provenance for ever

ReplicaDB ★ 504

ReplicaDB is open source tool for database replication, designed for efficiently transferring bulk data between relational and non-relational databases

Impler ★ 294

Seamless data import for your applications