olake-go is a free, open source data engineering & integration project written in Go and released under Apache-2.0. It has 1,672 GitHub stars, 282 forks and 195 open issues, and was last pushed 13 hours ago. On this registry it ranks #29 of 55 tracked projects in Data Engineering & Integration, with 5 head-to-head comparisons available.

What is olake-go?

OLake Go is an Apache-2.0-licensed data ingestion engine written in Go that replicates PostgreSQL, MySQL, MongoDB, Oracle, DB2, MSSQL, Apache Kafka and S3 into Apache Iceberg tables or plain Parquet, aimed at data engineering teams that want real-time pipelines without operating Spark, Flink, Kafka or Debezium.

What it is

OLake Go is an open-source ELT and ETL ingestion engine that lives in the Apache Iceberg lakehouse ecosystem. It reads from transactional databases, event streams and object storage and writes into vendor-neutral Iceberg tables that any compatible query engine or warehouse can then read, with support for Glue, Hive, JDBC and REST catalogs. It ships as a self-serve web UI plus a CLI, it was formerly published as datazip-inc/olake, and table maintenance work has been split out into a separate repository, datazip-inc/olake-fusion.

The concrete problem it solves is the stack normally required to move operational data into a lakehouse. Replicating a database today usually means assembling Debezium for change capture, Kafka for transport, and Spark or Flink for writing the files. OLake Go replaces that assembly with a single Go-based engine that performs full load and CDC, discovers schemas automatically, evolves them as sources change, and delivers writes exactly once using Arrow. The README explicitly frames the design goal as needing no Spark, no Flink, no Kafka and no Debezium, and its benchmark reports place it against Fivetran on throughput for both full load and CDC.

Key capabilities

  • Full load and change data capture from PostgreSQL through pgoutput (with wal2json marked deprecated), MySQL through binlog-based CDC, MongoDB through oplog-based CDC, and MSSQL, which supports full load, CDC and incremental sync.
  • JDBC-based full load and incremental sync for Oracle and DB2, covering warehouses where log-based capture is not offered.
  • Apache Kafka and S3 as sources alongside relational databases, so event streams and object storage land in the same destination format.
  • Two destination formats: Apache Iceberg, with Glue, Hive, JDBC and REST catalog support, and plain Parquet.
  • Automatic schema discovery and schema evolution during sync, so source-side changes are carried through without a separate modelling step.
  • Exactly-once delivery with Arrow-based writes, which is where the README locates its accuracy and throughput claims.
  • A self-serve web UI and a CLI for configuring and running syncs, with deployment through Docker Compose.

Who uses it and how

  • Teams with Postgres-heavy estates, where the published full-load benchmark records 580,113 RPS from Postgres to Iceberg and 55,555 RPS for CDC (as of 30 January 2026), cited as 12.5× and 2× faster than Fivetran respectively.
  • MySQL and MongoDB shops moving onto an open table format: 139,773 RPS full load and 59,951 RPS CDC for MySQL (as of 30 May 2026), and 37,879 RPS full load with 10,692 RPS CDC for MongoDB (as of 5 February 2026).
  • Organisations already running Kafka that want events landed in Iceberg; the Kafka to Iceberg benchmark reports 209,065 messages per second for bounded incremental loads, a comparison in which it is recorded as 1.23× slower than Flink.
  • Enterprises with Oracle, DB2 or MSSQL sources, including an MSSQL full-load benchmark of 345,866 RPS (as of 9 June 2026).
  • Small teams standing up a first pipeline, for whom the README's stated path is a Docker Compose deployment followed by syncing from the web UI.

Getting started

The README's route is to deploy with Docker Compose, which provides the self-serve web UI for creating and running syncs. Connector documentation and the full benchmark reports are published at olake.io/docs, with the project page at olake.io/olake-go/.

How it compares

OLake Go occupies the space next to Flink and Debezium-based CDC stacks, but its stated design is to do without them: ingestion runs without Spark, Flink, Kafka or Debezium, and the engine is self-hosted under Apache-2.0. Where the README quantifies performance, it benchmarks against Fivetran for full load and CDC throughput, and against Flink for Kafka ingestion.

When to use it — and when not to

A self-hoster takes on operating the engine together with whichever Iceberg catalog they choose, whether Glue, Hive, JDBC or REST, since the facts describe no hosted ingestion service. Teams that need log-based change capture from Oracle or DB2 should not pick it yet: Oracle CDC is marked as work in progress and DB2 lists no CDC support, leaving JDBC-based full load and incremental sync only. The repository also carries 195 open issues, so prospective adopters should expect a busy tracker alongside active development.

project readme (upstream, from github) — read inline

olake
OLake Go

OLake Go is a high-performance, open-source data ingestion engine for replicating databases, S3, and Kafka into Apache Iceberg (or plain Parquet).
Built for scalable, real-time pipelines, OLake Go provides a simple web UI and CLI - used to ingest into vendor-lock-in free Iceberg tables supporting all the query-engines/warehouses.

Read the docs and benchmarks at olake.io/docs. Join our active community on Slack.

GitHub issues Documentation slack Contribute to OLake

[!NOTE] 🎉 OLake Fusion is now live! — Automate your Apache Iceberg Table Maintenance. Check it out here → github.com/datazip-inc/olake-fusion 🎉

OLake Go — Super-fast Sync to Apache Iceberg

OLake Go supports replication from transactional databases such as PostgreSQL, MySQL, MongoDB, Oracle, DB2, and MSSQL, event-streaming systems like Apache Kafka and Object-store like S3, into open data lakehouse formats such as Apache Iceberg or Plain Parquet — delivering blazing-fast performance with minimal infrastructure cost.

pic


🚀 Why OLake Go?

  • 🧠 Smart sync: Full + CDC replication with automatic schema discovery & schema evolution
  • ⚡ High throughput: 580K RPS (Postgres) & 139K RPS (MySQL)
  • ➡️ Exactly once delivery & Arrow writes: Accuracy with speed.
  • 💾 Iceberg-native: Supports Glue, Hive, JDBC, REST catalogs
  • 🖥️ Self-serve UI: Deploy via Docker Compose and sync in minutes
  • 💸 Infra-light: No Spark, no Flink, no Kafka, no Debezium

📊 Benchmarks

Full Load
Source → Destination Full Load Relative Performance (Full Load) Full Report
Postgres → Iceberg
(as of 30th Jan 2026)
580,113 RPS 12.5× faster than Fivetran Full Report
MySQL → Iceberg
(as of 30th May 2026)
139,773 RPS 1.91× faster than Fivetran Full Report
MongoDB → Iceberg
(as of 5th Feb 2026)
37,879 RPS - Full Report
Oracle → Iceberg
(as of 30th Jan 2026)
526,337 RPS - Full Report
Kafka → Iceberg
(as of 27th Feb 2026)
209,065 MPS (Bounded Incremental) 1.23x slower than Flink Full Report
MSSQL → Iceberg
(as of 09th June 2026)
345,866 RPS 4.32x faster than Fivetran Full Report
CDC
Source → Destination CDC Relative Performance (CDC) Full Report
Postgres → Iceberg
(as of 30th Jan 2026)
55,555 RPS 2× faster than Fivetran Full Report
MySQL → Iceberg
(as of 30th May 2026)
59,951 RPS 1.52× faster than Fivetran Full Report
MongoDB → Iceberg
(as of 5th Feb 2026)
10,692 RPS - Full Report

🔧 Supported Sources and Destinations

Sources (Databases)
Source Full Load CDC Incremental Notes Documentation
PostgreSQL ✅ ✅ pgoutput ✅ wal2json deprecated Postgres Docs
MySQL ✅ ✅ ✅ Binlog-based CDC MySQL Docs
MongoDB ✅ ✅ ✅ Oplog-based CDC MongoDB Docs
Oracle ✅ WIP ✅ JDBC based Full Load & Incremental Oracle Docs
DB2 ✅ - ✅ JDBC based Full Load & Incremental DB2 Docs
MSSQL ✅ ✅ ✅ Full Load, CDC & Incremental MSSQL Docs
Source (S3)
Source Full Load CDC Incremental Notes Documentation
S3 ✅ - ✅ Ingests from Amazon S3 or S3-compatible (MinIO, LocalStack) S3 Docs
Source (Kafka)
Source Bounded Incremental Notes Documentation
Kafka ✅ Latest offset bounded incremental sync Kafka Docs
Destinations
Destination Format Supported Catalogs
Iceberg ✅ Glue, Hive, JDBC, REST (Nessie, Polaris, Unity, Lakekeeper, AWS S3 tables)
Parquet ✅ Filesystem
Writer Docs
  1. Apache Iceberg Docs

    1. Catalogs
      1. AWS Glue Catalog
      2. REST Catalog
        • Generic
        • Lakekeeper
        • Nessie
        • S3 Tables
        • Unity
        • Apache Polaris
      3. JDBC Catalog
      4. Hive Catalog
    2. Azure ADLS Gen2
    3. Google Cloud Storage (GCS)
    4. MinIO (local)
  2. Parquet Writer

    1. AWS S3 Docs
    2. Google Cloud Storage (GCS)
    3. Local FileSystem Docs

🧪 Quickstart (UI + Docker)

OLake UI is a web-based interface for managing OLake Go jobs, sources, destinations and configurations. You can run the entire OLake Go stack (UI, Backend, and all dependencies) using Docker Compose. This is the recommended way to get started. Run the UI, connect your source DB, and start syncing in minutes.

curl -sSL https://raw.githubusercontent.com/datazip-inc/olake-ui/master/docker-compose.yml | docker compose -f - up -d

Access the UI:

Detailed getting started using OLake UI can be found here.

olake-ui

Creating Your First Job

With the UI running, you can create a data pipeline in a few steps:

  1. Configure Source: Navigate to Source tab and click Create Source. Set up your source connection.
  2. Configure Destination: Navigate to Destination tab and click Create Destination. Set up your destination.
  3. Create a Job: Navigate to the Jobs tab and click Create Job.
  4. Configure & Run: Give your job a name, set a schedule, select your source and destination and click Next to finish.
  5. Select Streams: Choose which tables to sync and configure their sync mode (CDC, Full Refresh or Incremental).

For a detailed walkthrough, refer to the Jobs documentation.


🛠️ CLI Usage (Advanced)

For advanced users and automation, OLake Go's core logic is exposed via a powerful CLI. The core framework handles state management, configuration validation, logging, and type detection. It interacts with drivers using four main commands:

  • spec: Returns a render-able JSON Schema for a connector's configuration.
  • check: Validates connection configurations for sources and destinations.
  • discover: Returns all available streams (e.g., tables) and their schemas from a source.
  • sync: Executes the data replication job, extracting from the source and writing to the destination.
  • clear-destination : Clears data in the destination, only for the selected streams defined in streams.json.

Find out more about CLI here.


Install OLake Go

Below are other different ways you can run OLake Go:

  1. OLake Go UI (Recommended)
  2. Kubernetes using Helm
  3. Standalone Docker container
  4. Airflow on EC2
  5. Airflow on Kubernetes

Playground

  1. OLake Go + Apache Iceberg + REST Catalog + Presto
  2. OLake Go + Apache Iceberg + AWS Glue + Trino
  3. OLake Go + Apache Iceberg + AWS Glue + Athena
  4. OLake Go + Apache Iceberg + AWS Glue + Snowflake
  5. OLake Go + Apache Iceberg + REST Catalog + Spark

🌍 Use Cases

  • ✅ Migrate from OLTP to Iceberg without Spark or Flink
  • ✅ Enable BI over fresh CDC data using Athena, StarRocks, Trino, Presto, Dremio, Databricks, Snowflake and more!
  • ✅ Build near real-time data lake-house on cost-efficient cloud object stores
  • ✅ Move away from vendor-lock-in warehouse or tools with open data lake-house
  • ✅ Single copy for both analytics & machine learning

🧭 Roadmap Highlights

  • Oracle Full Load Support
  • Oracle Incremental
  • Filters for Full Load and Incremental
  • Interoperability (Coming Soon)
  • Iceberg V3 Support

🤝 Contributing

We ❤️ contributions, big or small!

Check out our Bounty Program. A huge thanks to all our amazing contributors!

Frequently asked questions

Is olake-go free to use?

olake-go is open source under the Apache-2.0 licence. There is no licence fee and no seat count — you can self-host it or, where the project offers one, pay a vendor for a managed version instead.

What does olake-go do?

OLake Go: open-source replication from databases, Kafka and S3 to Apache Iceberg and Parquet. Sources: Postgres, MySQL, MongoDB, Oracle, MSSQL, DB2, Kafka, S3.

What is olake-go written in?

olake-go is primarily written in Go. Its source is publicly available at https://github.com/datazip-inc/olake-go, and it has 1,672 GitHub stars.