What it is

OLake Go is an open-source data ingestion engine written in Go and licensed under Apache-2.0. It replicates transactional databases, Apache Kafka, and S3 into Apache Iceberg or plain Parquet, and it provides a web UI and a command-line interface for building scalable, real-time pipelines. The project sits in the Apache Iceberg and lakehouse ecosystem, where its stated goal is to move data into vendor-neutral table formats that can be read by query engines and warehouses.

The concrete problem it addresses is the operational complexity of building ELT pipelines that keep analytical tables current with source systems. OLake Go supports full load, change-data-capture, and incremental replication from PostgreSQL, MySQL, MongoDB, Oracle, DB2, MSSQL, Kafka, and S3. It also supports automatic schema discovery and schema evolution, so pipelines can reflect source changes without a separate transformation layer. The README describes the engine as infrastructure-light because the ingestion path does not require Spark, Flink, Kafka, or Debezium.

Key capabilities

• OLake Go supports full load, change-data-capture, and incremental replication from PostgreSQL, MySQL, MongoDB, Oracle, DB2, MSSQL, Kafka, and S3.
• It writes replicated data into Apache Iceberg or plain Parquet, and it supports Glue, Hive, JDBC, and REST catalogs for Iceberg table management.
• It provides automatic schema discovery and schema evolution, allowing pipelines to adapt when source tables change.
• It uses exactly-once delivery and Arrow writes, according to the README, to combine accuracy with high-throughput ingestion.
• It includes a self-serve web UI and a CLI, and it can be deployed with Docker Compose.
• A companion project, OLake Fusion, automates Apache Iceberg table maintenance.

Who uses it and how

• Data engineering teams can use OLake Go to replicate operational databases into a lakehouse for real-time analytics.
• Streaming workloads can ingest Apache Kafka into Apache Iceberg or Parquet with bounded incremental processing, as described in the README.
• Teams can replicate S3 object-store data into open table formats, with the README stating that Spark, Flink, Kafka, or Debezium are not required for the ingestion path.
• Users can configure and run syncs through the web UI or CLI, then expose the resulting Iceberg tables to query engines and warehouses through supported catalogs.

Getting started

Typical setup is to run OLake Go with Docker Compose, then use the web UI or CLI to define sources and destination catalogs. The README points users to the documentation and benchmarks at olake.io/docs.

When to use it — and when not to

Use OLake Go when a self-hosted Go-based ingestion engine for databases, Kafka, or S3 is needed, and when the pipeline should write to Apache Iceberg or Parquet without requiring additional Spark, Flink, Kafka, or Debezium infrastructure. The README compares Postgres, MySQL, and MSSQL throughput to Fivetran, and Kafka throughput to Flink, but those comparisons are project-reported benchmarks rather than independent evaluations. Avoid it if production-ready Oracle or DB2 change-data-capture is required, because the README lists Oracle CDC as work in progress and DB2 CDC as unsupported.