[opensourcedex]

lakeFS is a free, open source data warehousing & processing project written in Go and released under Apache-2.0. It has 5,535 GitHub stars, 477 forks and 443 open issues, and was last pushed 4 days ago. On this registry it ranks #9 of 13 tracked projects in Data Warehousing & Processing, with 5 head-to-head comparisons available.

What is lakeFS?

lakeFS is an open-source data version control tool — commonly described as Git for data — that turns object storage into a versioned, Git-like repository, and it is built for data engineers, analytics engineers, and data scientists who need repeatable, atomic, and auditable data lake operations.

What it is

lakeFS is a Go project released under the Apache-2.0 licence, with documentation hosted at docs.lakefs.io. It transforms object storage into a Git-like repository so that a data lake can be managed the way source code is managed. The underlying storage service can be AWS S3, Azure Blob Storage, or Google Cloud Storage, and lakeFS is API compatible with S3. Because of that compatibility it works with modern data frameworks such as Apache Spark, Apache Spark SQL, Hive, AWS Athena, DuckDB, and Presto without those tools needing to change how they read and write objects.

The concrete problem it solves is that a data lake normally keeps only one state of the data: the current one. That makes debugging a data issue difficult, makes machine learning training accuracy hard to validate because re-running a model over different data gives different results, and makes data audits hard to satisfy. The usual workaround is to create and maintain several full environments — one for development, one for testing, one for production — which is slow and expensive in storage. lakeFS replaces that copying model with branches: a branch gives a copy of the full production data without copying anything, so ETL and transformation jobs can be tested against realistic data before they touch production.

Key capabilities

  • Branching over object storage, giving an isolated copy of production data without duplicating the underlying objects.
  • S3 API compatibility, so existing frameworks such as Spark, Hive, AWS Athena, DuckDB, and Presto connect without modification.
  • Hooks that gate publication, ensuring only data that has passed tests becomes part of production.
  • Support for the Write-Audit-Publish pattern, where quality and governance checks run before data is deployed for business use.
  • Reproducibility: the state of the data at any point in time can be reproduced, which supports debugging, ML accuracy validation, and data audits.
  • Three storage backends: AWS S3, Azure Blob Storage, and Google Cloud Storage.
  • Quickstart launchers for local evaluation, through pip install lakefs with python -m lakefs.quickstart, or the Docker image treeverse/lakefs:latest.

Who uses it and how

  • ETL developers who need isolated development and test environments, so pipeline changes can be validated against a copy of production data before downstream consumers are affected.
  • Data quality and governance teams that use hooks to enforce checks such as file format validation, schema checks, and removal of personally identifiable information before data is published.
  • Machine learning teams that re-run models against an exact prior state of the data to keep training results consistent.
  • Auditors and compliance functions that need more than the current state of the data to reconstruct what was present at a given time.
  • Analytics teams already running Spark, Hive, Athena, DuckDB, or Presto, with end-to-end examples available in the lakeFS samples repository.

Getting started

The sandbox path is pip install lakefs followed by python -m lakefs.quickstart, after which the web interface is available at http://127.0.0.1:8000/. The Docker alternative is docker run --pull always --name lakefs -p 8000:8000 treeverse/lakefs:latest run --quickstart.

How it compares

The facts provided for this entry do not list any paid products that lakeFS replaces, and they do not name any directly comparable tool either. On the evidence available, lakeFS stands alone in this registry.

When to use it — and when not to

It fits teams running a data lake on AWS S3, Azure Blob Storage, or Google Cloud Storage that need branching, rollback, reproducibility, and quality gates on production data. The trade-off is maturity and maintenance attention: the project carries 443 open issues against 477 forks, which is a substantial open backlog, and the README excerpt provided is sparse and ends mid-section, so the full documentation at docs.lakefs.io should be treated as the real reference. Teams without object storage, or those who want a managed service rather than a self-operated one, should look elsewhere, as no hosted option is named in the facts.

project readme (upstream, from github) — read inline

Apache License Go tests status Node tests status Integration tests status Artifact HUB code of conduct

lakeFS is Data Version Control (Git for Data)

lakeFS is an open-source tool that transforms your object storage into a Git-like repository. It enables you to manage your data lake the way you manage your code.

With lakeFS you can build repeatable, atomic, and versioned data lake operations - from complex ETL jobs to data science and analytics.

lakeFS supports AWS S3, Azure Blob Storage, and Google Cloud Storage as its underlying storage service. It is API compatible with S3 and works seamlessly with all modern data frameworks such as Spark, Hive, AWS Athena, DuckDB, and Presto.

For more information, see the documentation.

Getting Started

You can spin up a standalone sandbox instance of lakeFS:

pip install lakefs
python -m lakefs.quickstart

Once you've got lakeFS running, open http://127.0.0.1:8000/ in your web browser.

Getting Started with Docker

Alternatively, you can run lakeFS a server using Docker:

docker run --pull always \
		   --name lakefs \
		   -p 8000:8000 \
		   treeverse/lakefs:latest \
		   run --quickstart

Quickstart

👉🏻 For a hands-on walk through of the core functionality in lakeFS head over to the quickstart to jump right in!

Make sure to also have a look at the lakeFS samples. These are a rich resource of examples of end-to-end applications that you can build with lakeFS.

Why Do I Need lakeFS?

ETL Testing with Isolated Dev/Test Environment

When working with a data lake, it’s useful to have replicas of your production environment. These replicas allow you to test these ETLs and understand changes to your data without impacting downstream data consumers.

Running ETL and transformation jobs directly in production without proper ETL Testing is a guaranteed way to have data issues flow into dashboards, ML models, and other consumers sooner or later. The most common approach to avoid making changes directly in production is to create and maintain multiple data environments and perform ETL testing on them. Dev environment to develop the data pipelines and test environment where pipeline changes are tested before pushing it to production. With lakeFS you can create branches, and get a copy of the full production data, without copying anything. This enables a faster and easier process of ETL testing.

Reproducibility

Data changes frequently. This makes the task of keeping track of its exact state over time difficult. Oftentimes, people maintain only one state of their data––its current state.

This has a negative impact on the work, as it becomes hard to:

  • Debug a data issue.
  • Validate machine learning training accuracy (re-running a model over different data gives different results). Comply with data audits.

In comparison, lakeFS exposes a Git-like interface to data that allows keeping track of more than just the current state of data. This makes reproducing its state at any point in time straightforward.

Write-Audit-Publish

Data pipelines feed processed data from data lakes to downstream consumers like business dashboards and machine learning models. As more and more organizations rely on data to enable business critical decisions, data reliability and trust are of paramount concern. Thus, it’s important to ensure that production data adheres to the data governance policies of businesses. These data governance requirements can be as simple as a file format validation, schema check, or an exhaustive PII(Personally Identifiable Information) data removal from all of organization’s data.

Thus, to ensure the quality and reliability at each stage of the data lifecycle, data quality gates need to be implemented. That is, we need to run quality and correctness tests on the data, and only if data governance requirements are met can the data can be published to production for business use.

Everytime there is an update to production data, the best practice would be to run tests and then publish (deploy) the data to production. With lakeFS you can create hooks that make sure that only data that passed these tests will become part of production.

Rollback

A rollback operation is used to to fix critical data errors immediately.

What is a critical data error? Think of a situation where erroneous or misformatted data causes a signficant issue with an important service or function. In such situations, the first thing to do is stop the bleeding.

Rolling back returns data to a state in the past, before the error was present. You might not be showing all the latest data after a rollback, but at least you aren’t showing incorrect data or raising errors. Since lakeFS provides versions of the data without making copies of the data, you can time travel between versions and roll back to the version of the data before the error was presented.

Community

Stay up to date and get lakeFS support via:

  • Share your lakeFS experience and get support on our Slack.
  • Follow us and join the conversation on Twitter.
  • Learn from video tutorials on our YouTube channel.
  • Read more on data versioning and other data lake best practices in our blog.
  • Feel free to contact us about anything else.

More information

Licensing

lakeFS is completely free and open-source and licensed under the Apache 2.0 License.

Treeverse considers that any sharing of source code with an AI agent, and any output an AI agent generates based on our source code, without the appropriate notices or attribution requirements, violates the terms of the Apache 2.0 license to which this source code is subject. We also do not consider AI provider guardrails as sufficient to prevent copyright infringement. Treeverse reserves all of its rights to pursue legal action against any party that inputs LakeFS code into an AI agent.

Who Uses lakeFS?

lakeFS is used by numerous companies, including those below. If you use lakeFS and would like to be included here please open a PR.

  • AirAsia
  • APEX Global
  • AppsFlyer
  • Auburn University
  • BAE Systems
  • Bureau of Labor Statistics
  • Cambridge Consultants
  • Connor, Clark & Lunn Financial Group
  • Context Labs Bv
  • Daimler Truck
  • Enigma
  • EPCOR
  • Ford Motor Company
  • Generali
  • Giesecke+Devrient
  • greehill
  • Karius
  • Luxonis
  • Mixpeek
  • Netflix
  • Paige
  • PETRONAS
  • Pollinate
  • Proton Technologies AG
  • ProtonMail
  • Renaissance Computing Institute
  • RHEA Group
  • RMS
  • Sensum
  • Similarweb
  • State Street Global Advisors
  • Terramera
  • Tredence
  • Volvo Cars
  • Webiks
  • Windward
  • Woven by Toyota

Frequently asked questions

Is lakeFS free to use?

lakeFS is open source under the Apache-2.0 licence. There is no licence fee and no seat count — you can self-host it or, where the project offers one, pay a vendor for a managed version instead.

What does lakeFS do?

lakeFS - Data version control for your data lake | Git for data

What is lakeFS written in?

lakeFS is primarily written in Go. Its source is publicly available at https://github.com/treeverse/lakeFS, and it has 5,535 GitHub stars.