[opensourcedex]

delta is a free, open source marketing & customer engagement project written in Scala and released under Apache-2.0. It has 8,999 GitHub stars, 2,176 forks and 955 open issues, and was last pushed 6 hours ago. On this registry it ranks #17 of 50 tracked projects in Marketing & Customer Engagement, with 5 head-to-head comparisons available.

What is delta?

Delta Lake is an open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive, aimed at data engineers and platform teams who keep large analytical tables on existing storage and need transactional guarantees across several engines.

What it is

Delta Lake is the storage framework maintained in the delta.io GitHub organization, alongside the delta-rs, delta-sharing, kafka-delta-ingest, and website repositories. It defines a table format and a transaction log, and it implements the Delta Transaction Log Protocol to give reads and writes on those tables their transactional guarantees. The project ships libraries and connectors rather than a server: APIs for Scala, Java, Rust, Ruby, and Python, plus engine integrations that let Spark, PrestoDB, Flink, Trino, and Hive address the same tables.

The concrete problem is that analytical data lives as files that engines read and write with no shared notion of a transaction. Delta Lake places a protocol-governed log on top of that storage, so multiple engines and jobs can read from and write to one table without stepping on each other. It replaces the informal arrangement of raw file directories on an underlying storage system with a table definition that the project's own connectors and the Delta Standalone library both understand.

Key capabilities

  • ACID transactions on Delta tables, implemented through the Delta Transaction Log Protocol as described in the Delta Standalone library.
  • Apache Spark connector that allows Spark to read from and write to Delta Lake.
  • Trino connector that allows Trino to read from and write to Delta Lake, and PrestoDB and Apache Hive connectors that read from it.
  • Apache Flink connector, marked Preview, that allows Flink to write to Delta Lake.
  • Delta Standalone, a single-node Java library that lets Scala- and Java-based projects, including Apache Flink, Apache Hive, Apache Beam, and PrestoDB, read from and write to Delta tables.
  • Delta Rust API, the deltalake crate with Python and Ruby bindings, giving low-level access intended for frameworks such as datafusion, ballista, rust-dataframe, and vega.
  • Published Python package delta-spark on PyPI, with API documentation for Scala, Java, and Python.

Who uses it and how

  • Teams running Spark jobs as the main compute engine, using the Spark connector for both reads and writes against Delta tables.
  • Organizations with several engines on one dataset, reading through PrestoDB or Hive and reading and writing through Trino against the same table.
  • Scala and Java projects that need table access without a full engine, using Delta Standalone to work directly with a table's metadata in the transaction log.
  • Rust and Python data-processing pipelines built on datafusion, ballista, rust-dataframe, or vega, using the Delta Rust API and its bindings.
  • Streaming ingest setups, represented in the organization by the kafka-delta-ingest repository, feeding Delta tables from Kafka.

Getting started

The Quick Start Guide at https://docs.delta.io/latest/quick-start.html covers Scala, Java, and Python, and the Python path uses the delta-spark package published on PyPI. Latest binaries and release details are listed in the online documentation at https://docs.delta.io/latest/.

How it compares

No list of paid products that this project replaces is provided in the facts, and the facts name no directly comparable table-format project. It stands alone in this registry as the storage framework for the Lakehouse architecture described in its own README.

When to use it — and when not to

A self-hoster must run and maintain the underlying storage systems and satisfy the project's stated requirements for those systems, and must reason about the concurrency control model documented in the README. Anyone who wants a managed, turnkey analytical service with no engine to operate should not start here, since Delta Lake is a framework integrated into Spark, Flink, Trino, PrestoDB, Hive, or a Scala, Java, Rust, Ruby, or Python application. The repository carries 955 open issues and the README excerpt leaves compatibility, road map, and build details to external documentation, so adopters should expect to read the docs site rather than the repository page alone; this registry also files the project under Business Software / Marketing & Customer Engagement, which does not reflect a data storage framework.

project readme (upstream, from github) — read inline

Test License PyPI PyPI - Downloads

Delta Lake is an open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs for Scala, Java, Rust, Ruby, and Python.

The following are some of the more popular Delta Lake integrations, refer to delta.io/integrations for the complete list:

  • Apache Spark™: This connector allows Apache Spark™ to read from and write to Delta Lake.
  • Apache Flink (Preview): This connector allows Apache Flink to write to Delta Lake.
  • PrestoDB: This connector allows PrestoDB to read from Delta Lake.
  • Trino: This connector allows Trino to read from and write to Delta Lake.
  • Delta Standalone: This library allows Scala and Java-based projects (including Apache Flink, Apache Hive, Apache Beam, and PrestoDB) to read from and write to Delta Lake.
  • Apache Hive: This connector allows Apache Hive to read from Delta Lake.
  • Delta Rust API: This library allows Rust (with Python and Ruby bindings) low level access to Delta tables and is intended to be used with data processing frameworks like datafusion, ballista, rust-dataframe, vega, etc.

Table of Contents

Latest Binaries

See the online documentation for the latest release.

API Documentation

Compatibility

Delta Standalone library is a single-node Java library that can be used to read from and write to Delta tables. Specifically, this library provides APIs to interact with a table’s metadata in the transaction log, implementing the Delta Transaction Log Protocol to achieve the transactional guarantees of the Delta Lake format.

API Compatibility

There are two types of APIs provided by the Delta Lake project.

  • Direct Java/Scala/Python APIs - The classes and methods documented in the API docs are considered as stable public APIs. All other classes, interfaces, methods that may be directly accessible in code are considered internal, and they are subject to change across releases.
  • Spark-based APIs - You can read Delta tables through the DataFrameReader/Writer (i.e. spark.read, df.write, spark.readStream and df.writeStream). Options to these APIs will remain stable within a major release of Delta Lake (e.g., 1.x.x).
  • See the online documentation for the releases and their compatibility with Apache Spark versions.

Data Storage Compatibility

Delta Lake guarantees backward compatibility for all Delta Lake tables (i.e., newer versions of Delta Lake will always be able to read tables written by older versions of Delta Lake). However, we reserve the right to break forward compatibility as new features are introduced to the transaction protocol (i.e., an older version of Delta Lake may not be able to read a table produced by a newer version).

Breaking changes in the protocol are indicated by incrementing the minimum reader/writer version in the Protocol action.

Roadmap

Transaction Protocol

Delta Transaction Log Protocol document provides a specification of the transaction protocol.

Requirements for Underlying Storage Systems

Delta Lake ACID guarantees are predicated on the atomicity and durability guarantees of the storage system. Specifically, we require the storage system to provide the following.

  1. Atomic visibility: There must be a way for a file to be visible in its entirety or not visible at all.
  2. Mutual exclusion: Only one writer must be able to create (or rename) a file at the final destination.
  3. Consistent listing: Once a file has been written in a directory, all future listings for that directory must return that file.

See the online documentation on Storage Configuration for details.

Concurrency Control

Delta Lake ensures serializability for concurrent reads and writes. Please see Delta Lake Concurrency Control for more details.

Reporting issues

We use GitHub Issues to track community reported issues. You can also contact the community for getting answers.

Contributing

We welcome contributions to Delta Lake. See our CONTRIBUTING.md for more details.

We also adhere to the Delta Lake Code of Conduct.

Building

Delta Lake is compiled using SBT. Ensure that your Java version is at least 17 (you can verify with java -version).

To compile, run

build/sbt compile

To generate artifacts, run

build/sbt package

To execute tests, run

build/sbt test

To execute a single test suite, run

build/sbt spark/'testOnly org.apache.spark.sql.delta.optimize.OptimizeCompactionSQLSuite'

To execute a single test within and a single test suite, run

build/sbt spark/'testOnly *.OptimizeCompactionSQLSuite -- -z "optimize command: on partitioned table - all partitions"'

Refer to SBT docs for more commands.

Running python tests locally

Setup Environment

Install Conda (Skip if you already installed it)

Follow Conda Download to install Anaconda.

Create an environment from environment file

Follow Create Environment From Environment file to create a Conda environment from /python/environment.yml and activate the newly created delta_python_tests environment.

# Note the `--file` argument should be a fully qualified path. Using `~` in file
# path doesn't work. Example valid path: `/Users/macuser/delta/python/environment.yml`

conda env create --name delta_python_tests --file=<absolute_path_to_delta_repo>/python/environment.yml`
JDK Setup

B

readme truncated — read the full docs on github

Frequently asked questions

Is delta free to use?

delta is open source under the Apache-2.0 licence. There is no licence fee and no seat count — you can self-host it or, where the project offers one, pay a vendor for a managed version instead.

What does delta do?

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs

What is delta written in?

delta is primarily written in Scala. Its source is publicly available at https://github.com/delta-io/delta, and it has 8,999 GitHub stars.