[opensourcedex]

SlateDB is a free, open source databases project written in Rust and released under Apache-2.0. It has 3,414 GitHub stars, 298 forks and 184 open issues, and was last pushed 6 hours ago. On this registry it ranks #64 of 81 tracked projects in Databases, with 5 head-to-head comparisons available. It gained 8 stars over the last 6 tracked days.

What is SlateDB?

What it is

SlateDB is an embedded storage engine written in Rust and distributed under the Apache-2.0 license. It is built as a log-structured merge-tree, and it stores data in object storage rather than on a local disk or dedicated server. The project lives in the Rust and cloud-native storage ecosystem, and it exposes a key-value interface through a Rust crate.

The concrete problem it addresses is the mismatch between embedded local storage engines and cloud object storage. Traditional embedded LSM-tree engines keep data on local disk, which limits capacity, durability, and replication options when an application needs to scale across cloud buckets. SlateDB writes to object storage services such as S3, GCS, ABS, MinIO, and Tigris, so an application can use those services from inside a Rust process.

Key capabilities

  • Stores data as an LSM-tree, with in-memory WAL and MemTable components before flushing to object storage.
  • Batches writes into string-sorted tables, with configurable flush interval, to reduce object storage PUT costs.
  • Returns a WriteHandle after put, and supports handle.await_durable().await or db.flush().await to wait for durability.
  • Supports put, get, delete, range scans, and seek.
  • Uses in-memory block caches, compression, bloom filters, and local SST disk caches to reduce read latency and GET costs.
  • Integrates with object storage through the object_store crate, and supports any ObjectStore implementation.

Who uses it and how

  • Rust applications can embed SlateDB as a local key-value store while persisting tables to a cloud bucket, using Db::open with an ObjectStore instance.
  • Developers can use an in-memory object store for tests or examples, then switch to S3, GCS, ABS, MinIO, or Tigris in a deployed Rust service.
  • Applications that write many small records can batch writes into MemTables and flush them periodically as SST files, rather than issuing one PUT per put call.
  • Read-heavy workloads can combine in-memory block caches, bloom filters, compression, and local SST disk caches to serve range scans and point lookups from an embedded Rust process.

Getting started

Add slatedb and tokio to Cargo.toml, then call Db::open with a path and an ObjectStore implementation in Rust. The README example uses an in-memory object store for local development.

When to use it — and when not to

Use SlateDB when a Rust application needs an embedded LSM-tree key-value engine whose durable state lives in object storage, and when bottomless storage capacity, durability, and replication matter more than local-disk latency. Avoid it when the workload requires consistently low-latency writes or reads from local disk, because object storage has higher latency and higher API cost. A self-hoster must operate the object storage target, while the engine relies on configurable flush intervals, in-memory caches, and local SST disk caches; the provided metadata also lists a repository age of zero years, zero contributors, and 184 open issues.

project readme (upstream, from github) — read inline
SlateDB

Crates.io Version GitHub License slatedb.io Discord Docs Dosu OpenCollective

Introduction

SlateDB is an embedded storage engine built as a log-structured merge-tree. Unlike traditional LSM-tree storage engines, SlateDB writes data to object storage (S3, GCS, ABS, MinIO, Tigris, and so on). Leveraging object storage allows SlateDB to provide bottomless storage capacity, high durability, and easy replication. The trade-off is that object storage has a higher latency and higher API cost than local disk.

To mitigate high write API costs (PUTs), SlateDB batches writes. Rather than writing every put() call to object storage, MemTables are flushed periodically to object storage as a string-sorted table (SST). The flush interval is configurable.

Write operations return a WriteHandle after updating the in-memory WAL and MemTable. Call handle.await_durable().await to wait for one write to become durable, or call db.flush().await to flush all pending writes.

To mitigate read latency and read API costs (GETs), SlateDB will use standard LSM-tree caching techniques: in-memory block caches, compression, bloom filters, and local SST disk caches.

Checkout slatedb.io to learn more.

Get Started

Add the following to your Cargo.toml:

[dependencies]
slatedb = "*"
tokio = "*"

Then you can use SlateDB in your Rust code:

use slatedb::{Db, Error};
use slatedb::object_store::{ObjectStore, memory::InMemory};
use std::sync::Arc;

#[tokio::main]
async fn main() -> Result<(), Error> {
    // Setup
    let object_store: Arc<dyn ObjectStore> = Arc::new(InMemory::new());
    let kv_store = Db::open("/tmp/test_kv_store", object_store).await?;

    // Put
    let key = b"test_key";
    let value = b"test_value";
    kv_store.put(key, value).await?;

    // Get
    assert_eq!(
        kv_store.get(key).await?,
        Some("test_value".into())
    );

    // Delete
    kv_store.delete(key).await?;
    assert!(kv_store.get(key).await?.is_none());

    kv_store.put(b"test_key1", b"test_value1").await?;
    kv_store.put(b"test_key2", b"test_value2").await?;
    kv_store.put(b"test_key3", b"test_value3").await?;
    kv_store.put(b"test_key4", b"test_value4").await?;

    // Scan over unbound range
    let mut iter = kv_store.scan(..).await?;
    let mut count = 1;
    while let Ok(Some(item)) = iter.next().await {
        assert_eq!(
            item.key,
            format!("test_key{count}").into_bytes()
        );
        assert_eq!(
            item.value,
            format!("test_value{count}").into_bytes()
        );
        count += 1;
    }

    // Scan over bound range
    let mut iter = kv_store.scan("test_key1"..="test_key2").await?;
    let item = iter.next().await?.expect("missing test_key1");
    assert_eq!(item.key.as_ref(), b"test_key1");
    assert_eq!(item.value.as_ref(), b"test_value1");
    let item = iter.next().await?.expect("missing test_key2");
    assert_eq!(item.key.as_ref(), b"test_key2");
    assert_eq!(item.value.as_ref(), b"test_value2");

    // Seek ahead to next key
    let mut iter = kv_store.scan(..).await?;
    let next_key = b"test_key4";
    iter.seek(next_key).await?;
    let item = iter.next().await?.expect("missing test_key4");
    assert_eq!(item.key.as_ref(), b"test_key4");
    assert_eq!(item.value.as_ref(), b"test_value4");
    assert_eq!(iter.next().await?, None);

    // Close
    kv_store.close().await?;

    Ok(())
}

SlateDB uses the object_store crate to interact with object storage, and therefore supports any object storage that implements the ObjectStore trait. You can use the crate in your project to interact with any object storage that implements the ObjectStore trait. SlateDB also re-exports the object_store crate for your convenience.

Documentation

Visit slatedb.io to learn more.

Bindings

Features

  • Basic API (get, put, delete)
  • SSTs on object storage
  • Range queries (#8)
  • Block cache (#15)
  • Disk cache (#9)
  • Compression (#10)
  • Bloom filters (#11)
  • Manifest persistence (#14)
  • Compaction (#7)
  • Transactions (#785)
  • Merge operator (#328)
  • Clones (#49)
  • Range deletions (#577)
  • Change data capture (CDC) (#249)
  • Database split/merge (RFC)

Projects

Check out CONTRIBUTING.md for fun (and useful) projects to work on.

Release Schedule

SlateDB follows Semantic Versioning. We release new versions approximately every 2 months at the end of each even month (February, April, and so on). We guarantee forward/backward compatibility for storage formats between adjacent versions, but we do not currently guarantee API compatibility at this time (we reserve the right to break compile-time API compatibility).

Adopters

See who's using SlateDB.

Talks

Infrastructure Sponsors

Thanks to the following companies for donating services and infrastructure to the SlateDB project.

  • Pulumi - Open source platform for automating, securing, and m

readme truncated — read the full docs on github

Frequently asked questions

Is SlateDB free to use?

SlateDB is open source under the Apache-2.0 licence. There is no licence fee and no seat count — you can self-host it or, where the project offers one, pay a vendor for a managed version instead.

What does SlateDB do?

Embedded storage engine built on object storage

What is SlateDB written in?

SlateDB is primarily written in Rust. Its source is publicly available at https://github.com/slatedb/slatedb, and it has 3,414 GitHub stars.