juicefs: Open Source Databases

What is juicefs?

JuiceFS is a Go-based, Apache-2.0 distributed POSIX file system that stores file data in object storage and metadata in a database engine such as Redis, MySQL, SQLite, or TiKV, and it is aimed at cloud-native infrastructure, big data, machine learning, and AI workloads.

What it is

JuiceFS is a shared, distributed POSIX file system. It splits responsibilities across three parts: the JuiceFS Client, which coordinates object storage and the metadata engine and implements the file system interfaces for POSIX, Hadoop, Kubernetes, and the S3 gateway; a data storage layer, which can be local disk, public or private cloud object storage, or HDFS; and a metadata engine, which holds file names, sizes, permission groups, creation and modification timestamps, and directory structure. Each file is split into "Chunk"s at a fixed size with a default upper limit of 64 MiB, and each Chunk is composed of one or more "Slice"s.

The concrete problem is the gap between cheap, scalable object storage and applications that expect a local file system. Object storage does not expose POSIX semantics, so big data, machine learning, and AI platforms cannot mount it directly. JuiceFS puts a POSIX-compatible file system layer over that storage, so massive cloud storage can be attached to those platforms in production without modifying application code, and the storage behaves as efficiently as local storage. It also makes the same file system shareable: thousands of clients can read and write it, and a confirmed modification is immediately visible on all servers mounted with the same file system.

Key capabilities

Fully POSIX-compatible, so existing applications mount it without breaking their workflow.
Fully Hadoop-compatible through a Hadoop Java SDK supporting Hadoop 2.x and Hadoop 3.x plus a range of Hadoop ecosystem components.
S3-compatible access through the JuiceFS S3 Gateway.
Cloud-native deployment through a Kubernetes CSI Driver.
Strong consistency: a confirmed modification is immediately visible to every server mounting the same file system.
Global file locks, covering both BSD locks (flock) and POSIX record locks (fcntl).
Data compression with LZ4 or Zstandard, and data encryption in transit and at rest.

Who uses it and how

Big data platforms: Hadoop 2.x and 3.x clusters and Hadoop ecosystem components consume JuiceFS through the Hadoop Java SDK.
Kubernetes clusters: workloads mount JuiceFS as a persistent volume through the Kubernetes CSI Driver.
Machine learning, artificial intelligence, and big data application platforms, where cloud object storage is presented as local storage without code changes.
Shared multi-client storage: deployments where thousands of clients read and write the same file system and require immediate visibility of confirmed changes.
S3-speaking applications, which reach JuiceFS through the S3 Gateway rather than a native mount.

Getting started

The documented entry point is the Quick Start Guide at https://juicefs.com/docs/community/quick_start_guide. The README excerpt does not name an install command, package name, or Docker image, so the guide is the authoritative starting reference.

How it compares

No list of paid products that JuiceFS replaces, and no other comparable tools by product name, appear in the facts provided. On the evidence here, it stands alone in this registry.

When to use it — and when not to

Adopting JuiceFS means operating more than one system: a metadata engine chosen from Redis, MySQL, SQLite, or TiKV, plus the underlying data storage, whether local disk, cloud object storage, or HDFS. Teams that cannot run and back up a metadata database, or that want a single self-contained file system with no external dependency, should not pick it. The project also carries 203 open issues, and the facts given here do not specify a release cadence or carry the full installation procedure, so the current documentation at juicefs.com should be checked before committing to a production deployment.

project readme (upstream, from github) — read inline

JuiceFS is a high-performance POSIX file system released under Apache License 2.0, particularly designed for the cloud-native environment. The data, stored via JuiceFS, will be persisted in Object Storage (e.g. Amazon S3), and the corresponding metadata can be persisted in various compatible database engines such as Redis, MySQL, and TiKV based on the scenarios and requirements.

With JuiceFS, massive cloud storage can be directly connected to big data, machine learning, artificial intelligence, and various application platforms in production environments. Without modifying code, the massive cloud storage can be used as efficiently as local storage.

📖 Document: Quick Start Guide

Highlighted Features

Fully POSIX-compatible: Use as a local file system, seamlessly docking with existing applications without breaking business workflow.
Fully Hadoop-compatible: JuiceFS' Hadoop Java SDK is compatible with Hadoop 2.x and Hadoop 3.x as well as a variety of components in the Hadoop ecosystems.
S3-compatible: JuiceFS' S3 Gateway provides an S3-compatible interface.
Cloud Native: A Kubernetes CSI Driver is provided for easily using JuiceFS in Kubernetes.
Shareable: JuiceFS is a shared file storage that can be read and written by thousands of clients.
Strong Consistency: The confirmed modification will be immediately visible on all the servers mounted with the same file system.
Outstanding Performance: The latency can be as low as a few milliseconds, and the throughput can be expanded nearly unlimitedly (depending on the size of the Object Storage). Test results
Data Encryption: Supports data encryption in transit and at rest (please refer to the guide for more information).
Global File Locks: JuiceFS supports both BSD locks (flock) and POSIX record locks (fcntl).
Data Compression: JuiceFS supports LZ4 or Zstandard to compress all your data.

Architecture

JuiceFS consists of three parts:

JuiceFS Client: Coordinates Object Storage and metadata storage engine as well as implementation of file system interfaces such as POSIX, Hadoop, Kubernetes, and S3 gateway.
Data Storage: Stores data, with supports of a variety of data storage media, e.g., local disk, public or private cloud Object Storage, and HDFS.
Metadata Engine: Stores the corresponding metadata that contains information of file name, file size, permission group, creation and modification time and directory structure, etc., with supports of different metadata engines, e.g., Redis, MySQL, SQLite and TiKV.

JuiceFS Architecture

JuiceFS can store the metadata of file system on different metadata engines, like Redis, which is a fast, open-source, in-memory key-value data storage, particularly suitable for storing metadata; meanwhile, all the data will be stored in Object Storage through JuiceFS client. Learn more

data-structure-diagram

Each file stored in JuiceFS is split into "Chunk" s at a fixed size with the default upper limit of 64 MiB. Each Chunk is composed of one or more "Slice"(s), and the length of the slice varies depending on how the file is written. Each slice is composed of size-fixed "Block" s, which are 4 MiB by default. These blocks will be stored in Object Storage in the end; at the same time, the metadata information of the file and its Chunks, Slices, and Blocks will be stored in metadata engines via JuiceFS. Learn more

How JuiceFS stores your files

When using JuiceFS, files will eventually be split into Chunks, Slices and Blocks and stored in Object Storage. Therefore, the source files stored in JuiceFS cannot be found in the file browser of the Object Storage platform; instead, there are only a chunks directory and a bunch of digitally numbered directories and files in the bucket. Don't panic! This is just the secret of the high-performance operation of JuiceFS!

Getting Started

Before you begin, make sure you have:

One supported metadata engine, see How to Set Up Metadata Engine
One supported Object Storage for storing data blocks, see Supported Object Storage
JuiceFS Client downloaded and installed

Please refer to Quick Start Guide to start using JuiceFS right away!

Command Reference

Check out all the command line options in command reference.

Containers

JuiceFS can be used as a persistent volume for Docker and Podman, please check here for details.

Kubernetes

It is also very easy to use JuiceFS on Kubernetes. Please find more information here.

Hadoop Java SDK

If you wanna use JuiceFS in Hadoop, check Hadoop Java SDK.

Advanced Topics

Please refer to JuiceFS Document Center for more information.

POSIX Compatibility

JuiceFS has passed all of the compatibility tests (8813 in total) in the latest pjdfstest .

All tests successful.

Test Summary Report
-------------------
/root/soft/pjdfstest/tests/chown/00.t          (Wstat: 0 Tests: 1323 Failed: 0)
  TODO passed:   693, 697, 708-709, 714-715, 729, 733
Files=235, Tests=8813, 233 wallclock secs ( 2.77 usr  0.38 sys +  2.57 cusr  3.93 csys =  9.65 CPU)
Result: PASS

Aside from the POSIX features covered by pjdfstest, JuiceFS also provides:

Close-to-open consistency. Once a file is written and closed, it is guaranteed to view the written data in

Frequently asked questions

Is juicefs free to use?

juicefs is open source under the Apache-2.0 licence. There is no licence fee and no seat count — you can self-host it or, where the project offers one, pay a vendor for a managed version instead.

What does juicefs do?

JuiceFS is a distributed POSIX file system built on top of Redis and S3.

What is juicefs written in?

juicefs is primarily written in Go. Its source is publicly available at https://github.com/juicedata/juicefs, and it has 14,436 GitHub stars.