[opensourcedex]

mountpoint-s3 is a free, open source data warehousing & processing project written in Rust and released under Apache-2.0. It has 5,770 GitHub stars, 252 forks and 121 open issues, and was last pushed 16 hours ago. On this registry it ranks #7 of 11 tracked projects in Data Warehousing & Processing, with 5 head-to-head comparisons available.

What is mountpoint-s3?

Mountpoint for Amazon S3 is a simple, high-throughput file client that mounts an Amazon S3 bucket as a local file system, written in Rust and released by AWS Labs under the Apache-2.0 licence for developers and operators who need file-level access to objects in S3.

What it is

Mountpoint for Amazon S3 is an open-source file client that presents an Amazon S3 bucket as a local file system. Applications access objects stored in S3 through ordinary file operations such as open and read, and Mountpoint automatically translates those operations into S3 object API calls. That translation gives an application the elastic storage and throughput of Amazon S3 behind a file interface, without the application needing to speak the S3 API itself. It lives in the AWS ecosystem, is developed in the awslabs organisation on GitHub, and is built in Rust with the FUSE topic among its listed topics.

The concrete problem it solves is the download step. Instead of copying objects from S3 to local disk before processing them, an application can read them directly through the mounted file system. It replaces the pattern of staging S3 data locally, or of hand-writing S3 SDK calls into an application that otherwise works with files. It is optimized for applications that need high read throughput on large objects, potentially from many clients at once, and for writing new objects sequentially from a single client at a time.

Key capabilities

  • Mounts an Amazon S3 bucket as a local file system through FUSE, exposing objects to file operations like open and read.
  • Translates file operations into S3 object API calls automatically, so applications do not call the S3 API directly.
  • Optimized for high read throughput on large objects, with concurrent readers across many clients.
  • Supports writing new objects sequentially from a single client at a time, including uploading output directly to S3.
  • Works with standard local tooling for uploads, such as cp from local storage.
  • Documents its POSIX deviations in doc/SEMANTICS.md and its failure modes in doc/TROUBLESHOOTING.md.
  • Ships as mount-s3 packages plus a Kubernetes path through the Mountpoint for Amazon S3 CSI driver.

Who uses it and how

  • Applications that read large objects from S3 from many instances concurrently, without downloading them to local storage first.
  • Workloads that touch only some objects out of a larger data set but cannot predict which objects in advance, since the mount resolves reads on demand.
  • Jobs that write output straight to S3, or that upload files from local storage using tools like cp.
  • Kubernetes applications, which can mount buckets through the Mountpoint for Amazon S3 CSI driver for EKS.
  • Teams running on Amazon Linux 2023, Fedora, CentOS or RHEL, and on Ubuntu, including Graviton instances via the arm64 packages.

Getting started

On Amazon Linux 2023, install directly from the repository with sudo dnf install mount-s3. Other RPM-based distributions use the mount-s3.rpm package and Ubuntu uses mount-s3.deb, both downloaded from the mountpoint-s3-release S3 bucket, with further options such as building from source and the CSI driver covered in doc/INSTALL.md.

How it compares

No paid products this project replaces are listed in the facts, and no similar tools are named there either, so Mountpoint for Amazon S3 stands alone in this registry. Its position is defined by the Apache-2.0 licence and by the fact that it is a client for one specific object store rather than a general-purpose network file system.

When to use it — and when not to

Use it when the workload is large sequential reads, selective reads from a big data set, or sequential writes of new objects, and when the application can tolerate a file system that does not implement all POSIX features. Do not pick it for directory renaming, symlinks, or edits to existing files, which S3 does not natively support, and do not run a Git repository or an editor like vim on a mount. One caveat is on record: version 1.4.0, released on January 26, 2024, contains an issue causing intermittent read failures, and users on it are advised to upgrade to v1.4.1 or later.

project readme (upstream, from github) — read inline

Mountpoint for Amazon S3

Tests Integration tests

Mountpoint for Amazon S3 is a simple, high-throughput file client for mounting an Amazon S3 bucket as a local file system. With Mountpoint for Amazon S3, your applications can access objects stored in Amazon S3 through file operations like open and read. Mountpoint for Amazon S3 automatically translates these operations into S3 object API calls, giving your applications access to the elastic storage and throughput of Amazon S3 through a file interface.

Mountpoint for Amazon S3 is optimized for applications that need high read throughput to large objects, potentially from many clients at once, and to write new objects sequentially from a single client at a time. This means it's a great fit for applications that use a file interface to:

  • read large objects from S3, potentially from many instances concurrently, without downloading them to local storage first
  • access only some S3 objects out of a larger data set, but can't predict which objects in advance
  • upload their output to S3 directly, or upload files from local storage with tools like cp

but probably not the right fit for applications that:

  • use file operations that S3 doesn't natively support, like directory renaming or symlinks
  • make edits to existing files (don't work on your Git repository or run vim in Mountpoint 😄)

Mountpoint for Amazon S3 does not implement all the features of a POSIX file system, and there are some differences that may affect compatibility with your application. See Mountpoint file system behavior for a detailed description of Mountpoint for Amazon S3's behavior and POSIX support and how they could affect your application. To troubleshoot file operations that may not be supported by Mountpoint, see the troubleshooting documentation.

Current status

Mountpoint for Amazon S3 is generally available! We're tracking future feature development on the Mountpoint for Amazon S3 public roadmap. We're always interested in feedback on features, performance, and compatibility. Please send feedback by opening a new GitHub issue or adding your input to an existing roadmap issue.

Notices

[!IMPORTANT] Mountpoint for Amazon S3 v1.4.0, released on January 26, 2024, contains an issue that causes intermittent read failures. We recommend that customers using v1.4.0 upgrade to v1.4.1 or later.

Getting started

On Amazon Linux 2023, you can install Mountpoint for Amazon S3 directly from the repository:

sudo dnf install mount-s3

On other RPM-based distributions (Fedora, CentOS, RHEL; excluding SUSE), run these two commands to install Mountpoint for Amazon S3 on your EC2 instance (for Graviton instances, replace x86_64 with arm64 in the URL):

wget https://s3.amazonaws.com/mountpoint-s3-release/latest/x86_64/mount-s3.rpm
sudo yum install -y ./mount-s3.rpm

On Ubuntu, use these commands instead (for Graviton instances, replace x86_64 with arm64 in the URL):

wget https://s3.amazonaws.com/mountpoint-s3-release/latest/x86_64/mount-s3.deb
sudo apt-get install -y ./mount-s3.deb

[!NOTE] See Installing Mountpoint for Amazon S3 for detailed instructions and other installation options, including the Mountpoint for Amazon S3 CSI driver for Kubernetes applications, or building from source.

Once you've got Mountpoint for Amazon S3 installed, you can mount your Amazon S3 bucket. You'll need valid AWS credentials to access your bucket. Mountpoint will automatically use credentials from an IAM role associated with your EC2 instance, or the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables, or from several other sources.

To mount your bucket, run this command, replacing amzn-s3-demo-bucket with the name of your bucket and /path/to/mount with the directory you want to mount the bucket to:

mount-s3 amzn-s3-demo-bucket /path/to/mount

Now you can work with your bucket contents as if they were a local file system:

ls /path/to/mount
echo "Hello World!" > /path/to/mount/Data.txt
cat /path/to/mount/Data.txt

When you're finished accessing your bucket, you can unmount it (you might need sudo):

umount /path/to/mount

[!NOTE] See Configuring Mountpoint for Amazon S3 for more details on how to configure and use Mountpoint.

Compatibility with other storage services

Mountpoint for Amazon S3 is designed for high-performance access to the Amazon S3 service. While it may be functional against other storage services that use S3-like APIs, we aren't able to provide support for those use cases, and they may inadvertently break when we make changes to better support Amazon S3. We welcome contributions of minor compatibility fixes or performance improvements for these services if the changes can be tested against Amazon S3.

Contributing

We welcome contributions to Mountpoint for Amazon S3! Please see CONTRIBUTING.md for more information on how to report bugs or submit pull requests. We especially welcome contributions to issues tagged as good first issues to work on.

For potential bug reports, see LOGGING.md for details on how to capture logging data. For performance issues, see BENCHMARKING.md for documentation about performance regression testing.

Security

If you discover a potential security issue in this project we ask that you notify AWS Security via our vulnerability reporting page. Please do not create a public GitHub issue.

Code of conduct

This project has adopted the Amazon Open Source Code of Conduct. See CODE_OF_CONDUCT.md for more details.

License

This project is licensed under the Apache-2.0 License. It builds on a number of other awesome projects with open source licenses, primarily:

See deny.toml for a list of licenses used by our dependencies.

Frequently asked questions

Is mountpoint-s3 free to use?

mountpoint-s3 is open source under the Apache-2.0 licence. There is no licence fee and no seat count — you can self-host it or, where the project offers one, pay a vendor for a managed version instead.

What does mountpoint-s3 do?

A simple, high-throughput file client for mounting an Amazon S3 bucket as a local file system.

What is mountpoint-s3 written in?

mountpoint-s3 is primarily written in Rust. Its source is publicly available at https://github.com/awslabs/mountpoint-s3, and it has 5,770 GitHub stars.