[opensourcedex]

aistore is a free, open source data warehousing & processing project written in Go and released under MIT. It has 1,938 GitHub stars, 284 forks and 5 open issues, and was last pushed 9 hours ago. On this registry it ranks #13 of 17 tracked projects in Data Warehousing & Processing, with 5 head-to-head comparisons available.

What is aistore?

AIStore (AIS) is a lightweight, MIT-licensed distributed object storage stack written in Go and tailored for AI workloads, intended for teams that need to keep training datasets and data-transformation pipelines on their own Linux or Kubernetes infrastructure.

What it is

AIStore is an elastic storage cluster that can grow and shrink at runtime and can be deployed ad hoc, with or without Kubernetes, anywhere from a single Linux machine to a bare-metal cluster of any size. It was built from scratch to provide linear scale-out, consistent performance and a flexible deployment model, and it operates natively on both in-cluster and remote data without treating either as a cache. The project lives in the Go ecosystem and is currently published at version v5.0, with intra-cluster request signing noted as enabled in v5.1. It is a reliable storage cluster rather than a client library or a gateway, and it consistently shows balanced I/O distribution and linear scalability across an arbitrary number of clustered nodes.

The concrete problem it solves is dataset placement for AI pipelines. Training and ETL jobs typically require content that lives in several clouds at once, or that has to be copied into scratch space before it can be used, which duplicates data and adds a caching layer that must be managed separately. AIS instead accesses and manages content across multiple cloud backends, including AWS S3, GCS, Azure and OCI, while giving same-name buckets distinct identity across accounts, endpoints and providers. It replaces the ad-hoc copy-and-cache tier with a cluster that presents a feature-rich native HTTP API, SDKs for Go and Python, and a compliant Amazon S3 API so unmodified S3 clients can run against it. Data transformation workloads, meanwhile, can execute I/O-intensive ETL close to the data instead of pulling objects to the compute.

Key capabilities

  • Multi-cloud access to AWS S3, GCS, Azure and OCI with fast-tier performance, configurable redundancy, and namespace-aware bucket identity so identical bucket names can coexist across accounts, endpoints and providers.
  • Deployment options spanning a minimal container-based setup documented under deploy/prod/docker/compose, Google Colab, and petascale Kubernetes clusters via NVIDIA/ais-k8s, with a stated no-built-in-limitations principle on size or functionality.
  • High availability through redundant control and data planes, self-healing, end-to-end protection, n-way mirroring, erasure coding, and an arbitrary number of lightweight access points known as AIS proxies.
  • A native HTTP API with Go and Python SDKs, alongside an Amazon S3-compatible API documented in docs/s3compat.md.
  • Chunked object representation with independently retrievable chunks, metadata v2 and checksum-protected manifests, supporting rechunking, parallel reads, multipart uploads, Get-Batch and the blob-downloader.
  • JWT-based authentication and authorization validated per request, using static keys or dynamic OIDC issuer JWKS lookup, for cluster- and bucket-level access control.
  • Observability through integrated Prometheus metrics, Grafana dashboards, configurable log verbosity and CLI-based performance tracking, plus load-aware throttling driven by a multi-dimensional load vector covering CPU, memory, disk, file descriptors and goroutines.

Who uses it and how

  • ML training teams that need a shared, linearly scalable store for training data rather than per-job local copies.
  • Data engineering groups running ETL offload, where transformations execute close to the stored data and are invoked inline.
  • Multi-cloud organisations that keep datasets in AWS S3, GCS, Azure or OCI and need one cluster to read and manage all of them, including same-named buckets.
  • Operators running Kubernetes at petascale through NVIDIA/ais-k8s, as well as small teams starting on a single Linux machine or a Google Colab notebook.
  • Environments that already standardise on S3 tooling and want existing unmodified S3 clients to work against self-hosted storage.

Getting started

The README points to a minimal container-based deployment described under deploy/prod/docker/compose, with Kubernetes-based deployments handled by the separate NVIDIA/ais-k8s repository; full documentation is hosted at https://docs.nvidia.com/aistore.

How it compares

The facts name no competing registry project to compare against, so AIStore stands alone in this registry; no list of paid products it replaces is provided, so no licence, self-hosting or cost-model contrast can be drawn. The only other products named are the cloud object stores it reads from as backends — AWS S3, GCS, Azure and OCI — and it deliberately interoperates with them through a compliant Amazon S3 API rather than positioning itself against them.

When to use it — and when not to

Choose AIStore when the workload genuinely needs a distributed cluster with redundancy, multi-cloud backends and ETL offload, because a self-hoster takes on running and monitoring that cluster, its proxies, and the Prometheus and Grafana observability stack. It is a poor fit for anyone who wants a single-binary object store or a fully managed service with no operational surface, since deployment is explicitly assumed to be on Linux hosts, virtual or physical, with Kubernetes only optional. Note also that several headline capabilities depend on version-specific behaviour: intra-cluster Ed25519 request signing arrives in v5.1 and relies on the mandatory v5.0 compatibility bridge, and the README excerpt provided here cuts off mid-sentence in its description of ETL offload, so the full transformation story should be checked against the linked documentation.

project readme (upstream, from github) — read inline

AIStore: High-Performance, Scalable Storage for AI Workloads

License Version Go Report Card

AIStore (AIS) is a lightweight distributed storage stack tailored for AI applications. It's an elastic cluster that can grow and shrink at runtime and can be ad-hoc deployed, with or without Kubernetes, anywhere from a single Linux machine to a bare-metal cluster of any size. Built from scratch, AIS provides linear scale-out, consistent performance, and a flexible deployment model.

AIS is a reliable storage cluster that can natively operate on both in-cluster and remote data, without treating either as a cache.

AIS consistently shows balanced I/O distribution and linear scalability across an arbitrary number of clustered nodes. The system supports fast data access, reliability, and rich customization for data transformation workloads.

Features

  • ✅ Multi-Cloud Access: Seamlessly access and manage content across multiple cloud backends (including AWS S3, GCS, Azure, and OCI), with fast-tier performance, configurable redundancy, and namespace-aware bucket identity (same-name buckets can coexist across accounts, endpoints, and providers).
  • ✅ Deploy Anywhere: AIS runs on any Linux machine, virtual or physical. Deployment options range from a minimal container-based deployment and Google Colab to petascale Kubernetes clusters. There are no built-in limitations on deployment size or functionality.
  • ✅ High Availability: Redundant control and data planes. Self-healing, end-to-end protection, n-way mirroring, and erasure coding. Arbitrary number of lightweight access points (AIS proxies).
  • ✅ HTTP-based API: A feature-rich, native API (with user-friendly SDKs for Go and Python), and compliant Amazon S3 API for running unmodified S3 clients.
  • ✅ Monitoring: Comprehensive observability with integrated Prometheus metrics, Grafana dashboards, detailed logs with configurable verbosity, and CLI-based performance tracking for complete cluster visibility and troubleshooting. See AIStore Observability for details.
  • ✅ Chunked Objects: High-performance chunked object representation, with independently retrievable chunks, metadata v2, and checksum-protected manifests. Supports rechunking, parallel reads, and seamless integration with Get-Batch, blob-downloader, and multipart uploads to supported cloud backends.
  • ✅ JWT Authentication and Authorization: Validates request JWTs to provide cluster- and bucket-level access control using static keys or dynamic OIDC issuer JWKS lookup.
  • ✅ Intra-Cluster Request Signing: Per-node Ed25519 identities sign and verify intra-cluster requests, signed redirects, and transport streams, with verifying keys distributed through the cluster map (enabled in v5.1, following the mandatory v5.0 compatibility bridge).
  • ✅ Load-Aware Throttling: Dynamic request throttling based on a multi-dimensional load vector (CPU, memory, disk, file descriptors, goroutines) to protect AIS clusters under stress.
  • ✅ Unified Namespace: Attach AIS clusters together to provide unified access to datasets across independent clusters, allowing users to reference shared buckets with cluster-specific identifiers.
  • ✅ Turn-key Cache: In addition to robust data protection features, AIS offers a per-bucket configurable LRU-based cache with eviction thresholds and storage capacity watermarks.
  • ✅ ETL Offload: Execute I/O intensive data transformations close to the data, either inline (on-the-fly as part of each read request) or offline (batch processing, with the destination bucket populated with transformed results).
  • ✅ Get-Batch: Retrieve multiple objects and/or archived files with a single call. Designed for ML/AI pipelines, Get-Batch fetches an entire training batch in one operation, assembling a TAR (or other supported serialization formats) that contains all requested items in the exact user-specified order (paper).
  • ✅ Data Consistency: Guaranteed consistency across all gateways, with write-through semantics in presence of remote backends.
  • ✅ Serialization & Sharding: Native, first-class support for TAR, TGZ, TAR.LZ4, and ZIP archives for efficient storage and processing of small-file datasets. Features include seamless integration with existing unmodified workflows across all APIs and subsystems.
  • ✅ Kubernetes: For production, AIS runs natively on Kubernetes. The dedicated ais-k8s repository includes the AIS K8s Operator, Ansible playbooks, Helm charts, and deployment guidance.
  • ✅ Batch Jobs: More than 30 cluster-wide batch operations that you can start, monitor, and control otherwise. The list currently includes:
$ ais show job --help

NAME:
    archive        blob-download  cleanup       copy-bucket    copy-objects      delete-objects
    download       dsort          ec-bucket     ec-get         ec-put            ec-resp
    elect-primary  etl-bucket     etl-inline    etl-objects    evict-objects     evict-remote-bucket
    get-batch      list           lru-eviction  mirror         prefetch-objects  promote-files
    put-copies     rebalance      rechunk       rename-bucket  resilver          summary
    warm-up-metadata

The feature set continues to grow and also includes: native bucket inventory (NBI); blob-downloader; AuthN - authentication and authorization server; runtime management of TLS certificates; full support for adding/removing nodes at runtime; adaptive rate limiting; and more.

For the original white paper and design philosophy, please see AIStore Overview, which also includes high-level block diagram, terminology, APIs, CLI, and more. For our 2024 KubeCon presentation, please see AIStore: Enhancing petascale Deep Learning across Cloud backends.

CLI

AIS includes an integrated, scriptable CLI for managing clusters, buckets, and objects, running and monitoring batch jobs, viewing and downloading logs, generating performance reports, and more:

$ ais <TAB-TAB>

advanced         cluster          etl              ls               prefetch         search           tls
alias            config           evict            ml               put              show             wait
archive          cp               get              mpu              remote-cluster   space-cleanup
auth             create           help             nbi              rmb              start
blob-download    download         job              object           rmo              stop
bucket           dsort            log              performance      scrub            storage

Developer Tools

AIS runs natively on Kubernetes and features open format - thus, the freedom to copy or move your data from AIS at any time using the familiar Linux tar(1), scp(1), rsync(1) and similar.

For developers and data scientists, there's also:

Quick Start

  1. Read the Getting Started Guide for a 5-minute local install, or
  2. Run a minimal container-based AIS cluster consisting of a single gateway and a single storage node, or
  3. Clone the repo and run make kill cli aisloader deploy followed by ais show cluster

Deployment options

AIS deployment options, as well as intended (development vs. production vs. first-time) usages, are all summarized here.

Prerequisites essentially boil down to having Linux with a disk. Deployment options range from a minimal container-based deployment to petascale bare-metal clusters of any size, and from a single VM to multiple racks of high-end servers. Practical use cases require, of course, further consideration.

Some of the most popular deployment options include:

Option Use Case
Local playground AIS developers or first-time users, Linux or Mac OS. Run `make kill cli aisloader deploy For performance tuning, see performance and AIS K8s Playbooks.

Existing Datasets

AIS supports multiple ingestion modes:

  • ✅ On Demand: Transparent cloud access during workloads.
  • ✅ PUT: Locally accessible files and directories.
  • ✅ Promote: Import local target directories and/or NFS/SMB shares mounted on AIS targets.
  • ✅ Copy: Full buckets, virtual subdirectories (recursively or non-recursively), lists or ranges (via Bash expansion).
  • ✅ Download: HTTP(S)-accessible datasets and objects.
  • ✅ Prefetch: Remote buckets or selected objects (from remote buckets), including subdirectories, lists, and/or ranges.
  • ✅ Archive: Group and store related small files from an original dataset.

Install from Release Binaries

You can install the CLI and benchmarking tools using:

./scripts/install_from_binaries.sh --help

The script installs aisloader and CLI from the latest or previous GitHub release and enables CLI auto-completions.

PyTorch integration

PyTorch integration is a growing set of datasets (both iterable and map-style), samplers, and dataloaders:

AIStore Badge

Let others know your project is powered by high-performance AI storage:

aistore

[![aistore](https://img.shields.io/badge/powered%20by-AIStore-76B900?style=flat&labelColor=000000)](https://github.com/NVIDIA/aistore)

More Docs & Guides

How to find information

  • See Extended Index
  • Use CLI search command, e.g.: ais search copy
  • Clone the repository and run git grep, e.g.: git grep -n out-of-band -- "*.md"

License

MIT

Author

Alex Aizman (NVIDIA)

Frequently asked questions

Is aistore free to use?

aistore is open source under the MIT licence. There is no licence fee and no seat count — you can self-host it or, where the project offers one, pay a vendor for a managed version instead.

What does aistore do?

AIStore: scalable storage for AI applications

What is aistore written in?

aistore is primarily written in Go. Its source is publicly available at https://github.com/NVIDIA/aistore, and it has 1,938 GitHub stars.