[opensourcedex]

backend.ai is a free, open source build & deployment project written in Python and released under LGPL-3.0. It has 669 GitHub stars, 183 forks and 1,617 open issues, and was last pushed 6 hours ago. On this registry it ranks #56 of 59 tracked projects in Build & Deployment, with 5 head-to-head comparisons available.

What is backend.ai?

Backend.AI is an open-source, container-based computing cluster platform that hosts machine learning frameworks and diverse programming languages and allocates isolated multi-tenant compute sessions on Docker or Podman with pluggable accelerator support behind REST and GraphQL APIs.

What it is

Backend.AI is a Python project published on PyPI as backend.ai-manager. It turns a pool of container hosts into a shared cluster: sessions, called kernels, are scheduled and isolated for many tenants at once, on demand or in batches, by the project's own orchestrator, Sokovan. The platform hosts popular computing and machine learning frameworks alongside general programming languages, and reaches hardware through pluggable accelerator backends covering CUDA GPU, ROCm GPU, Rebellions, FuriosaAI, HyperAccel, Intel Gaudi, Tenstorrent, Google TPU, GraphCore IPU and other NPUs. Every function is exposed as a REST and a GraphQL API, so the web UI, the CLI and third-party clients share one surface.

The concrete problem is accelerator allocation across tenants sharing a container host pool. What it replaces is a hand-assembled scheduler: instead of writing on-demand and batch dispatch, resource isolation and per-session plumbing on top of a bare container engine, a deployment gets Sokovan and customizable job schedulers as part of the platform, with the same stack hosting the frameworks and tunneling users into sessions over WebSocket.

Key capabilities

  • Multi-tenant compute sessions allocated on demand or in batches, isolating the underlying computing resources, with customizable job schedulers.
  • Sokovan, the in-house orchestrator that performs allocation and isolation.
  • Pluggable heterogeneous accelerators: CUDA GPU, ROCm GPU, Rebellions, FuriosaAI, HyperAccel, Intel Gaudi, Tenstorrent, Google TPU, GraphCore IPU and other NPUs.
  • REST and GraphQL APIs covering all platform functions, alongside the CLI and web UI.
  • WebSocket tunneling into individual compute sessions (kernels), so browsers and client CLIs reach in-container applications directly.
  • Choice of container engine: Docker 20.10+ with the nvidia runtime registered, or Podman 5.4.0+ with a CDI spec under /etc/cdi or /var/run/cdi.
  • Separately startable components: manager (mgr), agent (ag), storage proxy (storage), web server (web), account manager (am), and app proxy coordinator and worker.

Who uses it and how

  • Lab and research teams sharing accelerator hosts among many users, needing per-user container sessions rather than shared logins.
  • Platform and HPC operators standing up an internal PaaS on the manager, agent and storage proxy services and the REST and GraphQL APIs.
  • Groups with mixed accelerator fleets, since one deployment spans NVIDIA and AMD GPUs plus Intel Gaudi, Google TPU, GraphCore IPU and NPUs from Rebellions, FuriosaAI, HyperAccel and Tenstorrent.
  • Operators who want observability: Prometheus 3.x is required, with Grafana 11.x, Loki 3.x, Tempo 2.x and the OpenTelemetry Collector recommended through --profile observability.
  • Organizations needing SSO and a unified user profile, provided by the optional account manager service.

Getting started

Clone the repository and run ./scripts/install-dev.sh, which checks dependencies, builds a Python virtual environment with Pantsbuild, brings up halfstack infrastructure (PostgreSQL, Valkey, etcd, Apollo Router) via Docker Compose v2, with observability and object storage behind --profile observability and --profile storage, initializes database schemas and creates default API keypairs. The web UI then serves at http://localhost:8090.

How it compares

No paid products that this project replaces are listed in the facts, and no comparable tools are named either, so on this registry entry Backend.AI stands alone. What distinguishes it on the available facts is accelerator breadth combined with scheduling by its own orchestrator, Sokovan, rather than an outside scheduler.

When to use it — and when not to

A self-hoster operates real infrastructure: Docker Compose v2, PostgreSQL 16+, Valkey 9.1+, etcd 3.5+ and Prometheus 3.x at minimum, sudo access for installation, and up to seven service processes for the full feature set. Linux (Debian or RHEL based) or macOS is required, and 4+ CPU cores with 8 GB of RAM are recommended for development alone, so it suits neither anyone wanting a single binary nor a Windows host. The repository also carries 1617 open issues against 669 stars under LGPL-3.0.

project readme (upstream, from github) — read inline

Backend.AI

PyPI release version Supported Python versions Wheels Gitter

Backend.AI is a streamlined, container-based computing cluster platform that hosts popular computing/ML frameworks and diverse programming languages, with pluggable heterogeneous accelerator support including CUDA GPU, ROCm GPU, Rebellions, FuriosaAI, HyperAccel, Intel Gaudi, Tenstorrent, Google TPU, Graphcore IPU and other NPUs.

It allocates and isolates the underlying computing resources for multi-tenant computation sessions on-demand or in batches with customizable job schedulers with its own orchestrator named "Sokovan".

All its functions are exposed as REST and GraphQL APIs.

Requirements

Python & Build Tools

Infrastructure

Container engine (one of):

Engine Supported Accelerator requirements
Docker 20.10+ The nvidia runtime registered with the daemon (nvidia-ctk runtime configure --runtime=docker)
Podman 5.4.0+ A CDI spec for the accelerator under /etc/cdi or /var/run/cdi (nvidia-ctk cdi generate for NVIDIA GPUs)

Required:

  • Docker Compose v2
  • PostgreSQL 16+ (tested with 16.3)
  • Valkey 9.1+ (tested with 9.1.0; Redis-compatible)
  • etcd 3.5+ (tested with 3.5.14)
  • Prometheus 3.x (tested with 3.1.0)

Recommended (for observability):

  • Grafana 11.x (tested with 11.4.0)
  • Loki 3.x (tested with 3.5.0)
  • Tempo 2.x (tested with 2.7.2)
  • OpenTelemetry Collector

→ Detailed infrastructure setup: Infrastructure Documentation

System

  • OS: Linux (Debian/RHEL-based) or macOS
  • Permissions: sudo access for installation
  • Resources: 4+ CPU cores, 8GB+ RAM recommended for development

Getting Started

Quick Start (Development)

1. Clone and Install
git clone https://github.com/lablup/backend.ai.git
cd backend.ai
./scripts/install-dev.sh

This script will:

  • Check required dependencies (Docker, Python, etc.)
  • Set up Python virtual environment with Pantsbuild
  • Start halfstack infrastructure (PostgreSQL, Valkey, etcd, Apollo Router) — only required services by default; observability and object storage are opt-in via Compose profiles (--profile observability, --profile storage)
  • Initialize database schemas
  • Create default API keypairs and user accounts
2. Start Backend.AI Services

Start each component in separate terminals:

Manager (Terminal 1):

./backend.ai mgr start-server --debug

Agent (Terminal 2):

./backend.ai ag start-server --debug

Storage Proxy (Terminal 3):

./backend.ai storage start-server --debug

Web Server (Terminal 4):

./backend.ai web start-server --debug

Account Manager (Terminal 5, optional for SSO and unified user profile):

./backend.ai am start-server --debug

App Proxy (Terminal 6-7, optional for in-container service access):

./backend.ai app-proxy-coordinator start-server --debug
./backend.ai app-proxy-worker start-server --debug
3. Run Your First Session

Set up client environment:

source env-local-user-session.sh
# This script prints your default User ID and Password;
./backend.ai login
# When prompted, enter the User ID and Password shown above.

Run a simple Python session:

./backend.ai run python -c "print('Hello Backend.AI!')"

Or access Web UI at http://localhost:8090 with credentials from env-local-*.sh files.

Accessing Compute Sessions (aka Kernels)

Backend.AI provides websocket tunneling into individual computation sessions (containers), so that users can use their browsers and client CLI to access in-container applications directly in a secure way.

  • Jupyter: data scientists' favorite tool
    • Most container images have intrinsic Jupyter and JupyterLab support.
  • Web-based terminal
    • All container sessions have intrinsic ttyd support.
  • SSH
    • All container sessions have intrinsic SSH/SFTP/SCP support with auto-generated per-user SSH keypair. PyCharm and other IDEs can use on-demand sessions using SSH remote interpreters.
  • VSCode
    • Most container sessions have intrinsic web-based VSCode support.

Working with Storage

Backend.AI provides an abstraction layer on top of existing network-based storages (e.g., NFS/SMB), called vfolders (virtual folders). Each vfolder works like a cloud storage that can be mounted into any computation sessions and shared between users and user groups with differentiated privileges.

Installation for Multi-node Tests & Production

Please consult our documentation for community-supported materials. Contact the sales team ([email protected]) for professional paid support and deployment options.

Architecture

For comprehensive system architecture, component interactions, and infrastructure details, see:

Component Architecture Documentation

This document covers:

  • System architecture diagrams and component flow
  • Port numbers and infrastructure setup
  • Component dependencies and communication protocols
  • Development and production environment configuration

Contents in This Repository

This repository contains all open-source server-side components and the client SDK for Python as a reference implementation of API clients.

Directory Structure

  • src/ai/backend/: Source codes
    • manager/: Manager as the cluster control-plane
    • manager/api: Manager API handlers
    • account_manager/: Unified user profile and SSO management
    • agent/: Agent as per-node controller
    • agent/docker/: Agent's Docker backend
    • agent/kubernetes/: Agent's Kubernetes backend
    • agent/dummy/: Agent's dummy backend
    • kernel/: Agent's kernel runner counterpart
    • runner/: Agent's in-kernel prebuilt binaries
    • helpers/: Agent's in-kernel helper package
    • common/: Shared utilities
    • client/: Client SDK
    • cli/: Unified CLI for all components
    • install/: SCIE-based TUI installer
    • storage/: Storage proxy for offloading storage operations
    • storage/api: Storage proxy's manager-facing and client-facing APIs
    • appproxy/: App proxy for accessing container apps from outside
    • appproxy/coordinator: App proxy coordinator who provisions routing circuits
    • appproxy/worker: App proxy worker who forwards the traffic
    • web/: Web UI server
      • static/: Backend.AI WebUI release artifacts
    • logging/: Logging subsystem
    • plugin/: Plugin subsystem
    • test/: Integration test suite
    • testutils/: Shared utilities used by unit tests
    • meta/: Legacy meta package
    • accelerator/: Intrinsic accelerator plugins
  • docs/: Unified documentation
  • tests/
    • manager/, agent/, ...: Per-component unit tests
  • configs/
    • manager/, agent/, ...: Per-component sample configurations
  • docker/: Dockerfiles for auxiliary containers
  • fixtures/
    • manager/, ...: Per-component fixtures for development setup and tests
  • plugins/: A directory to place plugins such as accelerators, monitors, etc.
  • scripts/: Scripts to assist development workflows
    • install-dev.sh: The single-node development setup script from the working copy
  • stubs/: Type annotation stub packages written by us
  • tools/: A directory to host Pants-related tooling
  • dist/: A directory to put build artifacts (.whl files) and Pants-exported virtualenvs
  • changes/: News fragments for towncrier
  • pants.toml: The Pants configuration
  • pyproject.toml: Tooling configuration (towncrier, pytest, mypy)
  • BUILD: The root build config file
  • **/BUILD: Per-directory build config files
  • BUILD_ROOT: An indicator to mark the build root directory for Pants
  • AGENTS.md: The steering guide for agent-assisted development (CLAUDE.md is a symlink to it)
  • requirements.txt: The unified requirements file
  • *.lock, tools/*.lock: The dependency lock files
  • docker-compose.*.yml: Per-version recommended halfstack container configs
  • README.md: This file
  • MIGRATION.md: The migration guide for updating between major releases
  • VERSION: The unified version declaration

Server-side components are licensed under LGPLv3 to promote non-proprietary open innovation in the open-source community while other shared libraries and client SDKs are distributed under the MIT license.

There is no obligation to open your service/system codes

readme truncated — read the full docs on github

Frequently asked questions

Is backend.ai free to use?

backend.ai is open source under the LGPL-3.0 licence. There is no licence fee and no seat count — you can self-host it or, where the project offers one, pay a vendor for a managed version instead.

What does backend.ai do?

Backend.AI is a streamlined, container-based computing cluster platform that hosts popular computing/ML frameworks and diverse programming languages, with plugg

What is backend.ai written in?

backend.ai is primarily written in Python. Its source is publicly available at https://github.com/lablup/backend.ai, and it has 669 GitHub stars.