ServerlessLLM is an Apache-2.0, Python-based system for running many large language models on shared GPUs, aimed at teams that want serverless-style multi-model serving without provisioning a separate GPU for every model.

What it is

ServerlessLLM is an open-source LLM serving stack built around a custom checkpoint format and a storage-aware scheduler. It lives in the PyTorch and HuggingFace Transformers ecosystem, shares GPUs across models, and exposes an OpenAI-compatible HTTP API. The project ships as two pieces: a cluster (head node plus GPU worker) and ServerlessLLM Store, a standalone fast loader usable inside existing Python code. It was published at OSDI'24 and is developed under the ServerlessLLM organisation.

The concrete problem it solves is checkpoint load latency. Standard loaders, including the SafeTensors checkpoint loader, are slow enough that a model must stay resident once loaded, which forces one GPU per model. ServerlessLLM replaces that loader with a custom binary format optimised for sequential reads, O_DIRECT I/O that bypasses the OS page cache, a pinned memory pool for DMA-accelerated GPU transfers, and parallel multi-threaded loading. Reported results on NVIDIA H100 with NVMe SSD show Qwen/Qwen3-32B loading in 3.2s versus 20.6s, and Llama-3.1-8B-Instruct in 0.7s versus 4.4s. Faster loading is what makes scale-to-zero and rapid model switching practical.

Key capabilities

• Ultra-fast checkpoint loading with the custom binary format and O_DIRECT I/O, reported at 6-10x faster than the SafeTensors loader.
• ServerlessLLM Store as a standalone package, pip install serverless-llm-store, with sllm-store save to convert a model and sllm-store start --storage-path ~/models --mem-pool-size 4GB to run the store server.
• Python API via from sllm_store.transformers import load_model, which returns a normal PyTorch/Transformers model callable with model.generate.
• GPU multiplexing: run 10 or more models on one GPU with fast switching, storage-aware scheduling, and auto-scaling per model down to zero when idle.
• Live migration for zero-downtime resource optimisation during deployment.
• Unified inference and fine-tuning: serve a base model alongside hundreds of LoRA adapters, with serverless LoRA fine-tuning and on-demand adapter deployment.
• OpenAI-compatible endpoint at http://127.0.0.1:8343/v1/chat/completions, plus support for NVIDIA and AMD GPUs (a separate ROCm guide), vLLM, Transformers, and custom models.

Who uses it and how

• Teams serving many models behind one interface, where the README's "Random" scenario simulates serverless multi-model serving across a shared GPU pool.
• Deployments that must scale idle models to zero to cut GPU cost, using per-model auto-scaling and live migration instead of fixed allocations.
• Fine-tuning workflows that mix a base model with hundreds of LoRA adapters on the same hardware, deploying adapters for inference on demand.
• Existing PyTorch users who only want the loader, running sllm-store start and switching from_pretrained to load_model with no cluster involved.
• Cluster operators running the head node plus a GPU worker described in the compose example, with models held on a local MODEL_FOLDER.

Getting started

The quick start downloads examples/docker/docker-compose.yml, sets MODEL_FOLDER, runs docker compose up -d, then deploys with sllm deploy --model Qwen/Qwen3-0.6B --backend transformers. For a Docker-free path, install serverless-llm-store and use the Python loader directly.

How it compares

Against SafeTensors, whose loader it benchmarks against and replaces, ServerlessLLM trades a converted on-disk format for materially faster cold loads. It sits alongside vLLM and Transformers rather than excluding them, working with both as backends while adding storage-aware scheduling and multiplexing. Where it differs from a plain inference server is the assumption that models will be loaded and evicted repeatedly, not pinned permanently.

When to use it — and when not to

Self-hosting the full cluster means operating Docker, a head node, at least one GPU worker, and local model storage; the published benchmarks also assume NVMe SSD and H100-class hardware, so results on slower disks will be smaller. Anyone serving a single model continuously on a dedicated GPU gains little, since the scheduling and load-eviction machinery adds components without removing a bottleneck. Be aware that the project carries 45 open issues, and its store path works best when models are pre-converted with sllm-store save.

# Download the docker-compose.yml file
curl -O https://raw.githubusercontent.com/ServerlessLLM/ServerlessLLM/main/examples/docker/docker-compose.yml

# Set model storage location
export MODEL_FOLDER=/path/to/models

# Launch cluster (head node + worker with GPU)
docker compose up -d

# Wait for the cluster to be ready
docker logs -f sllm_head

docker exec sllm_head /opt/conda/envs/head/bin/sllm deploy --model Qwen/Qwen3-0.6B --backend transformers

curl http://127.0.0.1:8343/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-0.6B",
    "messages": [{"role": "user", "content": "What is ServerlessLLM?"}],
    "temperature": 0.7
  }'

pip install serverless-llm-store

sllm-store save --model Qwen/Qwen3-0.6B --backend transformers

# Start the store server first
sllm-store start --storage-path ~/models --mem-pool-size 4GB

from sllm_store.transformers import load_model

# Load model (6-10x faster than from_pretrained!)
model = load_model(
    "Qwen/Qwen3-0.6B",
    device_map="auto",
    torch_dtype="float16"
)

# Use as a normal PyTorch/Transformers model
output = model.generate(**inputs)

@inproceedings{fu2024serverlessllm,
  title={ServerlessLLM: Low-Latency Serverless Inference for Large Language Models},
  author={Fu, Yao and Xue, Leyang and Huang, Yeqi and Brabete, Andrei-Octavian and Ustiugov, Dmitrii and Patel, Yuvraj and Mai, Luo},
  booktitle={OSDI'24},
  year={2024}
}