OpenLLM is an Apache-2.0 Python project from BentoML that runs open-source or custom large language models as OpenAI-compatible API endpoints on self-hosted infrastructure, and it is aimed at developers and platform teams who want to serve Llama, Qwen, Mistral, DeepSeek and similar models in the cloud without rewriting their client code.

What it is

OpenLLM lets developers run any open-source LLM — Llama 3.3, Qwen2.5, Phi3, DeepSeek R1 and others — or a custom model as an OpenAI-compatible API with a single command. It ships a built-in chat UI, inference backends, and a simplified workflow for enterprise-grade cloud deployment with Docker, Kubernetes and BentoCloud. It lives in the Python packaging ecosystem on PyPI as openllm, sits inside the BentoML project family, and is licensed under Apache-2.0.

The concrete problem it solves is the endpoint. An application written against the OpenAI API needs a compatible server at a new base URL when it moves onto self-hosted models, and OpenLLM supplies one — by default on port 3000 with /v1 paths — so existing OpenAI clients keep working after a base URL change, with the API key optional. Its stated positioning is the cloud-deployment step beyond Ollama-style local runners, which is the framing of BentoML's write-up titled "From Ollama to OpenLLM: Running LLMs in the Cloud" linked from the README.

Key capabilities

• Starts a model server with one command using model:version specs such as openllm serve llama3.3:70b, openllm serve deepseek:r1-671b and openllm serve qwen2.5-coder:3b.
• Serves OpenAI-compatible APIs, so the OpenAI Python client can point at base_url='http://localhost:3000/v1' and run streaming chat completions with no client changes.
• Provides a built-in chat UI alongside the HTTP API.
• Covers a wide catalogue including deepseek r1-671b, gemma2, gemma3, jamba1.5, llama3.1, llama3.2, llama3.3, llama4, mistral, mistral-large, phi4, pixtral, qwen2.5, qwen2.5-coder and qwq, with the full list in the OpenLLM models repository.
• Supports custom models through a model repository, so private checkpoints are served the same way as public ones.
• Documents required GPU memory per model, from 12G for gemma3:3b up to 80Gx16 for deepseek r1-671b.
• Targets Docker, Kubernetes and BentoCloud for deployment, with an openllm hello command for first-run exploration.

Who uses it and how

• Platform teams planning capacity for large models, where per-model GPU requirements such as 80Gx2 for llama3.3:70b and 80Gx8 for llama4:17b16e drive node selection.
• Application developers replacing third-party API calls by repointing an OpenAI-compatible client at a self-hosted base URL, with client authentication optional.
• Teams running compact models on single GPUs, for example llama3.2:1b at 24G or gemma2:2b at 12G.
• Multimodal workloads, indicated by the llama3-2-vision topic and the pixtral 12b-2409 entry in the model table.
• Operations groups deploying through Kubernetes or BentoCloud rather than hand-managed servers.

Getting started

Install with pip install openllm (or pip3 install openllm), run openllm hello for an interactive tour, and start a server with openllm serve llama3.2:1b. For gated models, set export HF_TOKEN= with a Hugging Face token, because OpenLLM does not store model weights.

How it compares

The README frames OpenLLM as the cloud-oriented step beyond Ollama, through BentoML's write-up titled "From Ollama to OpenLLM: Running LLMs in the Cloud", where the emphasis is OpenAI-compatible serving plus Docker, Kubernetes and BentoCloud deployment rather than desktop convenience. The facts provided name no paid product that OpenLLM is stated to replace and no other comparable tool, so on this registry it stands largely alone.

When to use it — and when not to

Self-hosting means operating the GPU capacity the model table demands, managing a Hugging Face token for gated weights, and running the container or Kubernetes stack, since OpenLLM stores no weights and pulls them from Hugging Face on first run. The catalogue skews large, with several entries needing multiple 80G-class GPUs, so teams without that hardware or cluster access should look elsewhere. Licence and packaging are clear (Apache-2.0 on PyPI) and the repository shows recent activity, but the README excerpt is chiefly usage documentation, so adopters should check the repository itself for release cadence and support expectations.

pip install openllm  # or pip3 install openllm
openllm hello

openllm serve llama3.2:1b

from openai import OpenAI

client = OpenAI(base_url='http://localhost:3000/v1', api_key='na')

# Use the following func to get the available models
# model_list = client.models.list()
# print(model_list)

chat_completion = client.chat.completions.create(
    model="meta-llama/Llama-3.2-1B-Instruct",
    messages=[
        {
            "role": "user",
            "content": "Explain superconductors like I'm five years old"
        }
    ],
    stream=True,
)
for chunk in chat_completion:
    print(chunk.choices[0].delta.content or "", end="")

from llama_index.llms.openai import OpenAI

llm = OpenAI(api_bese="http://localhost:3000/v1", model="meta-llama/Llama-3.2-1B-Instruct", api_key="dummy")
...

openllm run llama3:8b

openllm model list

openllm repo update

openllm model get llama3.2:1b

openllm repo add <repo-name> <repo-url>

openllm deploy llama3.2:1b --env HF_TOKEN