[opensourcedex]

ome is a free, open source machine learning infrastructure project written in Go and released under Apache-2.0. It has 509 GitHub stars, 97 forks and 134 open issues, and was last pushed 5 hours ago. On this registry it ranks #54 of 57 tracked projects in Machine Learning Infrastructure, with 5 head-to-head comparisons available. It gained 1 stars over the last 3 tracked days.

What is ome?

OME (Open Model Engine) is an Apache-2.0 Kubernetes operator, written in Go, that manages model lifecycle, GPU scheduling, and LLM serving for platform teams running inference on Kubernetes.

What it is

OME is a Kubernetes operator for the management and serving of large language models. It treats models as first-class custom resources instead of configuration scattered across individual deployments, and its parser extracts architecture, parameter count, and capabilities directly from model files. Model storage is distributed, with automated repair, double encryption, and namespace scoping, and it accepts SafeTensors, PyTorch, TensorRT, and ONNX formats. A pre-configured catalogue in config/models/SUPPORTED_MODELS.md covers more than 200 models, including the Llama, Qwen, DeepSeek, Gemma, and Phi families. The operator exposes its API at version v1beta1.

The concrete problem OME solves is the gap between a raw inference engine and a working service. Engines such as SGLang, vLLM, TensorRT-LLM, and Triton can serve a model, but they do not choose which runtime suits a given model, pack GPUs across a cluster, or coordinate multi-pod rollouts. OME sits in the Kubernetes ecosystem above those engines and replaces the hand-written per-model manifests, manual runtime matching, and ad hoc scheduling that platform teams otherwise maintain themselves. Topics attached to the project — model-serving, model-as-a-service, multi-node-kubernetes, pd-disaggregation — describe that role.

Key capabilities

  • Models are first-class custom resources, with parsing that derives architecture, parameter count, and capabilities from the model files themselves.
  • Intelligent runtime selection scores candidate runtimes by weighted criteria covering architecture, format, quantization, parameter size, and framework compatibility.
  • Deployment patterns include prefill-decode disaggregation, multi-node inference, and standard Kubernetes deployments, with canary and blue-green rollout strategies.
  • GPU bin-packing scheduling re-optimizes dynamically, and AcceleratorClass resources carry discovery patterns and cost data for policies such as BestFit, Cheapest, and MostCapable.
  • First-class SGLang support covers cache-aware load balancing, multi-node deployment, prefill-decode disaggregated serving, and multi-LoRA adapter serving; vLLM is supported for high-throughput inference.
  • Kubernetes integration spans Kueue for gang scheduling, LeaderWorkerSet for resilient multi-node deployments, KEDA for custom-metrics autoscaling, K8s Gateway API for traffic routing, and the Gateway API Inference Extension for standardized inference endpoints.
  • The BenchmarkJob custom resource runs configurable traffic patterns and concurrent load tests, storing results for comparison across models and service configurations.
  • A separate web console, ome-projects/ome-console, manages models, serving runtimes, and inference services with live updates and HuggingFace model search.

Who uses it and how

  • Platform and ML infrastructure teams operating multi-node Kubernetes clusters, where multi-node-kubernetes and model-as-a-service are the working assumptions rather than an afterthought.
  • Teams serving large models from the DeepSeek, Kimi-K2, Llama, and Qwen families that need prefill-decode disaggregation or multi-node inference to fit within a cluster.
  • Cluster operators who need gang scheduling through Kueue and custom-metrics autoscaling through KEDA for workloads that span many pods.
  • Cost-sensitive environments that encode GPU cost information in AcceleratorClass resources and select accelerators with Cheapest or BestFit policies.
  • Groups that benchmark candidate configurations with BenchmarkJob before promoting them through canary or blue-green rollouts.

Getting started

The README directs new users to the documentation at http://ome-projects.github.io/ome/, which covers installation and capabilities for the v1beta1 API. The excerpt provided does not name a Helm chart, package, or image, so the exact install command should be taken from that documentation.

How it compares

Where comparable Kubernetes operators for LLM serving are concerned, the available facts name none, so OME stands alone in this registry on that axis. Its relationship to neighbouring tools is vertical rather than competitive: it orchestrates SGLang, vLLM, TensorRT-LLM, and Triton instead of replacing them, adding scheduling, runtime selection, and lifecycle management above the engines.

When to use it — and when not to

Pick OME when the cluster is already Kubernetes and the workload needs GPU scheduling, runtime selection, or multi-node patterns; a self-hoster must operate not only the operator but also the surrounding components its integrations assume, such as Kueue, LeaderWorkerSet, and KEDA, if those features are used. Do not pick it for single-machine or non-Kubernetes inference, where the operational weight is not repaid. Weigh the project's maturity: the API is still v1beta1, the console lives in a separate repository, and the repository carries 134 open issues against 508 stars, so the documentation and issue tracker are worth reading before a production commitment.

project readme (upstream, from github) — read inline

OME (Open Model Engine) — Kubernetes Operator for LLM Serving

codecov Latest Release API Reference License Ask DeepWiki

What is OME?

OME (Open Model Engine) is a Kubernetes operator for enterprise-grade management and serving of Large Language Models (LLMs). It optimizes the deployment and operation of LLMs by automating model management, intelligent runtime selection, efficient resource utilization, and sophisticated deployment patterns.

Read the documentation to learn more about OME capabilities and features.

Features Overview

  • Model Management: Models are first-class citizen custom resources in OME. Sophisticated model parsing extracts architecture, parameter count, and capabilities directly from model files. Supports distributed storage with automated repair, double encryption, namespace scoping, and multiple formats (SafeTensors, PyTorch, TensorRT, ONNX). See the supported models reference for the comprehensive catalog of 200+ pre-configured models, including the Llama, Qwen, DeepSeek, Gemma, and Phi families.

  • Intelligent Runtime Selection: Automatic matching of models to optimal runtime configurations through weighted scoring based on architecture, format, quantization, parameter size, and framework compatibility.

  • Optimized Deployments: Supports multiple deployment patterns including prefill-decode disaggregation, multi-node inference, and traditional Kubernetes deployments, with canary and blue-green rollout strategies and advanced scaling controls.

  • Resource Optimization: Specialized GPU bin-packing scheduling with dynamic re-optimization to maximize cluster efficiency while ensuring high availability.

  • Runtime Integrations: First-class support for SGLang - the most advanced inference engine with cache-aware load balancing, multi-node deployment, prefill-decode disaggregated serving, multi-LoRA adapter serving, and much more. Also supports vLLM for high-throughput inference.

  • Accelerator Management: Hardware-aware scheduling through AcceleratorClass resources that define GPU capabilities, discovery patterns, and cost information. Enables intelligent accelerator selection with policies like BestFit, Cheapest, or MostCapable.

  • Web Console: Modern web interface for managing models, serving runtimes, and inference services with real-time updates and HuggingFace model search integration. Developed separately in ome-projects/ome-console.

  • Kubernetes Ecosystem Integration: Deep integration with modern Kubernetes components including Kueue for gang scheduling of multi-pod workloads, LeaderWorkerSet for resilient multi-node deployments, KEDA for advanced custom metrics-based autoscaling, K8s Gateway API for sophisticated traffic routing, and Gateway API Inference Extension for standardized inference endpoints.

  • Automated Benchmarking: Built-in performance evaluation through the BenchmarkJob custom resource, supporting configurable traffic patterns, concurrent load testing, and comprehensive result storage. Enables systematic performance comparison across models and service configurations.

Production Readiness Status

  • ✅ API version: v1beta1
  • ✅ Comprehensive documentation
  • ✅ Unit and integration test coverage
  • ✅ Production deployments with large-scale LLM workloads
  • ✅ Monitoring via standard metrics and Kubernetes events
  • ✅ Security: RBAC-based access control and model encryption
  • ✅ High availability mode with redundant model storage

Installation

Requires Kubernetes 1.28 or newer

Option 1: OCI Registry (Recommended)

Install OME directly from the OCI registry:

# Install OME CRDs
helm upgrade --install ome-crd oci://ghcr.io/moirai-internal/charts/ome-crd --namespace ome --create-namespace

# Install OME resources
helm upgrade --install ome oci://ghcr.io/moirai-internal/charts/ome-resources --namespace ome

Option 2: Install from Source

For development or customization:

# Clone the repository
git clone https://github.com/ome-projects/ome.git
cd ome

# Install from local charts
helm install ome-crd charts/ome-crd --namespace ome --create-namespace
helm install ome charts/ome-resources --namespace ome

Optional: Serving Resources

The ome-serving chart deploys a set of pre-configured ClusterBaseModels, ClusterServingRuntimes, and InferenceServices on top of the core installation:

helm upgrade --install ome-serving oci://ghcr.io/moirai-internal/charts/ome-serving --namespace ome

Read the installation guide for more options and advanced configurations.

Learn more about:

Architecture

OME uses a component-based architecture built on Kubernetes custom resources:

  • BaseModel/ClusterBaseModel: Define model sources and metadata with automatic parsing of architecture, parameters, and capabilities
  • FineTunedWeight: Define LoRA adapters and fine-tuned weights that extend base models
  • ServingRuntime/ClusterServingRuntime: Define how models are served with runtime-specific configurations
  • InferenceService: Connects models to runtimes for deployment with support for prefill-decode disaggregation and multi-node inference
  • AcceleratorClass: Define GPU hardware classes with capabilities, discovery patterns, and cost information for intelligent scheduling
  • BenchmarkJob: Measures model performance under different workloads with configurable traffic patterns

OME's controller automatically:

  1. Downloads and parses models to understand their characteristics
  2. Selects the optimal runtime configuration for each model
  3. Matches models to appropriate accelerators based on requirements
  4. Generates Kubernetes resources for efficient deployment
  5. Continuously optimizes resource utilization across the cluster

Roadmap

High-level overview of the main priorities:

  • Unified multi-cluster workload management (APIs merged; reconciliation in development)
  • Multi-cloud model storage and authentication
  • PVC-backed model storage
  • KV cache pooling
  • Model Context Protocol (MCP) gateway support
  • Accelerator-aware runtime selection for heterogeneous GPU clusters
  • A kubectl plugin for managing OME resources from the command line

Community and Support

License

OME is licensed under the Apache License 2.0.

Frequently asked questions

Is ome free to use?

ome is open source under the Apache-2.0 licence. There is no licence fee and no seat count — you can self-host it or, where the project offers one, pay a vendor for a managed version instead.

What does ome do?

Open Model Engine (OME) — Kubernetes operator for LLM serving, GPU scheduling, and model lifecycle management. Works with SGLang, vLLM, TensorRT-LLM, and Triton

What is ome written in?

ome is primarily written in Go. Its source is publicly available at https://github.com/ome-projects/ome, and it has 509 GitHub stars.