vLLM-Omni is an Apache-2.0 Python framework from the vLLM project that extends vLLM's text-only inference engine to serve omni-modality models — text, image, audio, video, and action — for teams that need to run diffusion transformers, autoregressive models, and realtime duplex pipelines from a single serving stack.

What it is

vLLM-Omni lives in the vLLM ecosystem, the widely adopted serving engine for large language models. Upstream vLLM was originally built for text-based autoregressive generation tasks. vLLM-Omni extends that engine with omni-modality model inference and serving: text, image, audio, video, and action data processing, plus support for non-autoregressive architectures such as Diffusion Transformers (DiT) and other parallel generation models, and heterogeneous outputs that reach beyond text to multimodal and action results.

The concrete problem it solves is that a text-only autoregressive server cannot express the stage graphs omni-modality models require. Serving a joint video-and-audio model, a text-to-speech model, or a realtime speech-to-speech agent means chaining several models with different execution models — autoregressive decoding on one stage, iterative diffusion denoising on another — and moving tensors between them. vLLM-Omni replaces ad-hoc per-model serving scripts with a pipelined, heterogeneously-staged runtime that reuses vLLM's KV cache management and adds a unified AR/DiT paged KV cache runtime, so one deployment can host the whole pipeline behind a single API.

Key capabilities

• Omni-modality processing across text, image, audio, video, and action data, with heterogeneous output types rather than text alone.
• A unified AR/DiT paged KV cache runtime, introduced in release 0.28.0, so autoregressive and diffusion stages share one cache design.
• Full-duplex realtime serving with streaming audio input and output, documented at docs/serving/realtime_duplex_api.md.
• An OpenAI-compatible API server, so existing clients point at the deployment without a custom SDK.
• A heterogeneous pipeline abstraction that manages complex multi-model workflows, with pipelined stage execution overlapping for throughput and streaming outputs.
• Disaggregation built on OmniConnector, with dynamic resource allocation across stages.
• Tensor, pipeline, data, and expert parallelism for distributed inference, with broad CUDA, ROCm, MUSA, NPU, and XPU support.

Who uses it and how

• Teams serving joint video and audio generation in production, following the MiniMax H3 recipe at recipes/MiniMaxAI/MiniMax-H3.md on both GPU and NPU hardware.
• Voice and conversational AI builders running the MiniCPM-o series (recipes/OpenBMB/MiniCPM-o-4_5.md) for full-duplex realtime sessions with streaming audio in and out.
• Reinforcement learning groups training diffusion models through the VeRL-Omni integration at v0.2.0, which uses request-level and step-wise batching with FA3 and supports DPO and GSPO on Qwen3-Omni.
• World-model and robotics teams serving generative world models and policies, including NVIDIA Cosmos3 (recipes/cosmos3/Cosmos3-Nano.md) and DreamZero.
• Speech and TTS teams deploying models such as MOSS-TTS and Lance alongside image and video generation pipelines.

Getting started

The README points to the project documentation at https://docs.vllm.ai/projects/vllm-omni, and per-model deployment recipes live in the recipes/ directory of the repository, for example recipes/MiniMaxAI/MiniMax-H3.md and recipes/OpenBMB/MiniCPM-o-4_5.md. Stable releases are tagged after every even-numbered upstream vLLM minor version, with 0.28.0 the latest release called out in the README.

How it compares

vLLM-Omni is not a replacement for upstream vLLM but a sibling project from the same organization: upstream vLLM remains the engine for text-only autoregressive serving, while vLLM-Omni adds the non-autoregressive and multimodal stages on top of it. VeRL-Omni sits on the other side of that boundary, consuming vLLM-Omni as the rollout and inference backend for diffusion reinforcement learning rather than competing with it as a server.

When to use it — and when not to

Self-hosters must operate GPU, NPU, or other accelerator hardware and manage distributed execution settings such as tensor, pipeline, data, and expert parallelism, along with the KV cache and quantization configuration the runtime exposes. Teams that only need text generation should stay on upstream vLLM, because vLLM-Omni's added complexity buys nothing for a pure autoregressive text workload. A prospective adopter should also weigh the project's maturity signals: 1,977 open issues against 6,827 stars, and a README excerpt that covers releases and capabilities but includes no quickstart install command, so the documentation site is the practical entry point.

@article{yin2026vllmomni,
  title={vLLM-Omni: Fully Disaggregated Serving for Any-to-Any Multimodal Models},
  author={Peiqi Yin, Jiangyun Zhu, Han Gao, Chenguang Zheng, Yongxiang Huang, Taichang Zhou, Ruirui Yang, Weizhi Liu, Weiqing Chen, Canlin Guo, Didan Deng, Zifeng Mo, Cong Wang, James Cheng, Roger Wang, Hongsheng Liu},
  journal={arXiv preprint arXiv:2602.02204},
  year={2026}
}