SGLang-Omni is an Apache-2.0 Python serving framework that runs audio models (text-to-speech, automatic speech recognition, music generation) and unified multimodal models behind OpenAI-compatible endpoints, aimed at teams deploying those models on their own GPU infrastructure.

What it is

SGLang-Omni is a multi-stage serving runtime for omni, speech, and TTS models. Its design target is multi-stage decoding: generation split across heterogeneous stages with different compute patterns, dependency structures, and resource needs. The project owns the pipeline topology, stage lifecycle, inter-stage transport, the model-family integration layer, and an OpenAI-compatible serving surface, and it composes with SGLang for high-performance autoregressive scheduling and model execution where applicable. The runtime models generation as coordinated stages — preprocessing, encoders, autoregressive engines, talkers, decoders, vocoders, and aggregators — with each stage running behind a scheduler matched to its workload. A control plane coordinates requests while a relay data plane moves tensor payloads across shared-memory, NCCL, NIXL, and Mooncake backends. The repository sits in the AI and machine learning infrastructure category, is written in Python, and carries 1205 stars, 492 forks, and topics including asr, audio-generation, cuda, distributed-inference, model-serving, multimodal, music-generation, openai-api, and pytorch.

The concrete problem it solves is the absence of one runtime that can serve speech, music, and omni-modal models together. Speech pipelines are typically assembled from per-model servers: a TTS process, a separate ASR process, a vocoder step, and a hand-written gateway that fakes an OpenAI-compatible API on top. Those stacks duplicate transport code, cannot share reference encoders or vocoder scheduling, and treat each model family as an isolated deployment. SGLang-Omni replaces that arrangement inside the SGLang ecosystem by providing the shared pipeline state, stage construction, inter-stage transport, and API surface once, so individual model families plug into the same runtime rather than shipping their own servers.

Key capabilities

• OpenAI-compatible API surface exposing multimodal chat, speech generation, batch speech, streaming speech, uploaded voices, and transcription.
• Text-to-speech on /v1/audio/speech and audio editing on /generate, including AuK and AuK-Flash support for text plus voice instructions to 24 kHz speech, and audio plus editing instructions to edited speech.
• Music generation with MiniMax Music 3, taking lyrics plus a caption and producing a 32 kHz stereo song on /v1/audio/speech.
• Native-streaming speech through MOSS-TTS Local Transformer v1.5 at 48 kHz, and real-time controllable speech through Higgs Audio v3.
• Broad model-family integration: Qwen3-Omni, Ming-Omni, MOSS-TTS, Fish Speech S2-Pro, Qwen3-TTS, Voxtral TTS, and Ming-Omni-TTS.
• Transport-aware execution with a control plane for requests and a relay data plane over shared-memory, NCCL, NIXL, and Mooncake.
• Distribution as the sglang-omni package on PyPI, currently version 0.1.5, built on PyTorch and CUDA.

Who uses it and how

• Teams that already speak the OpenAI API and want speech and omni-modal models reachable through the same endpoints their text models use, without writing a custom gateway.
• Voice-agent and real-time speech deployments that need streaming synthesis, where MOSS-TTS Local v1.5 native-streaming 48 kHz output or Higgs Audio v3 real-time control fit the latency profile.
• Music and audio generation pipelines that take lyrics plus captions as input and return stereo songs, using the MiniMax Music 3 path.
• Multimodal chat services that accept audio and images and return text or audio, served through the Qwen3-Omni and Ming-Omni cookbooks.
• Distributed inference on CUDA GPU fleets, where stages with different resource profiles are spread across machines and connected with NCCL, NIXL, or Mooncake transport.

Getting started

Install the published release from PyPI with uv pip install --prerelease=allow "sglang-omni==0.1.5". The homepage links a Quick Start, Installation, and Cookbook documentation set, with per-model cookbook pages for each supported family.

How it compares

The facts provide no list of paid products that SGLang-Omni replaces, so no licence, cost, or data-ownership contrast can be drawn here. It sits alongside SGLang: SGLang-Omni owns pipeline topology, stage lifecycle, inter-stage transport, and model-family integration, and composes with SGLang for high-performance autoregressive scheduling and model execution where applicable. No directly comparable tool is named in this registry, so it stands alone as a multi-stage serving runtime for omni and speech models.

When to use it — and when not to

A self-hoster must operate a CUDA GPU environment and the distributed transport backends the runtime relies on, including shared-memory, NCCL, NIXL, and Mooncake, because the design assumes stages spread across heterogeneous resources rather than a single process. The install line requires --prerelease=allow, so the released package is treated as a pre-release by uv, and the repository carries 644 open issues. The README excerpts give no storage, database, or mail requirements, so the non-GPU operational surface is not documented in the material available, and anyone needing a CPU-only or single-model HTTP wrapper should look elsewhere.