PegaInfer is a pure Rust and CUDA large language model inference engine that serves models ranging from Qwen3 to Kimi-K2 behind an OpenAI-compatible API, with no PyTorch or Python runtime in the default serving path.

What it is

PegaInfer is an inference engine written in Rust that runs large language models directly on NVIDIA GPUs through hand-written CUDA kernels. Each model owns its own scheduler, state, and kernels, while the serving layer and KV cache infrastructure are shared across models. The engine ships as a server binary named pegainfer (with pegainfer-server as the server entrypoint, and model crates holding model implementations and diagnostics), and it exposes an OpenAI-compatible HTTP API, so existing clients can point at it without protocol changes.

The concrete problem it addresses is the Python and PyTorch runtime layer that usually sits between a model checkpoint and a GPU. PegaInfer removes that layer for the default Qwen3 build: the release bundles CUDA 13 and cuBLAS, and the compiled binary serves the model with no Python interpreter involved. It replaces the common pattern of a Python inference stack — the same role that vLLM occupies, which PegaInfer benchmarks against directly. The ecosystem it lives in is GPU-accelerated LLM serving for Rust and CUDA, distributed under Apache-2.0.

Key capabilities

• Serves multiple model families from one engine: Qwen3, Qwen3.5, Gemma 4 26B-A4B, GLM-5.2, DeepSeek, and Kimi-K2, with K3 kernels generated through TileLang.
• Exposes an OpenAI-compatible API, with the server listening on port 8000 by default.
• Ships a prebuilt binary installed by install.sh, bundling CUDA 13 and cuBLAS for the Qwen3-only release; PEGAINFER_VERSION pins an exact release instead of the latest.
• Runs without a Python runtime on the default Qwen3 build; the model crates separate implementation from diagnostics.
• Supports model-specific kernel compilation paths: Qwen3.5 uses Triton AOT kernels, and K3 uses TileLang kernel generation.
• Configures builds through environment variables including CUDA_HOME, PEGAINFER_CUDA_SM for explicit GPU architecture targets, PEGAINFER_TRITON_PYTHON, and PEGAINFER_TILELANG_PYTHON.
• Supports distributed serving topologies, including co-located EP4 across 4 GPUs and disaggregated TP4 prefill plus EP4 decode across 8 GPUs total for GLM-5.2.

Who uses it and how

• Single-GPU deployments: Qwen3-4B served in BF16 with TP1 on one RTX 5090, benchmarked against vLLM 0.24.0.
• Concurrency-driven capacity planning: Qwen3.5 at 8B, 2B, 9B, and 27B measured in a GH200 concurrency sweep using random 1,024-token prompts and 128-token outputs.
• Long-context and MoE workloads: Gemma 4 26B-A4B served in four-round long-context reports with BF16 KV cache.
• Multi-GPU cluster serving: GLM-5.2 with native MTP, either co-located on 4 GPUs or split across 8 GPUs for prefill and decode.
• Windows developers building from source through PowerShell with cargo run --release -p pegainfer-server.

Getting started

Install the prebuilt Qwen3 release with curl -fsSL https://raw.githubusercontent.com/pegainfer-project/pegainfer/main/install.sh | bash, download Qwen3-4B into models/Qwen3-4B, then run pegainfer --model-path models/Qwen3-4B. Source builds use the toolchain pinned in rust-toolchain.toml and cargo run --release -- --model-path models/Qwen3-4B, with --release required for GPU builds.

How it compares

The README benchmarks PegaInfer directly against vLLM 0.24.0, measuring Qwen3-4B serving footprint on a single RTX 5090 in BF16 with TP1, and running separate BF16 and default-FP8 vLLM comparisons. PegaInfer is Apache-2.0 and self-hosted, with model weights downloaded separately rather than bundled.

When to use it — and when not to

A self-hoster must supply the hardware and toolchain: Linux x86_64, an NVIDIA GPU with compute capability 8.x through 12.x, driver 580 or newer, glibc 2.35 or newer, and OpenSSL 3 for the prebuilt binary, plus a CUDA Toolkit with nvcc and cuBLAS and a compatible driver for source builds. Qwen3.5 additionally requires Python and Triton at build time, and K3 requires Python and TileLang, so builds that need those kernel paths are not Python-free. If the target platform is not Linux x86_64, or the team cannot manage CUDA driver floors, this engine is a poor fit; the repository also carries 89 open issues, and several model lines come with their own build requirements that a reader must verify before committing.

curl -fsSL https://raw.githubusercontent.com/pegainfer-project/pegainfer/main/install.sh | bash

pegainfer --model-path models/Qwen3-4B

export PATH="$HOME/.local/bin:$PATH"

export CUDA_HOME=/usr/local/cuda
cargo run --release -- --model-path models/Qwen3-4B

uv venv
uv pip install triton
export PEGAINFER_TRITON_PYTHON=.venv/bin/python
cargo run --release --features qwen35 -- --model-path models/Qwen3.5-4B

$env:CUDA_PATH = "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.x"
cargo run --release -p pegainfer-server -- --model-path models/Qwen3-4B

# Qwen3.5 additionally needs Triton at build time
uv venv .venv --python 3.12
uv pip install "triton-windows<3.7"
$env:PEGAINFER_TRITON_PYTHON = ".venv\Scripts\python.exe"
cargo run --release --features qwen35 -- --model-path models/Qwen3.5-4B