Open source inference projects
Every project in the registry tagged inference, ranked by real GitHub adoption.
A high-throughput and memory-efficient inference and serving engine for LLMs
A framework for efficient model inference with omni-modality models
A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.
TypeDB: Built for systems, not records
High-performance Inference and Deployment Toolkit for LLMs and VLMs based on PaddlePaddle
Route GenAI traffic with confidence using open source
Manages Unified Access to Generative AI Services built on Envoy Gateway
GPU orchestration across clouds, Kubernetes, and on-prem
Open Source Continuous Inference Benchmark Research Platform — Kimi K3 2.8T, MiniMax M3, DeepSeekv4, GLM5 - GB200 NVL72 vs MI355X vs B200 vs GB300 NVL72 & soon™
Unified API for all LLM providers with analytics
SGLang-Omni is a high-performance serving framework for audio models (TTS, ASR) and unified multimodal models.
Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2
High-performance serverless GPU task orchestration — the scheduling layer behind wavespeed.ai
Related tags
Frequently asked questions
How many open source inference projects are there?
This registry tracks 13 projects tagged inference, with 131,516 GitHub stars between them. The most-adopted is vllm at 96,158 stars.
Are these inference projects free to use?
Yes — 12 of the 13 carry an explicit open-source licence across 2 distinct licences, so there is no licence fee. Where a project also sells a hosted or enterprise version, the self-hosted path remains free.
Which inference project should I choose?
The list above is ranked by GitHub stars, but stars measure attention rather than fit. Check three things on each card: the licence (permissive versus copyleft), the language it is written in, and the last-push date — a high-star project that has not been pushed in a year is a liability.
Are these inference projects still maintained?
13 of the 13 were pushed in the last 90 days, and every card shows its exact last-push date so you can see the rest. Sort your shortlist by that date before committing to a migration.