vLLM is a high-throughput, memory-efficient Python library for LLM inference and serving, built for ML engineers and platform teams who need to run open-weight models on their own hardware at production scale.

What it is

vLLM is an open-source inference and serving engine for large language models, written in Python and released under the Apache-2.0 licence. It was originally developed in the Sky Computing Lab at UC Berkeley and has since grown into a project maintained by over 2000 contributors across many dozens of academic institutions and companies. Its central technical idea is PagedAttention, an attention key-and-value memory management scheme described in a paper presented at the ACM SIGSIGOPS 29th Symposium on Operating Systems Principles in 2023. The library sits inside the Python and Hugging Face ecosystem and supports more than 200 model architectures hosted on Hugging Face, spanning decoder-only LLMs such as Llama, Qwen and Gemma, mixture-of-experts models such as Mixtral, DeepSeek-V3, Qwen-MoE and GPT-OSS, hybrid attention and state-space models such as Mamba and Qwen3.5, multi-modal models, embedding and retrieval models, and reward and classification models.

The concrete problem it solves is the cost and complexity of serving those models directly. Rather than paying a hosted provider per token, an operator runs vLLM on their own accelerators and exposes an OpenAI-compatible API server, which means existing clients written against the OpenAI API can be pointed at a self-hosted endpoint without rewriting application code. It therefore replaces the hosted inference endpoint as the serving layer for teams that want their models, weights and request data to stay on their own infrastructure.

Key capabilities

• PagedAttention for efficient management of attention key and value memory, the mechanism behind the library's memory efficiency.
• Continuous batching of incoming requests, chunked prefill and prefix caching, all used to raise serving throughput.
• Quantization support across FP8, MXFP8/MXFP4, NVFP4, INT8, INT4, GPTQ/AWQ, GGUF, compressed-tensors, ModelOpt and TorchAO.
• Optimized attention kernels including FlashAttention, FlashInfer, TRTLLM-GEN, FlashMLA and Triton, plus GEMM and MoE kernels built with CUTLASS, TRTLLM-GEN and CuTeDSL.
• Speculative decoding with n-gram, suffix, EAGLE and DFlash methods.
• Tensor, pipeline, data, expert and context parallelism for distributed inference, including disaggregated prefill, decode and encode.
• Serving interfaces covering an OpenAI-compatible API server, the Anthropic Messages API and gRPC, with streaming outputs, structured output generation through xgrammar or guidance, tool calling and reasoning parsers, and multi-LoRA support for both dense and MoE layers.

Who uses it and how

• Teams replacing a hosted inference API in existing applications, since the OpenAI-compatible server accepts requests from clients already written against that interface.
• Operators of mixed hardware fleets, because vLLM runs on NVIDIA, AMD and Intel GPUs as well as x86, ARM and PowerPC CPUs, with plugins for Google TPUs, Intel Gaudi, IBM Spyre, Huawei Ascend, Rebellions NPU, Apple Silicon and MetaX GPUs.
• Multi-tenant serving deployments that need many adapters at once, served through multi-LoRA support for dense and MoE layers.
• Large distributed inference clusters that separate prefill, decode and encode stages, using the parallelism options the library exposes.
• Research and engineering groups that need reproducible, self-hosted serving of open-weight models rather than a vendor-controlled endpoint.

Getting started

Install with uv pip install vllm, which the README recommends, or with pip. Building a wheel from source is the documented route for development, and installation, quickstart and supported-model guides live in the documentation at docs.vllm.ai.

How it compares

No list of paid products that vLLM replaces is provided in the facts, and no directly comparable serving engine is named, so on the evidence available it stands alone in this registry. Its positioning is defined instead by the interfaces it exposes: an OpenAI-compatible API server, the Anthropic Messages API and gRPC support mean it can be substituted for a hosted inference endpoint while remaining self-hosted and Apache-2.0 licensed.

When to use it — and when not to

A self-hoster must operate accelerator hardware and the serving stack around it, because vLLM is an inference engine rather than a managed service, and it depends on GPU kernels such as CUDA and HIP graphs for its performance characteristics. Teams without GPU access, or those who want a vendor to run the endpoint for them, should look elsewhere. Two limitations are visible in the facts: the repository carries roughly 7953 open issues, and the README is largely a set of links outward to vllm.ai and docs.vllm.ai, so the substantive operational detail lives off-repo.

uv pip install vllm

@inproceedings{kwon2023efficient,
  title={Efficient Memory Management for Large Language Model Serving with PagedAttention},
  author={Woosuk Kwon and Zhuohan Li and Siyuan Zhuang and Ying Sheng and Lianmin Zheng and Cody Hao Yu and Joseph E. Gonzalez and Hao Zhang and Ion Stoica},
  booktitle={Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles},
  year={2023}
}