[opensourcedex]

llama.cpp is a free, open source machine learning infrastructure project written in C++ and released under MIT. It has 128,581 GitHub stars, 23,334 forks and 2,464 open issues, and was last pushed 5 hours ago. On this registry it ranks #2 of 57 tracked projects in Machine Learning Infrastructure, with 5 head-to-head comparisons available. It gained 586 stars over the last 6 tracked days.

What is llama.cpp?

What it is

llama.cpp is a C/C++ library and set of command-line tools for running large language models (LLMs) and vision-language models (VLMs) locally. It enables inference without external dependencies, targeting diverse hardware including Apple Silicon, x86 CPUs, NVIDIA GPUs (via CUDA), AMD GPUs (via HIP), and other accelerators. The project lives in the ggml ecosystem, leveraging the ggml tensor computation library for low-level operations and quantized model execution.

It solves the problem of deploying LLM inference in resource-constrained or air-gapped environments where cloud APIs are unavailable, expensive, or undesirable for privacy reasons. By supporting quantized GGUF models and hardware-specific backends, it delivers high performance on commodity devices—from laptops to embedded systems—without requiring Python runtimes or heavy frameworks.

Key capabilities

  • Run GGUF-formatted models directly from Hugging Face using llama cli -hf
  • Launch an OpenAI-compatible REST API server with llama serve
  • Support for CPU-only inference on ARM, x86, and RISC-V with SIMD and instruction set optimizations
  • GPU acceleration via CUDA (NVIDIA), HIP (AMD), Vulkan, SYCL, and MUSA (Moore Threads)
  • Hybrid CPU+GPU inference for models exceeding local VRAM capacity
  • Built-in web UI for model interaction when running llama serve
  • Quantization support for 1.5-bit through 8-bit integer formats to reduce memory and improve speed

Who uses it and how

  • Developers embed llama.cpp in desktop or mobile apps for offline LLM features
  • Researchers run experiments on local hardware without cloud billing or API rate limits
  • Edge deployments use quantized models on Raspberry Pi, laptops, or embedded devices with minimal power draw

Getting started

Install via pre-built binaries from GitHub releases, build from source, use the Docker image ggmlorg/llama-cpp, or install via winget. Run models directly with llama cli -hf or start a server with llama serve -hf.

When to use it — and when not to

Use llama.cpp when you need local, dependency-free LLM inference with broad hardware support and quantization. Avoid it if you require full training support, complex model architectures not yet implemented in ggml, or enterprise-grade orchestration (e.g., autoscaling, multi-tenant isolation). Self-hosting requires manual model download, storage management, and optionally SMTP for email features—no built-in database or auth backend is included. Performance lags behind optimized cloud APIs on large models without GPU acceleration.

project readme (upstream, from github) — read inline

llama.cpp

llama

LLM inference in C/C++

License: MIT Release Nightly Server Docker Winget

ggml / ops / maintainer PRs / dev stats / lib llama API / llama-server REST API

Quick start

A few options to get llama.cpp installed on your machine:

Once installed:

# Download and run a model directly from Hugging Face
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF

# Launch OpenAI-compatible API server
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF
VLM session with `llama cli` VLM session with llama cli Built-in web UI against `llama serve` running Qwen 3.6 Built-in web UI against llama serve

Description

The main goal of llama.cpp is to enable LLM (and VLM) inference with minimal setup and state-of-the-art performance on a wide range of hardware - locally and in the cloud.

  • Plain C/C++ implementation without any dependencies
  • Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks
  • AVX, AVX2, AVX512 and AMX support for x86 architectures
  • RVV, ZVFH, ZFH, ZICBOP and ZIHINTPAUSE support for RISC-V architectures
  • 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use
  • Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs via HIP and Moore Threads GPUs via MUSA)
  • Vulkan and SYCL backend support
  • CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity

The llama.cpp project is build on top of the ggml library.

Supported backends

Backend Target devices
BLAS All
BLIS All
CANN Ascend NPU
CUDA Nvidia GPU
HIP AMD GPU
Hexagon Snapdragon
IBM zDNN IBM Z & LinuxONE
MUSA Moore Threads GPU
Metal Apple Silicon
OpenCL Adreno GPU
OpenVINO [In Progress] Intel CPUs, GPUs, and NPUs
RPC All
SYCL Intel GPU
VirtGPU VirtGPU APIR
Vulkan GPU
WebGPU All
ZenDNN AMD CPU

Documentation

Tools
Development

Contributing

Acknowledgements

Frequently asked questions

Is llama.cpp free to use?

llama.cpp is open source under the MIT licence. There is no licence fee and no seat count — you can self-host it or, where the project offers one, pay a vendor for a managed version instead.

What does llama.cpp do?

Run LLMs locally with minimal setup, maximum hardware support

What is llama.cpp written in?

llama.cpp is primarily written in C++. Its source is publicly available at https://github.com/ggml-org/llama.cpp, and it has 128,581 GitHub stars.