What it is

llama.cpp is a C/C++ library and set of command-line tools for running large language models (LLMs) and vision-language models (VLMs) locally. It enables inference without external dependencies, targeting diverse hardware including Apple Silicon, x86 CPUs, NVIDIA GPUs (via CUDA), AMD GPUs (via HIP), and other accelerators. The project lives in the ggml ecosystem, leveraging the ggml tensor computation library for low-level operations and quantized model execution.

It solves the problem of deploying LLM inference in resource-constrained or air-gapped environments where cloud APIs are unavailable, expensive, or undesirable for privacy reasons. By supporting quantized GGUF models and hardware-specific backends, it delivers high performance on commodity devices—from laptops to embedded systems—without requiring Python runtimes or heavy frameworks.

Key capabilities

• Run GGUF-formatted models directly from Hugging Face using llama cli -hf
• Launch an OpenAI-compatible REST API server with llama serve
• Support for CPU-only inference on ARM, x86, and RISC-V with SIMD and instruction set optimizations
• GPU acceleration via CUDA (NVIDIA), HIP (AMD), Vulkan, SYCL, and MUSA (Moore Threads)
• Hybrid CPU+GPU inference for models exceeding local VRAM capacity
• Built-in web UI for model interaction when running llama serve
• Quantization support for 1.5-bit through 8-bit integer formats to reduce memory and improve speed

Who uses it and how

• Developers embed llama.cpp in desktop or mobile apps for offline LLM features
• Researchers run experiments on local hardware without cloud billing or API rate limits
• Edge deployments use quantized models on Raspberry Pi, laptops, or embedded devices with minimal power draw

Getting started

Install via pre-built binaries from GitHub releases, build from source, use the Docker image ggmlorg/llama-cpp, or install via winget. Run models directly with llama cli -hf or start a server with llama serve -hf.

When to use it — and when not to

Use llama.cpp when you need local, dependency-free LLM inference with broad hardware support and quantization. Avoid it if you require full training support, complex model architectures not yet implemented in ggml, or enterprise-grade orchestration (e.g., autoscaling, multi-tenant isolation). Self-hosting requires manual model download, storage management, and optionally SMTP for email features—no built-in database or auth backend is included. Performance lags behind optimized cloud APIs on large models without GPU acceleration.

# Download and run a model directly from Hugging Face
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF

# Launch OpenAI-compatible API server
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF