[opensourcedex]

LocalAI is a free, open source machine learning infrastructure project written in Go and released under MIT. It has 49,144 GitHub stars, 4,455 forks and 201 open issues, and was last pushed 5 hours ago. On this registry it ranks #6 of 57 tracked projects in Machine Learning Infrastructure, with 5 head-to-head comparisons available. It gained 63 stars over the last 6 tracked days.

LocalAI — Self-hosted AI runtime for text, voice, vision, and agents

What is LocalAI?

What it is

LocalAI is an open-source AI runtime that enables running large language models (LLMs), vision, audio, image, and video models locally or on-premises. It operates as a modular engine where each model type is backed by a dedicated, lightweight backend—such as llama.cpp, whisper.cpp, or stable-diffusion—pulled only when needed. This composable architecture avoids bundling unnecessary dependencies, keeping the core minimal while supporting diverse modalities and hardware configurations.

It solves the problem of deploying AI workloads without relying on cloud services or requiring high-end GPUs. Users gain full data control and avoid per-token costs or vendor lock-in, while maintaining compatibility with popular APIs like OpenAI’s. The system supports CPU-only inference on commodity hardware, as well as acceleration via NVIDIA, AMD, Intel, Apple Silicon, and Vulkan backends.

Key capabilities

  • Runs LLMs, vision, audio, image, and video models behind a single unified API
  • Supports OpenAI, Anthropic, and ElevenLabs API compatibility across all backends
  • Pulls backends on-demand as separate Docker images, eliminating unused dependencies
  • Includes built-in AI agents with tool use, RAG, MCP, and skill execution
  • Provides multi-user support with API key authentication, quotas, and role-based access
  • Enables fine-tuning and quantization workflows via integrated tooling
  • Supports decentralized and distributed deployments using libp2p and MCP protocols

Who uses it and how

  • Developers deploy LocalAI in development or testing environments to iterate quickly without cloud dependencies or rate limits
  • Enterprises use it for private AI deployments where data sovereignty is required, such as internal chatbots or document analysis
  • Edge or on-premises teams run it on ARM64 devices (e.g., NVIDIA Jetson) for embedded vision or voice applications
  • Researchers run custom or fine-tuned models with full control over training data and inference logic
  • Hobbyists run lightweight models on laptops or Raspberry Pi–class hardware for personal assistants or creative tools

Getting started

Install via Docker using localai/localai:latest (CPU) or variant images for GPU support (e.g., localai/localai:latest-gpu-nvidia-cuda-13). Alternatively, use the CLI to run models directly from Hugging Face, Ollama, OCI registries, or YAML configs. Prebuilt binaries and macOS DMG are also available.

When to use it — and when not to

Use LocalAI when you need full control over model deployment, data privacy, or cost predictability, especially without GPU resources. It replaces paid APIs but requires managing infrastructure like storage, memory, and model loading. It is not ideal for high-throughput production workloads where latency or scalability demands exceed what CPU-based inference can sustain, or when relying on vendor-specific optimizations (e.g., CUDA-only features in vLLM) without fallback paths.

project readme (upstream, from github) — read inline




LocalAI License

Follow LocalAI_API Join LocalAI Discord Community

mudler%2FLocalAI | Trendshift

Deutsch | Español | français | 日本語 | 한국어 | Português | Русский | 中文

LocalAI is the open-source AI engine. Run any model - LLMs, vision, voice, image, video - on any hardware. No GPU required.

A small core, not a bundle. Each backend wraps a best-in-class engine (llama.cpp, vLLM, whisper.cpp, stable-diffusion, MLX...) in its own image, pulled only when a model needs it. You install nothing you don't use.

  • Composable by design: backends are separate and pulled on demand, so you install only what your model needs
  • Open and extensible: load any model, or build your own backend in any language against an open interface
  • Drop-in API compatibility: OpenAI, Anthropic, and ElevenLabs APIs across every backend
  • Any model, any modality: LLMs, vision, voice, image, and video behind one API
  • Any hardware: NVIDIA, AMD, Intel, Apple Silicon, Vulkan, or CPU-only
  • Multi-user ready: API key auth, user quotas, role-based access
  • Built-in AI agents: autonomous agents with tool use, RAG, MCP, and skills
  • Privacy-first: your data never leaves your infrastructure

A small LocalAI core with backends (llama.cpp, vLLM, MLX, whisper.cpp, stable-diffusion, kokoro, parakeet.cpp...) plugged in as separate on-demand images

Created by Ettore Di Giacinto and maintained by the LocalAI team.

:book: Documentation | :speech_balloon: Discord | 💻 Quickstart | 🖼️ Models | ❓FAQ

Guided tour

https://github.com/user-attachments/assets/08cbb692-57da-48f7-963d-2e7b43883c18

Click to see more!
User and auth

https://github.com/user-attachments/assets/228fa9ad-81a3-4d43-bfb9-31557e14a36c

Agents

https://github.com/user-attachments/assets/6270b331-e21d-4087-a540-6290006b381a

Usage metrics per user

https://github.com/user-attachments/assets/cbb03379-23b4-4e3d-bd26-d152f057007f

Fine-tuning and Quantization

https://github.com/user-attachments/assets/5ba4ace9-d3df-4795-b7d4-b0b404ea71ee

WebRTC

https://github.com/user-attachments/assets/ed88e34c-fed3-4b83-8a67-4716a9feeb7b

Quickstart

macOS

Download LocalAI for macOS

Note: The DMG is not signed by Apple. After installing, run: sudo xattr -d com.apple.quarantine /Applications/LocalAI.app. See #6268 for details.

Containers (Docker, podman, ...)

Already ran LocalAI before? Use docker start -i local-ai to restart an existing container.

CPU only:
docker run -ti --name local-ai -p 8080:8080 localai/localai:latest
NVIDIA GPU:
# CUDA 13
docker run -ti --name local-ai -p 8080:8080 --gpus all localai/localai:latest-gpu-nvidia-cuda-13

# CUDA 12
docker run -ti --name local-ai -p 8080:8080 --gpus all localai/localai:latest-gpu-nvidia-cuda-12

# NVIDIA Jetson ARM64 (CUDA 12, for AGX Orin and similar)
docker run -ti --name local-ai -p 8080:8080 --gpus all localai/localai:latest-nvidia-l4t-arm64

# NVIDIA Jetson ARM64 (CUDA 13, for DGX Spark)
docker run -ti --name local-ai -p 8080:8080 --gpus all localai/localai:latest-nvidia-l4t-arm64-cuda-13
AMD GPU (ROCm):
docker run -ti --name local-ai -p 8080:8080 --device=/dev/kfd --device=/dev/dri --group-add=video localai/localai:latest-gpu-hipblas
Intel GPU (oneAPI):
docker run -ti --name local-ai -p 8080:8080 --device=/dev/dri/card1 --device=/dev/dri/renderD128 localai/localai:latest-gpu-intel
Vulkan GPU:
docker run -ti --name local-ai -p 8080:8080 localai/localai:latest-gpu-vulkan

Loading models

# From the model gallery (see available models with `local-ai models list` or at https://models.localai.io)
local-ai run llama-3.2-1b-instruct:q4_k_m
# From Huggingface
local-ai run huggingface://TheBloke/phi-2-GGUF/phi-2.Q8_0.gguf
# From the Ollama OCI registry
local-ai run ollama://gemma:2b
# From a YAML config
local-ai run https://gist.githubusercontent.com/.../phi-2.yaml
# From a standard OCI registry (e.g., Docker Hub)
local-ai run oci://localai/phi-2:latest

To work with a running LocalAI server from the terminal, start the built-in agent from another shell. It answers questions, reads your files and runs commands on your machine, asking you to approve anything that changes state. Inside a session, /models lists installed models and /model switches between them. See the Terminal agent docs.

# Terminal 1
local-ai run llama-3.2-1b-instruct:q4_k_m

# Terminal 2
local-ai chat --model llama-3.2-1b-instruct:q4_k_m

Automatic Backend Detection: LocalAI automatically detects your GPU capabilities and downloads the appropriate backend. For advanced options, see GPU Acceleration.

For more details, see the Getting Started guide.

Latest News

readme truncated — read the full docs on github

Frequently asked questions

Is LocalAI free to use?

LocalAI is open source under the MIT licence. There is no licence fee and no seat count — you can self-host it or, where the project offers one, pay a vendor for a managed version instead.

What does LocalAI do?

Self-hosted AI runtime for text, voice, vision, and agents

What is LocalAI written in?

LocalAI is primarily written in Go. Its source is publicly available at https://github.com/mudler/LocalAI, and it has 49,144 GitHub stars.