LlamaFarm - Edge AI for Everyone

Enterprise AI capabilities on your own hardware. No cloud required.

LlamaFarm is an open-source AI platform that runs entirely on your hardware. Build RAG applications, train custom classifiers, detect anomalies, and run document processing—all locally with complete privacy.

• 🔒 Complete Privacy — Your data never leaves your device
• 💰 No API Costs — Use open-source models without per-token fees
• 🌐 Offline Capable — Works without internet once models are downloaded
• ⚡ Hardware Optimized — Automatic GPU/NPU acceleration on Apple Silicon, NVIDIA, and AMD

Desktop App Downloads

Get started instantly — no command line required:

What Can You Build?

Video demo (90 seconds): https://youtu.be/W7MHGyN0MdQ

Quickstart

Option 1: Desktop App

Download the desktop app above and run it. No additional setup required.

Option 2: CLI + Development Mode

1. Install the CLI

   macOS / Linux:

   curl -fsSL https://raw.githubusercontent.com/llama-farm/llamafarm/main/install.sh | bash
   

   Windows (PowerShell):

   irm https://raw.githubusercontent.com/llama-farm/llamafarm/main/install.ps1 | iex
   

   Or download directly from releases.

2. Create and run a project

   lf init my-project      # Generates llamafarm.yaml
   lf start                # Starts services and opens Designer UI
   

3. Chat with your AI

   lf chat                           # Interactive chat
   lf chat "Hello, LlamaFarm!"       # One-off message
   

The Designer web interface is available at http://localhost:14345.

Option 3: Development from Source

git clone https://github.com/llama-farm/llamafarm.git
cd llamafarm

# Install Nx globally and initialize the workspace
npm install -g nx
nx init --useDotNxInstallation --interactive=false  # Required on first clone

# Start all services (run each in a separate terminal)
nx start server           # FastAPI server (port 14345)
nx start rag              # RAG worker for document processing
nx start universal-runtime # ML models, OCR, embeddings (port 11540)

Architecture

LlamaFarm consists of three main services:

All configuration lives in llamafarm.yaml—no scattered settings or hidden defaults.

Runtime Options

Universal Runtime (Recommended)

The Universal Runtime provides access to HuggingFace models plus specialized ML capabilities:

• Text Generation - Any HuggingFace text model
• Embeddings - sentence-transformers and other embedding models
• OCR - Text extraction from images/PDFs (Surya, EasyOCR, PaddleOCR, Tesseract)
• Document Extraction - Forms, invoices, receipts via vision models
• Text Classification - Pre-trained or custom models via SetFit
• Named Entity Recognition - Extract people, organizations, locations
• Reranking - Cross-encoder models for improved RAG quality
• Anomaly Detection - Isolation Forest, One-Class SVM, Local Outlier Factor, Autoencoders

runtime:
  models:
    default:
      provider: universal
      model: Qwen/Qwen2.5-1.5B-Instruct
      base_url: http://127.0.0.1:11540/v1

Ollama

Simple setup for GGUF models with CPU/GPU acceleration:

runtime:
  models:
    default:
      provider: ollama
      model: qwen3:8b
      base_url: http://localhost:11434/v1

OpenAI-Compatible

Works with vLLM, Together, Mistral API, or any OpenAI-compatible endpoint:

runtime:
  models:
    default:
      provider: openai
      model: gpt-4o
      base_url: https://api.openai.com/v1
      api_key: ${OPENAI_API_KEY}

Core Workflows

CLI Commands

RAG Pipeline

1. Create a dataset linked to a processing strategy and database
2. Upload files (PDF, DOCX, Markdown, TXT) — processing runs automatically unless you pass --no-process
3. Process manually only when you intentionally skipped auto-processing (e.g., large batches)
4. Query using semantic search with optional metadata filtering

lf datasets create -s default -b main_db research
lf datasets upload research ./papers/*.pdf                 # auto-processes by default
# For large batches:
# lf datasets upload research ./papers/*.pdf --no-process
# lf datasets process research
lf rag query --database main_db "What are the key findings?"

Designer Web UI

The Designer at http://localhost:14345 provides:

• Project management with briefs and quick actions
• Visual dataset management with drag-and-drop uploads
• Database & RAG configuration with built-in query testing
• Prompt engineering with template variables and testing
• Interactive chat with RAG toggle and retrieved context display
• Config editor with syntax highlighting, validation, and auto-completion
• Switch between visual Designer and raw YAML modes in any section

See the Designer Features Guide for details.

Configuration

llamafarm.yaml is the source of truth for each project:

version: v1
name: my-assistant
namespace: default

## Multi-model configuration
runtime:
  default_model: fast

  models:
    fast:
      description: "Fast local model"
      provider: universal
      model: Qwen/Qwen2.5-1.5B-Instruct
      base_url: http://127.0.0.1:11540/v1

    powerful:
      description: "More capable model"
      provider: universal
      model: Qwen/Qwen2.5-7B-Instruct
      base_url: http://127.0.0.1:11540/v1

## System prompts
prompts:
  - name: default
    messages:
      - role: system
        content: You are a helpful assistant.

## RAG configuration
rag:
  databases:
    - name: main_db
      type: ChromaStore
      default_embedding_strategy: default_embeddings
      default_retrieval_strategy: semantic_search
      embedding_strategies:
        - name: default_embeddings
          type: UniversalEmbedder
          config:
            model: sentence-transformers/all-MiniLM-L6-v2
            base_url: http://127.0.0.1:11540/v1
      retrieval_strategies:
        - name: semantic_search
          type: BasicSimilarityStrategy
          config:
            top_k: 5

  data_process