🔗Integrations & Examples

There are integrations for the following libraries, making it easier to develop your APP. These integrations are developed in their own repositories.

• BotSharp: an open source machine learning framework for AI Bot platform builder.
• Langchain: a framework for developing applications powered by language models.
• MaIN.NET: simplistic approach to orchestrating agents/chats from different (llm) providers

The following examples show how to build APPs with LLamaSharp.

• Official Console Examples
• Unity Demo
• LLamaStack (with WPF and Web demo)
• Blazor Demo (with Model Explorer)
• ASP.NET Demo
• LLamaWorker (ASP.NET Web API like OAI and Function Calling Support)
• VirtualPet (Desktop Application)
• KaiROS AI (Local AI Assistant for Windows & Android)

🚀Get started

Installation

To gain high performance, LLamaSharp interacts with native libraries compiled from c++, these are called backends. We provide backend packages for Windows, Linux and Mac with CPU, CUDA, Metal and Vulkan. You don't need to compile any c++, just install the backend packages.

If no published backend matches your device, please open an issue to let us know. If compiling c++ code is not difficult for you, you could also follow this guide to compile a backend and run LLamaSharp with it.

1. Install LLamaSharp package on NuGet:

PM> Install-Package LLamaSharp

2. Install one or more of these backends, or use a self-compiled backend.

   • LLamaSharp.Backend.Cpu: Pure CPU for Windows, Linux & Mac. Metal (GPU) support for Mac.
   • LLamaSharp.Backend.Cuda11: CUDA 11 for Windows & Linux.
   • LLamaSharp.Backend.Cuda12: CUDA 12 for Windows & Linux.
   • LLamaSharp.Backend.Vulkan: Vulkan for Windows & Linux.

Model preparation

There are two popular formats of model file of LLMs, these are PyTorch format (.pth) and Huggingface format (.bin). LLamaSharp uses a GGUF format file, which can be converted from these two formats. To get a GGUF file, there are two options:

1. Search model name + 'gguf' in Huggingface, you will find lots of model files that have already been converted to GGUF format. Please take note of the publishing time of them because some old ones may only work with older versions of LLamaSharp.

2. Convert PyTorch or Huggingface format to GGUF format yourself. Please follow the instructions from this part of llama.cpp readme to convert them with python scripts.

Generally, we recommend downloading models with quantization rather than fp16, because it significantly reduces the required memory size while only slightly impacting the generation quality.

Example of LLaMA chat session

Here is a simple example to chat with a bot based on a LLM in LLamaSharp. Please replace the model path with yours.

using LLama;
using LLama.Common;
using LLama.Sampling;

string modelPath = @"<Your Model Path>"; // change it to your own model path.

var parameters = new ModelParams(modelPath)
{
    ContextSize = 1024, // The longest length of chat as memory.
    GpuLayerCount = 5 // How many layers to offload to GPU. Please adjust it according to your GPU memory.
};
using var model = LLamaWeights.LoadFromFile(parameters);
using var context = model.CreateContext(parameters);
var executor = new InteractiveExecutor(context);

// Add chat histories as prompt to tell AI how to act.
var chatHistory = new ChatHistory();
chatHistory.AddMessage(AuthorRole.System, "Transcript of a dialog, where the User interacts with an Assistant named Bob. Bob is helpful, kind, honest, good at writing, and never fails to answer the User's requests immediately and with precision.");
chatHistory.AddMessage(AuthorRole.User, "Hello, Bob.");
chatHistory.AddMessage(AuthorRole.Assistant, "Hello. How may I help you today?");

ChatSession session = new(executor, chatHistory);

InferenceParams inferenceParams = new InferenceParams()
{
    MaxTokens = 256, // No more than 256 tokens should appear in answer. Remove it if antiprompt is enough for control.
    AntiPrompts = new List<string> { "User:" }, // Stop generation once antiprompts appear.

    SamplingPipeline = new DefaultSamplingPipeline(),
};

Console.ForegroundColor = ConsoleColor.Yellow;
Console.Write("The chat session has started.\nUser: ");
Console.ForegroundColor = ConsoleColor.Green;
string userInput = Console.ReadLine() ?? "";

while (userInput != "exit")
{
    await foreach ( // Generate the response streamingly.
        var text
        in session.ChatAsync(
            new ChatHistory.Message(AuthorRole.User, userInput),
            inferenceParams))
    {
        Console.ForegroundColor = ConsoleColor.White;
        Console.Write(text);
    }
    Console.ForegroundColor = ConsoleColor.Green;
    userInput = Console.ReadLine() ?? "";
}

For more examples, please refer to LLamaSharp.Examples.

💡FAQ

Why is my GPU not use

readme truncated — read the full docs on github