What it is

Chitu is an open-source Python inference framework for large language models. It lives in the PyTorch GPU model-serving ecosystem, with topics including deepseek, llm-serving, model-serving, and pytorch, and it is Apache-2.0 licensed. The project describes itself as a high-performance engine focused on efficiency, flexibility, and availability for production deployment.

The concrete problem is running large language models on diverse compute resources without a fixed hardware stack. Its milestones describe support for NVIDIA, Huawei Ascend, Moore Threads, Muxi, and Hygon devices, plus pure CPU, single GPU, all-in-one machine, and cluster scenarios. It also targets models such as DeepSeek-R1 671B, Qwen, GLM, and Kimi, with milestones describing quantization, single-card inference, and cluster deployment.

Key capabilities

• It provides accelerator support across NVIDIA, Huawei Ascend, Moore Threads, Muxi, and Hygon hardware, as indicated by milestones and the multi-compute adaptation section.
• It supports DeepSeek, Qwen, GLM, and Kimi model families, including DeepSeek-R1 671B and GLM-4.5 MoE on Ascend 910B.
• It includes quantization and dtype conversion operators, including FP4 online to FP8 and BF16, FP8 online to BF16, and DeepSeek-R1 671B FP4 quantized model support.
• It supports CPU+GPU heterogeneous mixed inference, which v0.2.2 says enables single-card inference of DeepSeek-R1 671B.
• It scales from pure CPU and single GPU to large clusters, and v0.6.0 adds chitu.run for launching multi-node, multi-instance, and PD-separated tasks.

Who uses it and how

• Teams deploying DeepSeek-R1 671B on a single card can use the CPU+GPU path and quantization operators described in milestones.
• Organizations with mixed NVIDIA and domestic accelerator fleets can use Chitu as a common serving layer for supported models.
• Engineers preparing all-in-one machine inference can follow v0.4.0 emphasis on stability and performance for such deployments.
• Operators building multi-node or PD-separated serving topologies can use chitu.run to start tasks from a single executable.

Getting started

The README recommends deploying with the chitu.run executable from GitHub Releases and consulting the development manual for installation and usage. It also points to supported-model documentation.

When to use it — and when not to

Chitu is useful when a self-hosted team needs an Apache-2.0 Python inference framework targeting multiple GPU vendors, CPU+GPU mixed inference, and cluster-scale LLM serving. It is less suitable when an organization needs guaranteed support, because the README says the team cannot ensure timely resolution of all user issues. It also requires operators to manage hardware, deployment topology, performance validation, and model selection, since performance data depends on hardware, software version, and test load.