[opensourcedex]

CosyVoice is a free, open source ai interaction & interfaces project written in Python and released under Apache-2.0. It has 23,661 GitHub stars, 2,690 forks and 698 open issues, and was last pushed 4 months ago. On this registry it ranks #26 of 76 tracked projects in AI Interaction & Interfaces, with 5 head-to-head comparisons available. It gained 33 stars over the last 3 tracked days.

What is CosyVoice?

CosyVoice is an Apache-2.0 Python text-to-speech system built on large language models that ships inference, training and deployment for multi-lingual voice generation, aimed at teams who need self-hosted zero-shot voice cloning and streaming speech synthesis rather than a hosted API.

What it is

CosyVoice is a multi-lingual large voice generation model released by FunAudioLLM, distributed as Python source under Apache-2.0 with model weights published on ModelScope and HuggingFace. It lives in the AI and machine learning ecosystem, specifically AI interaction and interfaces, and it is built around large language model decoding rather than a conventional acoustic pipeline. Three model generations are documented in the README: CosyVoice 1.0 with CosyVoice-300M, CosyVoice 2.0 with CosyVoice2-0.5B, and Fun-CosyVoice 3.0 with Fun-CosyVoice3-0.5B-2512, each with its own demos, paper and weights. The repository covers the full stack, from training and inference scripts to a Fastapi server and client, so it is meant to be run and adapted, not only called.

The concrete problem it solves is the traditional text-to-speech frontend. CosyVoice performs text normalization for numbers, special symbols and various text formats without a traditional frontend module, and it falls back to WeTextProcessing when ttsfrd is unavailable. It replaces the patchwork of a grapheme-to-phoneme frontend, a separate normalizer and a separate vocoder with one LLM-backed model that handles pronunciation explicitly through pronunciation inpainting of Chinese Pinyin and English CMU phonemes. For teams that previously had to assemble these pieces or depend on a closed hosted speech service, the repository provides the training code, the evaluation set and the serving path in one place.

Key capabilities

  • Covers 9 common languages (Chinese, English, Japanese, Korean, German, Spanish, French, Italian, Russian) plus 18+ Chinese dialects and accents including Guangdong, Minnan, Sichuan, Dongbei, Shan3xi, Shan1xi, Shanghai, Tianjin, Shandong, Ningxia and Gansu.
  • Supports multi-lingual and cross-lingual zero-shot voice cloning, so a target voice can be reproduced without per-speaker training.
  • Offers pronunciation inpainting for Chinese Pinyin and English CMU phonemes, which makes output correctable for production use.
  • Provides bi-streaming with both text-in streaming and audio-out streaming, reaching latency as low as 150ms.
  • Accepts instructions for language, dialect, emotion, speed and volume through instruct support.
  • Ships optimized runtimes and decoding: vLLM support for CosyVoice2-0.5B, Triton TRT-LLM runtime support contributed by NVIDIA, GRPO training for CosyVoice2, flow matching training, and Repetition Aware Sampling for LLM stability.
  • Delivers 25hz CosyVoice2-0.5B and 25hz CosyVoice-300M models with a voice conversion function, plus a Fastapi server and client.

Who uses it and how

  • Builders of chatbots and voice assistants, indicated by the chatbot, chatgpt and gpt-4o topics, who need spoken output rather than text replies.
  • Cantonese and Chinese dialect content teams, since dialect and accent coverage is a named feature rather than an afterthought.
  • Real-time agent developers who need streaming synthesis, because the text-in and audio-out streaming modes target low latency.
  • Speech research and fine-tuning teams, who can use the base model, RL model, training and inference scripts, the eval set and CV3-Eval to reproduce or extend results.
  • Cross-lingual dubbing and cloning workflows, where one reference voice must speak several languages.

Getting started

The README routes users to the Fun-CosyVoice3-0.5B-2512 weights on ModelScope (FunAudioLLM/Fun-CosyVoice3-0.5B-2512) and HuggingFace, the ModelScope Gradio space, and the repository training and inference scripts alongside the Fastapi server and client. The excerpt does not pin an install command or distribution package name, so plan to follow the repository setup directly.

How it compares

The evaluation table places CosyVoice2 among other open-source systems such as F5-TTS at 0.3B, Spark TTS at 0.5B, FireRedTTS2 at 1.5B, Index-TTS2 at 1.5B and VibeVoice-1.5B, while Seed-TTS and MiniMax-Speech are marked as not open source. Against the open-source entries, CosyVoice competes at the 0.5B size with dialect coverage, streaming and instruct control rather than on size alone. Against the closed systems, the difference is that its weights, training scripts and serving code are available for self-hosting.

When to use it — and when not to

A self-hoster must operate a Python environment with GPU inference runtimes such as vLLM and the Triton TRT-LLM path, fetch weights from ModelScope or HuggingFace, and run either the Gradio space or the Fastapi server. The project carries 698 open issues and the README excerpt omits pinned install commands and dependency versions, so integration effort should be budgeted. Avoid it if a managed API is required, if GPU hosting is unavailable, or if upstream maintenance responsiveness is a hard requirement.

project readme (upstream, from github) — read inline

SVG Banners

👉🏻 CosyVoice 👈🏻

Fun-CosyVoice 3.0: Demos; Paper; Modelscope; Huggingface; CV3-Eval

CosyVoice 2.0: Demos; Paper; Modelscope; HuggingFace

CosyVoice 1.0: Demos; Paper; Modelscope; HuggingFace

Highlight🔥

Fun-CosyVoice 3.0 is an advanced text-to-speech (TTS) system based on large language models (LLM), surpassing its predecessor (CosyVoice 2.0) in content consistency, speaker similarity, and prosody naturalness. It is designed for zero-shot multilingual speech synthesis in the wild.

Key Features

  • Language Coverage: Covers 9 common languages (Chinese, English, Japanese, Korean, German, Spanish, French, Italian, Russian), 18+ Chinese dialects/accents (Guangdong, Minnan, Sichuan, Dongbei, Shan3xi, Shan1xi, Shanghai, Tianjin, Shandong, Ningxia, Gansu, etc.) and meanwhile supports both multi-lingual/cross-lingual zero-shot voice cloning.
  • Content Consistency & Naturalness: Achieves state-of-the-art performance in content consistency, speaker similarity, and prosody naturalness.
  • Pronunciation Inpainting: Supports pronunciation inpainting of Chinese Pinyin and English CMU phonemes, providing more controllability and thus suitable for production use.
  • Text Normalization: Supports reading of numbers, special symbols and various text formats without a traditional frontend module.
  • Bi-Streaming: Support both text-in streaming and audio-out streaming, and achieves latency as low as 150ms while maintaining high-quality audio output.
  • Instruct Support: Supports various instructions such as languages, dialects, emotions, speed, volume, etc.

Roadmap

  • 2025/12

    • release Fun-CosyVoice3-0.5B-2512 base model, rl model and its training/inference script
    • release Fun-CosyVoice3-0.5B modelscope gradio space

  • 2025/08

    • Thanks to the contribution from NVIDIA Yuekai Zhang, add triton trtllm runtime support and cosyvoice2 grpo training support

  • 2025/07

    • release Fun-CosyVoice 3.0 eval set

  • 2025/05

    • add CosyVoice2-0.5B vllm support

  • 2024/12

    • 25hz CosyVoice2-0.5B released

  • 2024/09

    • 25hz CosyVoice-300M base model
    • 25hz CosyVoice-300M voice conversion function

  • 2024/08

    • Repetition Aware Sampling(RAS) inference for llm stability
    • Streaming inference mode support, including kv cache and sdpa for rtf optimization

  • 2024/07

    • Flow matching training support
    • WeTextProcessing support when ttsfrd is not available
    • Fastapi server and client

Evaluation

Model Open-Source Model Size test-zh
CER (%) ↓		 test-zh
SS (%) ↑		 test-en
WER (%) ↓		 test-en
SS (%) ↑		 test-hard
CER (%) ↓		 test-hard
SS (%) ↑
Human - - 1.26 75.5 2.14 73.4 - -
Seed-TTS ❌ - 1.12 79.6 2.25 76.2 7.59 77.6
MiniMax-Speech ❌ - 0.83 78.3 1.65 69.2 - -
F5-TTS ✅ 0.3B 1.52 74.1 2.00 64.7 8.67 71.3
Spark TTS ✅ 0.5B 1.2 66.0 1.98 57.3 - -
CosyVoice2 ✅ 0.5B 1.45 75.7 2.57 65.9 6.83 72.4
FireRedTTS2 ✅ 1.5B 1.14 73.2 1.95 66.5 - -
Index-TTS2 ✅ 1.5B 1.03 76.5 2.23 70.6 7.12 75.5
VibeVoice-1.5B ✅ 1.5B 1.16 74.4 3.04 68.9 - -
VibeVoice-Realtime ✅ 0.5B - - 2.05 63.3 - -
HiggsAudio-v2 ✅ 3B 1.50 74.0 2.44 67.7 - -
VoxCPM ✅ 0.5B 0.93 77.2 1.85 72.9 8.87 73.0
GLM-TTS ✅ 1.5B 1.03 76.1 - - - -
GLM-TTS RL ✅ 1.5B 0.89 76.4 - - - -
Fun-CosyVoice3-0.5B-2512 ✅ 0.5B 1.21 78.0 2.24 71.8 6.71 75.8
Fun-CosyVoice3-0.5B-2512_RL ✅ 0.5B 0.81 77.4 1.68 69.5 5.44 75.0

Install

Clone and install

  • Clone the repo

    git clone --recursive https://github.com/FunAudioLLM/CosyVoice.git
# If you failed to clone the submodule due to network failures, please run the following command until success
cd CosyVoice
git submodule update --init --recursive

  • Install Conda: please see https://docs.conda.io/en/latest/miniconda.html

  • Create Conda env:

    conda create -n cosyvoice -y python=3.10
conda activate cosyvoice
pip install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/ --trusted-host=mirrors.aliyun.com

# If you encounter sox compatibility issues
# ubuntu
sudo apt-get install sox libsox-dev
# centos
sudo yum install sox sox-devel

Model download

We strongly recommend that you download our pretrained Fun-CosyVoice3-0.5B CosyVoice2-0.5B CosyVoice-300M CosyVoice-300M-SFT CosyVoice-300M-Instruct model and CosyVoice-ttsfrd resource.

# modelscope SDK model download
from modelscope import snapshot_download
snapshot_download('FunAudioLLM/Fun-CosyVoice3-0.5B-2512', local_dir='pretrained_models/Fun-CosyVoice3-0.5B')
snapshot_download('iic/CosyVoice2-0.5B', local_dir='pretrained_models/CosyVoice2-0.5B')
snapshot_download('iic/CosyVoice-300M', local_dir='pretrained_models/CosyVoice-300M')
snapshot_download('iic/CosyVoice-300M-SFT', local_dir='pretrained_models/CosyVoice-300M-SFT')
snapshot_download('iic/CosyVoice-300M-Instruct', local_dir='pretrained_models/CosyVoice-300M-Instruct')
snapshot_download('iic/CosyVoice-ttsfrd', local_dir='pretrained_models/CosyVoice-ttsfrd')

# for oversea users, huggingface SDK model download
from huggingface_hub import snapshot_download
snapshot_download('FunAudioLLM/Fun-CosyVoice3-0.5B-2512', local_dir='pretrained_models/Fun-CosyVoice3-0.5B')
snapshot_download('FunAudioLLM/CosyVoice2-0.5B', local_dir='pretrained_models/CosyVoice2-0.5B')
snapshot_download('FunAudioLLM/CosyVoice-300M', local_dir='pretrained_models/CosyVoice-300M')
snapshot_download('FunAudioLLM/CosyVoice-300M-SFT', local_dir='pretrained_models/CosyVoice-300M-SFT')
snapshot_download('FunAudioLLM/CosyVoice-300M-Instruct', local_dir='pretrained_models/CosyVoice-300M-Instruct')
snapshot_download('FunAudioLLM/CosyVoice-ttsfrd', local_dir='pretrained_models/CosyVoice-ttsfrd')

Optionally, you can unzip ttsfrd resource and install ttsfrd package for better text normalization performance.

Notice that this step is not necessary. If you do not install ttsfrd package, we will use wetext by default.

cd pretrained_models/CosyVoice-ttsfrd/
unzip resource.zip -d .
pip install ttsfrd_dependency-0.1-py3-none-any.whl
pip install ttsfrd-0.4.2-cp310-cp310-linux_x86_64.whl

Basic Usage

We strongly recommend using Fun-CosyVoice3-0.5B for better performance. Follow the code in example.py for detailed usage of each model.

python example.py
vLLM Usage

CosyVoice2/3 now supports vLLM 0.11.x+ (V1 engine) and vLLM 0.9.0 (legacy). Older vllm version(<0.9.0) do not support CosyVoice inference, and versions in between (e.g., 0.10.x) are not tested.

Notice that vllm has a lot of specific requirements. You can create a new env to in case your hardward do not support vllm and old env is corrupted.

conda create -n cosyvoice_vllm --clone cosyvoice
conda activate cosyvoice_vllm
# for vllm==0.9.0
pip install vllm==v0.9.0 transformers==4.51.3 numpy==1.26.4 -i https://mirrors.aliyun.com/pypi/simple/ --trusted-host=mirrors.aliyun.com
# for vllm>=0.11.0
pip install vllm==v0.11.0 transformers==4.57.1 numpy==1.26.4 -i https://mirrors.aliyun.com/pypi/simple/ --trusted-host=mirrors.aliyun.com
python vllm_example.py
Start web demo

You can use our web demo page to get familiar with CosyVoice quickly.

Please see the demo website for details.

# change iic/CosyVoice-300M-SFT for sft inference, or iic/CosyVoice-300M-Instruct for instruct inference
python3 webui.py --port 50000 --model_dir pretrained_models/CosyVoice-300M
Advanced Usage

For advanced users, we have provided training and inference scripts in examples/libritts.

Build for deployment

Optionally, if you want service deployment, You can run the following steps.

cd runt

readme truncated — read the full docs on github

Frequently asked questions

Is CosyVoice free to use?

CosyVoice is open source under the Apache-2.0 licence. There is no licence fee and no seat count — you can self-host it or, where the project offers one, pay a vendor for a managed version instead.

What does CosyVoice do?

Multi-lingual large voice generation model, providing inference, training and deployment full-stack ability.

What is CosyVoice written in?

CosyVoice is primarily written in Python. Its source is publicly available at https://github.com/QwenAudio/CosyVoice, and it has 23,661 GitHub stars.