[opensourcedex]

Amphion is a free, open source compliance & risk management project written in Python and released under MIT. It has 10,329 GitHub stars, 855 forks and 174 open issues, and was last pushed 7 months ago. On this registry it ranks #7 of 56 tracked projects in Compliance & Risk Management, with 5 head-to-head comparisons available. It gained 25 stars over the last 30 tracked days.

What is Amphion?

Amphion is an MIT-licensed Python toolkit for audio, music and speech generation that gives researchers and engineers a shared, reproducible base for tasks such as text to speech, singing voice synthesis, voice conversion and text to audio.

What it is

Amphion lives in the open-source Python audio and machine-learning research ecosystem, maintained under the open-mmlab GitHub organisation. The repository states a North-Star objective of offering a platform for studying the conversion of any inputs into audio. It covers several individual generation tasks in one place: Text to Speech (TTS), Singing Voice Synthesis (SVS), Voice Conversion (VC), Accent Conversion (AC), Singing Voice Conversion (SVC) and Text to Audio (TTA) are supported, while Text to Music (TTM) is still in development. Beyond the tasks themselves, the project ships vocoders, which produce the high-quality audio signal, and evaluation metrics, which keep scores comparable across experiments. It also includes work on building large-scale datasets for speech synthesis, such as Emilia-Large at over 200,000 hours, and it offers visualisations of classic models and architectures, which the README presents as its unique feature.

The concrete problem it addresses is fragmentation. Each generation task normally arrives as its own paper, its own codebase, its own vocoder choice and its own measurement convention, so results are hard to compare and hard for a newcomer to enter. Amphion gathers the tasks, the vocoders and the evaluation metrics behind one project, which is what allows the metrics to stay consistent between generation tasks and what supports reproducible research. The stated audience is junior researchers and engineers who want to get started in audio, music and speech generation research and development, and the architecture visualisations exist specifically so that those readers can understand a model before running it. Distribution follows that audience: code on GitHub, pretrained checkpoints on Hugging Face, and at least one model runnable from a Colab notebook.

Key capabilities

  • Six supported generation tasks in one toolkit: TTS, SVS, VC, AC, SVC and TTA, with TTM in development.
  • Vevo2 (models/svc/vevo2/README.md), a unified and controllable framework that bridges speech and singing voice generation through unified prosody learning, covering TTS, SVS, VC, SVC, singing voice editing, singing style conversion and melody control.
  • DualCodec (models/codec/dualcodec/README.md), a low-frame-rate neural audio codec operating at 12.5Hz or 25Hz, semantically enhanced with an SSL feature, extracting discrete tokens for efficient speech generation.
  • Metis (models/tts/metis), a foundation model for unified speech generation supporting zero-shot text-to-speech, voice conversion, target speaker extraction, speech enhancement and lip-to-speech.
  • Vevo1.5 (models/svc/vevosing/README.md), which extends Vevo for unified and controllable generation of both speech and singing voice.
  • Bundled vocoders and evaluation metrics, so audio quality and scoring stay consistent across tasks.
  • Large-scale dataset work, including the Emilia-Large dataset with over 200,000 hours of data.

Who uses it and how

  • Junior researchers and engineers entering audio, music and speech generation research and development, using the architecture visualisations as the entry point.
  • Research groups that need reproducible experiments and consistent evaluation metrics across several generation tasks rather than one metric per paper.
  • Teams building speech-synthesis datasets at scale, using the project's large-scale dataset work such as Emilia-Large.
  • Singing voice and voice conversion practitioners, who can work with Vevo2 and Vevo1.5 for melody control and singing style conversion.
  • Developers needing zero-shot text-to-speech, speech enhancement or lip-to-speech, who can adopt Metis as a foundation model.

Getting started

Clone the open-mmlab/Amphion repository and follow the per-model README files, such as models/tts/metis, models/svc/vevo2/README.md, models/svc/vevosing/README.md and models/codec/dualcodec/README.md, with pretrained checkpoints published on Hugging Face as amphion/metis and RMSnow/Vevo2. DualCodec is additionally runnable from a hosted Colab notebook, and Vevo2 has a demo page at versasinger.github.io.

How it compares

No paid products or comparable third-party toolkits are named in the facts for this page. The model names that appear as topics on the project, such as FastSpeech2, NaturalSpeech2, MaskGCT and AudioLDM, are tags on Amphion rather than separate alternatives to it, so it stands alone in this registry entry.

When to use it — and when not to

A self-hoster must operate a Python environment and work through each model's own README, and training or fine-tuning models such as Metis or Vevo2 means supplying the compute and managing large datasets such as Emilia-Large independently; no pip package, Docker image or compose file is given in the facts, so there is no single-command deployment path. Teams that want a turnkey hosted API, a support contract or a managed service should not pick this project, and Text to Music is not yet supported. The repository also carries 174 open issues, its registry category of Business Software / Compliance & Risk Management does not describe an audio generation toolkit, and the tagline is truncated in the listing, so readers should go to the project documentation rather than rely on this page alone.

project readme (upstream, from github) — read inline

Amphion: An Open-Source Audio, Music, and Speech Generation Toolkit

open-mmlab%2FAmphion | Trendshift


Amphion (/æmˈfaɪən/) is a toolkit for Audio, Music, and Speech Generation. Its purpose is to support reproducible research and help junior researchers and engineers get started in the field of audio, music, and speech generation research and development. Amphion offers a unique feature: visualizations of classic models or architectures. We believe that these visualizations are beneficial for junior researchers and engineers who wish to gain a better understanding of the model.

The North-Star objective of Amphion is to offer a platform for studying the conversion of any inputs into audio. Amphion is designed to support individual generation tasks, including but not limited to,

  • TTS: Text to Speech (⛳ supported)
  • SVS: Singing Voice Synthesis (⛳ supported)
  • VC: Voice Conversion (⛳ supported)
  • AC: Accent Conversion (⛳ supported)
  • SVC: Singing Voice Conversion (⛳ supported)
  • TTA: Text to Audio (⛳ supported)
  • TTM: Text to Music (👨‍💻 developing)
  • more…

In addition to the specific generation tasks, Amphion includes several vocoders and evaluation metrics. A vocoder is an important module for producing high-quality audio signals, while evaluation metrics are critical for ensuring consistent metrics in generation tasks. Moreover, Amphion is dedicated to advancing audio generation in real-world applications, such as building large-scale datasets for speech synthesis.

🚀 News

  • 2026/03/25: We release Vevo2, a unified and controllable framework for speech and singing voice generation. Vevo2 bridges controllable speech and singing voice generation via unified prosody learning, supporting TTS, SVS, VC, SVC, Singing Voice Editing, Singing Style Conversion, Melody Control, and more. arXiv hf WebPage readme
  • 2025/05/26: We release DualCodec, a low-frame-rate (12.5Hz or 25Hz), semantically-enhanced (with SSL feature) Neural Audio Codec designed to extract discrete tokens for efficient speech generation.paperOpen In Colabdemo pagecode
  • 2025/04/12: We release Vevo1.5, which extends Vevo and focuses on unified and controllable generation for both speech and singing voice. Vevo1.5 can be applied into a series of speech and singing voice generation tasks, including VC, TTS, AC, SVS, SVC, Speech/Singing Voice Editing, Singing Style Conversion, and more. blog
  • 2025/02/26: We release Metis, a foundation model for unified speech generation. The system supports zero-shot text-to-speech, voice conversion, target speaker extraction, speech enhancement, and lip-to-speech. arXiv hf
  • 2025/02/26: The Emilia-Large dataset, featuring over 200,000 hours of data, is now available!!! Emilia-Large combines the original 101k-hour Emilia dataset (licensed under CC BY-NC 4.0) with the brand-new 114k-hour Emilia-YODAS dataset (licensed under CC BY 4.0). Download at hf. Check details at arXiv.
  • 2025/01/30: We release Amphion v0.2 Technical Report, which provides a comprehensive overview of the Amphion updates in 2024. arXiv
  • 2025/01/23: MaskGCT and Vevo got accepted by ICLR 2025! 🎉
  • 2024/12/22: We release the reproduction of Vevo, a zero-shot voice imitation framework with controllable timbre and style. Vevo can be applied into a series of speech generation tasks, including VC, TTS, AC, and more. The released pre-trained models are trained on Emilia dataset and achieve SOTA zero-shot VC performance. arXiv hf WebPage readme
  • 2024/10/19: We release MaskGCT, a fully non-autoregressive TTS model that eliminates the need for explicit alignment information between text and speech supervision. MaskGCT is trained on Emilia dataset and achieves SOTA zero-shot TTS performance. arXiv hf hf ModelScope ModelScope readme
  • 2024/09/01: Amphion, Emilia and DSFF-SVC got accepted by IEEE SLT 2024! 🤗
  • 2024/08/28: Welcome to join Amphion's Discord channel to stay connected and engage with our community！
  • 2024/08/20: SingVisio got accepted by Computers & Graphics, available here! 🎉
  • 2024/08/27: The Emilia dataset is now publicly available! Discover the most extensive and diverse speech generation dataset with 101k hours of in-the-wild speech data now at hf or OpenDataLab! 👑👑👑
  • 2024/07/01: Amphion now releases Emilia, the first open-source multilingual in-the-wild dataset for speech generation with over 101k hours of speech data, and the Emilia-Pipe, the first open-source preprocessing pipeline designed to transform in-the-wild speech data into high-quality training data with annotations for speech generation! arXiv hf demo readme
  • 2024/03/12: Amphion now support NaturalSpeech3 FACodec and release pretrained checkpoints. arXiv hf hf readme
  • 2024/02/22: The first Amphion visualization tool, SingVisio, release. arXiv openxlab Video readme
  • 2023/12/18: Amphion v0.1 release. arXiv hf youtube readme
  • 2023/11/28: Amphion alpha release. readme

⭐ Key Features

TTS: Text to Speech

  • Amphion achieves state-of-the-art performance compared to existing open-source repositories on text-to-speech (TTS) systems. It supports the following models or architectures:
    • FastSpeech2: A non-autoregressive TTS architecture that utilizes feed-forward Transformer blocks. code
    • VITS: An end-to-end TTS architecture that utilizes conditional variational autoencoder with adversarial learning code
    • VALL-E: A zero-shot TTS architecture that uses a neural codec language model with discrete codes. code
    • NaturalSpeech2: An architecture for TTS that utilizes a latent diffusion model to generate natural-sounding voices. code
    • Jets: An end-to-end TTS model that jointly trains FastSpeech2 and HiFi-GAN with an alignment module. code
    • MaskGCT: A fully non-autoregressive TTS architecture that eliminates the need for explicit alignment information between text and speech supervision. code
    • Vevo-TTS: A zero-shot TTS architecture with controllable timbre and style. It consists of an autoregressive transformer and a flow-matching transformer. code
    • DualCodec-VALLE: A VALLE model trained on 12.5Hz DualCodec tokens for super fast generation.

VC: Voice Conversion

Amphion supports the following voice conversion models:

  • Vevo: A zero-shot voice imitation framework with controllable timbre and style. Vevo-Timbre conducts the style-preserved voice conversion, and Vevo-Voice conducts the style-converted voice conversion. code
  • FACodec: FACodec decomposes speech into subspaces representing different attributes like content, prosody, and timbre. It can achieve zero-shot voice conversion. code
  • Noro: A noise-robust zero-shot voice conversion system. Noro introduces innovative components tailored for VC using noisy reference speeches, including a dual-branch reference encoding module and a noise-agnostic contrastive speaker loss. code

Neural Audio Codec

  • DualCodec, a low-frame-rate (12.5Hz or 25Hz), semantically-enhanced (with SSL feature) Neural Audio Codec designed to extract discrete tokens for efficient speech generation.paperOpen In Colabdemo pagecode
  • FACodec: FACodec decomposes speech into subspaces representing different attributes like content, prosody, and timbre. code

AC: Accent Conversion

  • Amphion supports AC with Vevo-Style. Particularly, it can conduct the accent conversion in a zero-shot manner. code

SVC: Singing Voice Conversion

  • Vevo2: A unified and controllable framework for speech and singing voice generation. Vevo2 supports style-preserved VC/SVC, style-converted VC/SVC, TTS, SVS, Singing Voice Editing, Singing Style Conversion, Melody Control, and more. arXiv code
  • Vevo1.5: A versatile zero-shot voice imitation framework for unified speech and singing voice generation, supporting VC, TTS, AC, SVS, SVC, Editing, Singing Style Conversion, and more. code
  • Ampion supports multiple content-based features from various pretrained models, including WeNet, Whisper, and ContentVec. Their specific roles in SVC has been investigated in our SLT 2024 paper. arXiv code
  • Amphion implements several state-of-the-art model architectures, including diffusion-, transformer-, VAE- and flow-based models. The diffusion-based architecture uses Bidirectional dilated CNN as a backend and supports several sampling algorithms such as DDPM, DDIM, and PNDM. Additionally, it supports single-step inference based on the Consistency Model. code

TTA: Text to Audio

  • Amphion supports the TTA with a latent diffusion model. It is designed like AudioLDM, Make-an-Audio, and AUDIT. It is also the official implementation of the text-to-audio generation part of our NeurIPS 2023 paper. arXiv code

Vocoder

Evaluation

Amphion provides a comprehensive objective evaluation of the generated audio. code

The supported evaluation metrics contain:

  • F0 Modeling: F0 Pearson Coefficients, F0 Periodicity Root Mean Square Error, F0 Root Mean Square Error, Voiced/Unvoiced F1 Score, etc.
  • Energy Modeling: Energy Root Mean Square Error, Energy Pearson Coefficients, etc.
  • Intelligibility: Character/Word Error Rate, which can be calculated based on Whisper and more.
  • Spectrogram Distortion: Frechet Audio Distance (FAD), Mel Cepstral Distortion (MCD), Multi-Resolution STFT Distance (MSTFT), Perceptual Evaluation of Speech Quality (PESQ), Short Time Objective Intelligibility (STOI), etc.
  • Speaker Similarity: Cosine similarity, which can be calculated based on RawNet3, Resemblyzer, WeSpeaker, WavLM and more.

Datasets

Visualization

Amphion provides visualization tools to interactively illustrate the internal processing mechanism of classic models. This provides an invaluable resource for educational purposes and for facilitating understandable research.

Currently, Amphion supports SingVisio, a visualization tool of the diffusion model for singing voice conversion. arXiv openxlab Video

📀 Installation

Amphion can be installed through either Setup Installer or Docker Image.

Setup Installer

git clone https://github.com/open-mmlab/Amphion.git
cd Amphion

# Install Python Environment
conda create --name amphion python=3.9.15
conda activate amphion

# Install Python Packages Dependencies
sh env.sh

Docker Image

  1. Install Docker, NVIDIA Driver, NVIDIA Container Toolkit, and CUDA.

  2. Run the following commands:

git clone https://github.com/open-mmlab/Amphion.git
cd Amphion

docker pull realamphion/amphion
docker run --runtime=nvidia --gpus all -it -v .:/app realamphion/amphion

Mount dataset by argument -v is necessary when using Docker. Please refer to Mount dataset in Docker container and Docker Docs for more details.

🐍 Usage in Python

We detail the instructions of different tasks in the following recipes:

👨‍💻 Contributing

We appreciate all contributions to improve Amphion. Please refer to CONTRIBUTING.md for the contributing guideline.

🙏 Acknowledgement

©️ License

Amphion is under the MIT License. It is free for both research and commercial use cases.

📚 Citations

Amphion v0.2:

@article{amphion_v0.2,
  title        = {Overview of the Amphion Toolkit (v0.2)},
  author       = {Jiaqi Li and Xueyao Zhang and Yuancheng Wang and Haorui He and Chaoren Wang and Li Wang and Huan Liao and Junyi Ao and Zeyu Xie and Yiqiao Huang and Junan Zhang and Zhizheng Wu},
  year         = {2025},
  journal      = {arXiv preprint arXiv:2501.15442},
}

Amphion v0.1: 

@inproceedings{amphion,
    author={Xueyao Zhang and Liumeng Xue and Yicheng Gu and Yuancheng Wang and Jiaqi Li and Haorui He and Chaoren Wang and Ting Song and Xi Chen and Zihao Fang and Haopeng Chen and Junan Zhang and Tze Ying Tang and Lexiao Zou and Mingxuan Wang and Jun Han and Kai Chen and Haizhou Li and Zhizheng Wu},
    title={Amphion: An Open-Source Audio, Music and Speech Generation Toolkit},
    booktitle={{IEEE} Spoken Language Technology Workshop, {SLT} 2024},
    year={2024}
}

Frequently asked questions

Is Amphion free to use?

Amphion is open source under the MIT licence. There is no licence fee and no seat count — you can self-host it or, where the project offers one, pay a vendor for a managed version instead.

What does Amphion do?

Amphion (/æmˈfaɪən/) is a toolkit for Audio, Music, and Speech Generation. Its purpose is to support reproducible research and help junior researchers and engin

What is Amphion written in?

Amphion is primarily written in Python. Its source is publicly available at https://github.com/open-mmlab/Amphion, and it has 10,329 GitHub stars.