Pheme Model

This repo contains recipes and models used for training Pheme TTS models. It is the official implementation for the paper: Pheme: Efficient and Conversational Speech Generation. Demo is available here, while a selection of audio samples can be found here.

Our Pheme TTS framework validates several hypotheses:

1. We can train Transformer-based conversational TTS models with much fewer training data than e.g., VALL-E or SoundStorm (e.g., 10x fewer data).
2. Training can be performed with conversational, podcast, and noisy data like GigaSpeech.
3. Efficiency is paramount, which includes parameter efficiency (compact models), data efficiency (fewer training data) and inference efficiency (reduced latency).
4. One fundamental ingredient is the separation of semantic and acoustic tokens and the adequate speech tokenizer.
5. Inference can be run parallelly through MaskGit-style inference with 15x speed-ups compared to similarly sized autoregressive models.
6. The single-speaker quality can be improved through student-teacher training with (synthetic) data generated by third-party providers.

Set Up the Environment

Set up conda environment:

conda create --name pheme3 python=3.10
conda activate pheme3

pip3 install torch torchvision torchaudio
pip3 install -r requirements.txt --no-deps

Download pre-trained SpeechTokenizer and unique token list models:

st_dir="ckpt/speechtokenizer/"
mkdir -p ${st_dir}
cd ${st_dir}
wget "https://huggingface.co/fnlp/SpeechTokenizer/resolve/main/speechtokenizer_hubert_avg/SpeechTokenizer.pt"
wget "https://huggingface.co/fnlp/SpeechTokenizer/resolve/main/speechtokenizer_hubert_avg/config.json" 
cd ..
wget "https://huggingface.co/fnlp/USLM/resolve/main/USLM_libritts/unique_text_tokens.k2symbols" 

You need to create an access token to use the speaker embedding of pyannote.

export HUGGING_FACE_HUB_TOKEN=YOUR_PRIVATE_TOKEN

Download pre-trained T2S and S2A models (the 100M Pheme variant):