download model_ckpt_steps_2168000.ckpt, config.yaml, from https://drive.google.com/drive/folders/1n_0tROauyiAYGUDbmoQ__eqyT_G4RvjN?usp=sharing to pretrained/hifigan_hifitts
Data Preprocess
# The default dataset is ``vctk``.
python data_gen/tts/base_preprocess.py
python data_gen/tts/run_mfa_train_align.sh
python data_gen/tts/base_binarizer.py
Train
# Example run for Fluenteditor.
CUDA_VISIBLE_DEVICES=0 python tasks/run.py --dir /path/to/your/fluenteditor --config egs/fluenteditor.yaml --exp_name fluenteditor --reset
Inference
We provide the data structure of inference in inference/example.csv. text and edited_text refer to the original text and target text. region refers to the word idx range (start from 1 ) that you want to edit. edited_region refers to the word idx range of the edited_text.
| id | item_name | text | edited_text| wav_fn_orig | edited_region| region|
| -- | -- | -- | -- | -- | -- | -- |
| 0 | 1 | "I'd love to be at the world cup." | "I'd <mark>absolutely</mark> love to be at the world cup." | inference/audio_example/1.wav | [1,3] | [1,2] |
# run with one example
python inference/tts/fluenteditor.py --exp_name fluenteditor
Evaluation
# Example Objective Evaluation for Fluenteditor.
# You can use the following objective evaluation metrics: MCD, STOI, PESQ
python eval/get_metrics.py
License and Agreement
Any organization or individual is prohibited from using any technology mentioned in this paper to generate someone's speech without his/her consent, including but not limited to government leaders, political figures, and celebrities. If you do not comply with this item, you could be in violation of copyright laws.
Tips
If you find the mfa_dict.txt, mfa_model.zip, phone_set.json, or word_set.json are missing in inference, you need to run the preprocess script in our repo to get them. You can also download all of these files you need for inferencing the pre-trained model from
https://drive.google.com/drive/folders/1BOFQ0j2j6nsPqfUlG8ot9I-xvNGmwgPK?usp=sharing and put them in data/processed/vctk.
Please specify the MFA version as 2.0.0rc3.
Citing
To cite this repository:
@inproceedings{liu24p_interspeech,
title = {FluentEditor: Text-based Speech Editing by Considering Acoustic and Prosody Consistency},
author = {Rui Liu and Jiatian Xi and Ziyue Jiang and Haizhou Li},
year = {2024},
booktitle = {Interspeech 2024},
pages = {3435--3439},
doi = {10.21437/Interspeech.2024-2320} ,
issn = {2958-1796},
}
Author
E-mail：liurui_imu@163.com, xjiatian@163.com
Ecosystem Role
Standard MoltPulse indexed agent.
Embed Badge
Show off your Pulse Score in your GitHub README to build trust and rank higher.