This project uses conda to manage all the dependencies, you should install anaconda if you have not done so.

# Clone the repo
git clone https://github.com/ttslr/MonTTS.git
cd $PROJECT_ROOT_DIR

conda env create -f Environment/environment.yaml

conda activate montts

Prepare our MonSpeech dataset in the following format:

|- MonSpeech/
|   |- metadata.csv
|   |- wavs/
|       |- file1.wav
|       |- ...

Where metadata.csv has the following format: id|transcription. This is a ljspeech-like format.

Here are some speech samples from our MonSpeech.

You can find MonSpeech in the Links section.

The preprocessing has two steps:

1. Preprocess audio features
   • Convert characters to IDs
   • Compute mel spectrograms
   • Normalize mel spectrograms to [-1, 1] range
   • Split the dataset into train and validation
   • Compute the mean and standard deviation of multiple features from the training split
2. Standardize mel spectrogram based on computed statistics

To reproduce the steps above:

tensorflow-tts-preprocess --rootdir /home/rui/MonSpeech  --outdir ./dump_mon --config preprocess/mon_preprocess.yaml --dataset mon

tensorflow-tts-normalize --rootdir ./dump_mon --outdir ./dump_mon --config preprocess/mon_preprocess.yaml --dataset mon

Based on the script train_fastspeech2.py.

This example code show you how to train MonTTS from scratch with Tensorflow 2 based on custom training loop and tf.function.

Here is an example command line to training MonTTS from scratch:

CUDA_VISIBLE_DEVICES=0 python examples/fastspeech2/train_fastspeech2.py \
  --train-dir ./dump_mon/train/ \
  --dev-dir ./dump_mon/valid/ \
  --outdir ./examples/fastspeech2/exp/train.fastspeech2-mon.v1/ \
  --config ./examples/fastspeech2/conf/fastspeech2.v1.yaml \
  --use-norm 1 \
  --f0-stat ./dump_mon/stats_f0.npy \
  --energy-stat ./dump_mon/stats_energy.npy \
  --mixed_precision 1 \
  --resume ""

IF you want to use MultiGPU to training you can replace CUDA_VISIBLE_DEVICES=0 by CUDA_VISIBLE_DEVICES=0,1,2,3 for example. You also need to tune the batch_size for each GPU (in config file) by yourself to maximize the performance. Note that MultiGPU now support for Training but not yet support for Decode.

In case you want to resume the training progress, please following below example command line:

--resume ./examples/fastspeech2/exp/train.fastspeech2-mon.v1/checkpoints/ckpt-100000

If you want to finetune a model, use --pretrained like this with your model filename

--pretrained pretrained.h5

First, you need training generator with only stft loss:

CUDA_VISIBLE_DEVICES=0 python examples/hifigan/train_hifigan.py \
  --train-dir ./dump_mon/train/ \
  --dev-dir ./dump_mon/valid/ \
  --outdir ./examples/hifigan/exp/train.hifigan-mon.v1/ \
  --config ./examples/hifigan/conf/hifigan.v1.yaml \
  --use-norm 1 \
  --generator_mixed_precision 1 \
  --resume ""

Then resume and start training generator + discriminator:

CUDA_VISIBLE_DEVICES=0 python examples/hifigan/train_hifigan.py \
  --train-dir ./dump_mon/train/ \
  --dev-dir ./dump_mon/valid/ \
  --outdir ./examples/hifigan/exp/train.hifigan-mon.v1/ \
  --config ./examples/hifigan/conf/hifigan.v1.yaml \
  --use-norm 1 \
  --resume ./examples/hifigan/exp/train.hifigan-mon.v1/checkpoints/ckpt-100000

You should find a dir log in all of your output dirs, that is the LOG_DIR you should use below.

tensorboard --logdir=${LOG_DIR}

For example, you can follow below example command lines to access the tensrobords to check the training progress:

tensorboard --logdir examples/fastspeech2/exp/train.fastspeech2-mon-exp1

tensorboard --logdir examples/hifigan/exp/train.hifigan-mon.v1

You can follow below example command line to generate synthesized speeh for given text in 'dump_mon/inference.txt' using Griffin-Lim and trained HiFi-GAN vocoder:

CUDA_VISIBLE_DEVICES=1 python examples/fastspeech2/inference_fastspeech2-mon.py \
    --outdir prediction/mon_inference_fastspeech2 \
    --infile dump_mon/inference.txt  \
    --tts_ckpt examples/fastspeech2/exp/train.fastspeech2-mon.v1/checkpoints/model-200000.h5 \
    --vocoder_ckpt  examples/hifigan/exp/train.hifigan-mon.v1/checkpoints/generator-420000.h5 \
    --stats_path dump_mon/stats.npy \
    --dataset_config preprocess/mon_preprocess.yaml \
    --tts_config examples/fastspeech2/conf/fastspeech2.v1.yaml \
    --vocoder_config examples/hifigan/conf/hifigan.v1.yaml \
    --lan_json dump_mon/mon_mapper.json 

You can find pre-trained models in the Links section.

The synthesized speech will save to prediction/mon_inference_fastspeech2 folder.

You can also run demo_server.py to build a online demo.

python examples/fastspeech2/demo_server.py

Note that you need to point your browser at localhost:9000 and then type what you want to synthesize.

• Please contact the Author to access the MonSpeech corpus and Pre-trained models
• Demo: link to synthesized audio samples

Rui Liu
E-mail: [email protected]

Please kindly cite the following paper if you use this code repository in your work,

 @inproceedings{liu2021montts,
  title={MonTTS: A Real-time and High-fidelity Mongolian TTS Model with Complete Non-autoregressive Mechanism (in Chinese)},
  author={Rui, Liu and Shiyin, Kang and Jingdong, Li amd Feilong, Bao and Guanglai, Gao},
  booktitle={JOURNAL OF CHINESE INFORMATION PROCESSING (中文信息学报)},
  year={2021}
}

Tensorflow-TTS: https://github.com/TensorSpeech/TensorFlowTTS

This work is released under MIT License.