how to train for mandarin asr? about auxiliaryasr HOT 8 CLOSED

You shouldn't put hanzi characters in train.txt, and it will be better if you transform the pinyin sequences into phoneme sequences as in Mandarin TTS projects (e.g. kan4-> k an4) because there are too many unique pinyins in Mandarin.

from auxiliaryasr.

I believe it should be like SSB06930002.wav | wu shu shi zhong bei kan zuo wo guo de guo cui | 0. You don't want the tones (that should be handled by the F0 model) nor the Chinese characters (they aren't phonemes) here. The ASR model should only be used for phonemes.

from auxiliaryasr.

Thanks ,it helps a lot!!! I am still confused about dict.txt for mandarin.my fomart is like this:
"a",1
"ao",2
"an",3
"w",4
"t",5
"d",6
"n",7
or：
“wa”,1
"ta",2
"na",3
"tao",4
"dao",5
"wan",6
"tan",7
which one should i use? @yl4579 @Charlottecuc

from auxiliaryasr.

I use g2pM as phonemizer,

sentence = "然而，他红了20年以后，他竟退出了大家的视线。"
model(sentence, tone=False, char_split=False)
['ran', 'er', '，', 'ta', 'hong', 'le', '2', '0', 'nian', 'yi', 'hou', '，', 'ta', 'jing', 'tui', 'chu', 'le', 'da', 'jia', 'de', 'shi', 'xian', '。']
so the dict.txt 's format should be like this?
“wa”,1
"ta",2
"na",3
"tao",4
"dao",5
"wan",6
"tan",7
@yl4579 @Charlottecuc

from auxiliaryasr.

I use g2pM as phonemizer,

sentence = "然而，他红了20年以后，他竟退出了大家的视线。"
model(sentence, tone=False, char_split=False)
['ran', 'er', '，', 'ta', 'hong', 'le', '2', '0', 'nian', 'yi', 'hou', '，', 'ta', 'jing', 'tui', 'chu', 'le', 'da', 'jia', 'de', 'shi', 'xian', '。']
so the dict.txt 's format should be like this?
“wa”,1
"ta",2
"na",3
"tao",4
"dao",5
"wan",6
"tan",7
@yl4579 @Charlottecuc

Thanks ,it helps a lot!!! I am still confused about dict.txt for mandarin.my fomart is like this: "a",1 "ao",2 "an",3 "w",4 "t",5 "d",6 "n",7 or： “wa”,1 "ta",2 "na",3 "tao",4 "dao",5 "wan",6 "tan",7 which one should i use? @yl4579 @Charlottecuc

I use the first type of format (which is quite common when training Mandarin TTS models) because I don't want too many tokens in my dictionary . Besides, I suggest you transform "20" to "er sh i" in this context.

from auxiliaryasr.

I believe it should be like SSB06930002.wav | wu shu shi zhong bei kan zuo wo guo de guo cui | 0. You don't want the tones (that should be handled by the F0 model) nor the Chinese characters (they aren't phonemes) here. The ASR model should only be used for phonemes.

@yl4579 I would like to ask if tones information is not added to the asr model training, then how to align the text encoder and mel. For example: (你好 ->ni3 hao3 -> n i3 h ao3), (n i h ao).

from auxiliaryasr.

@Moonmore You do not need tones for this. The pre-trained aligner in StyleTTS and StyleTTS 2 repo was already trained on Mandarin (AiShell) without tones. The conversion table between IPA and pinyin is here: yl4579/StyleTTS#10 (comment)
Once you use this table to convert pinyin to IPA, you can directly use the pre-trained aligner and you don't need to train it yourself. However, if you want to train it yourself on pinyin, you do not need to include tones.

from auxiliaryasr.

@Moonmore You do not need tones for this. The pre-trained aligner in StyleTTS and StyleTTS 2 repo was already trained on Mandarin (AiShell) without tones. The conversion table between IPA and pinyin is here: yl4579/StyleTTS#10 (comment) Once you use this table to convert pinyin to IPA, you can directly use the pre-trained aligner and you don't need to train it yourself. However, if you want to train it yourself on pinyin, you do not need to include tones.

@yl4579 thanks for your reply. I successfully used pinyin to train the model.

from auxiliaryasr.