Inference from Raw Input about diffsinger HOT 5 CLOSED

The pretrained model provided by the original DiffSinger repo is Chinese-only and cannot sing English. Also as far as I know, the code of original DiffSinger is not compatible with languages like English, and is only suitable for two-phase phoneme systems like Chinese. Many things are hard-coded and cannot be changed easily :(.

from diffsinger.

Thanks for the reply. I assume English language will not be compatible with the fork you guys are maintaining as well right?
Moreover, do you know of any other tool which might be compatible with English and can generate singing given it the notes and notes duration aligned with lyrics. So far whatever i've tested seems to be heavily focused on either Chinese, Japanese or Korean.

from diffsinger.

This repo supports any language. You can find documentation for the making process.

from diffsinger.

Oh i see, a few things i would like some clarity on:

• Are there any test models available somewhere which i can use to test for inference
• I see from the sample .ds files that there are quite a few values in there: (offset, text, ph_seq, ph_dur, ph_num, note_seq, note_dur, note_slur, f0_seq, f0_timestep) suppose if i want to use it with English text will i need to use all the values mentioned, as i tried to align lyrics and i only had text, notes, and notes_duration from the original DiffSinger version. Also i assume all these values in .ds file within {} explain the tone and style for each lyrics line or chorus part
• And also can you shed some light on these variables and also on how will i go on extracting f0 for example
• Whats the main difference between the acoustic and variance model?

I am planning to use all of this via command line so that's why i'm asking all of this stuff. Thanks in advance for the help!

from diffsinger.

1. There are no test models, you need to train by yourself, or you can ask for people from the English voicebank developing community.
2. offset: the start position in seconds of each segment; text: not in use now; ph_seq: phone sequence; ph_dur: phone duration sequence in seconds; ph_num: the number of phones of each word/syllable (each word/syllable usually starts with an onset vowel); note_seq: note name sequence; note_dur: note duration sequence in seconds; note_slur: whether each note is a slur (1) or not (0) (notes that share the same word/syllable with other notes are slurs); f0_seq: F0 sequence, in Hz; f0_timestep: the interval between two neighbor F0 curve points. For the alignment method see https://github.com/openvpi/MakeDiffSinger/tree/main/variance-temp-solution#4-estimate-note-values.
3. There is a method in utils/binarizer_utils.py called get_pitch_parselmouth that you can use to extract F0.
4. Difference of the two models is shown in the image in README.

from diffsinger.