Training data structure about dancing2music HOT 4 OPEN

@HsiaoYingLu Hi, Did you know how to prepare the dataset now?

from dancing2music.

@zhuhaozh Unfortunately no. Without raw data and data format clarification, I cannot decode how the author prepare the dataset. Also, whether aud.npy and unit.npy with the same filename are correspondent still remains unknown.

from dancing2music.

Hi @HsinYingLee

Thanks for the demo code update.

I still have some questions:

1. I found that in composition training, only the first 30 value in *_fps.npy file would be used. Then, what are the purposes of other .npy files in aud folder?

2. I see that there are different numbers of .npy files in folders aud, unit, unitseq3, unitseq4. What are the correspondent relations between these .npy files in the folders above?

3. In composition training phase, it is always trained on pairs consist of 3 dance units and 30-dim audio features. Hence, in testing phase, 1 music clip is correspondent to 3 generated dance units, which makes the length of output video 3 times longer than input audio. How should I fix this problem?

4. In order the fix the length problem, I have tried to use the modulate function, but the generated video turned out to be like a fast-forward one for the first few second. Any thought on this?

Any thought will be helpful, thanks!

from dancing2music.

Hi @HsinYingLee,

I was wondering the same thing. Could you be so kind to elaborate? Thank you so much!

from dancing2music.