taewookim / keras-syncnet Goto Github PK

Hello,
Thank you for the excellent work and publicly available code.
I am using syncnet to find if there is lip-sync error in the video. I am getting very random values of AV offset and confidence, after using the train weights available on official website.

I am confused about this paragraph from the paper -

Determining the lip-sync error -
To find the time offset between the audio and the video, we take a sliding-window
approach. For each sample, the distance is computed between one 5-frame video
feature and all audio features in the ± 1 second range. The correct offset is when
this distance is at a minimum. However as Table 2 suggests, not all samples in
a clip are discriminative (for example, there may be samples in which nothing
is being said at that particular time), therefore multiple samples are taken for
each clip, and then averaged.

I am missing something in this paragraph. How do I collect multiple samples for each clip?
I would like to know how to get a proper value of metric (AV offset, Confidence) that show the out of sync of video and audio on sample.

Thank you

Hi,thank you for the excellent work and publicly available code.
have you trained the syncnet? I'm trying to use the mvlrs_v1 datasets to train the Syncnet,but the loss keep oscillating.
Since most video in the mvlrs_v1 is short,I randomly shift the audio up to 25-frame in order to generate synthetic false audio-video pairs.
1、Is each batch train data generated by one video or multiple videos ?
2、Does the false pairs have to be shifted by up to 2s?
3、Can you show me your script to generate the train datasets？
thank you!