About video frames and labels correspondence about ego4d_talknet_asd HOT 6 CLOSED

Hello,

Ego4D does not provide the same labels as AVA Active Speaker Detection dataset. Active speaker labels used here are generated from the VAD labels. The speaker who is speaking and has a visible face will be labeled as active speaker. But this may differ from AVA's annotation guideline. VAD results do not always align with the faces in the field of view. That's why we interpolate the faces.

These labels are also not strictly correct because we cannot verify each frame manually. Our motivation is to validate whether training on Ego4D dataset can improve the final diarization results for Ego4D AVD becnhmark.

from ego4d_talknet_asd.

Sorry for late reply! And thank you for the quick response! Sorry I might cause some confusions in the question. After reading your code, I found that in some cases, your dataloader will load the videos of one clip but use annotations of another clip.

For this video clip with trackid : a1055434-9e9b-4d69-bac3-374a39f801da:track_85:0, the video clip that your dataloader loads (including interpolation) is of frame 1619-1694. And the labels of this clip that the dataloader uses belong to frame 1619-1677 and frame 1694-1711. There is a difference between video frames and labels.

from ego4d_talknet_asd.

Hello, I'm not sure what is the reason for this inconsistency. Is this the error from code or the preprocessed labels?

from ego4d_talknet_asd.

Hey! The preprocessed labels in the annotated bbox json file have inconsistent timestamps for some video clips. I don't think it's an error. But in the dataloader code it seems to assume that the timestamps are consistent. Thanks!

from ego4d_talknet_asd.

Hey! The preprocessed labels in the annotated bbox json file have inconsistent timestamps for some video clips. I don't think it's an error. But in the dataloader code it seems to assume that the timestamps are consistent. Thanks!

There are errors in the preprocessed labels. These are actually generated by ourselves cuz ego4d doesn't provide labels for active speaker detection. If this issue has side effect on your research, my suggestion is to regenerate these labels for ASD.

from ego4d_talknet_asd.

Hi! Thanks for your quick response and suggestion! I will check their paper and try to regenerate the labels.

from ego4d_talknet_asd.