Questions about the transformer about everything_at_once HOT 3 CLOSED

Hi! We used one transformer to enable fusion between modalities, but in our ablation, we also considered using different transformers for different modalities (Table 4 “single-mod. transformer per mod”). The config for this experiment is here. For t->va retrieval, we found it beneficial to use one transformer. With different transformers for different modalities, video and audio modalities are processed separately in corresponding transformers, and then video and audio embeddings are just summed up to obtain a video-audio representation. However, when we use one transformer (which is trained on all modality combinations) we forward tokens from both video and audio modalities together, allowing them to attend to each other, and therefore we obtain a fused video-audio representation.

from everything_at_once.

thx~

from everything_at_once.

I have tried your method on Ads risk recall situation, and got about 1% boost🎉. We use a lot of modals, so this is still more troublesome. Hope you have more and better solutions in the future～

from everything_at_once.