For hand gestures, add "token_type_ids" about multimodal-depression-from-video HOT 1 CLOSED

As we have 21 landmarks of 5 coordinates for each N frame of each W window when dealing with hand landmarks:

• Original Shape: (W, N, 2, 21, 5)
• Input Transformer Shape: (N, embed_size)

So, we need a linear projection layer to transform the dimension of each modality to the embed_size of the Transformer. I am a bit confused how to preserve the distinction between both hands. Because the order in the forward pass would be:

1. x = linear_proj(x)
2. x += pos_emb
3. x += modality_emb
4. x[:, :, :embed_size//2] += left_hand_type_emb & x[:, :, embed_size//2:] += right_hand_type_emb
5. x = transformer(x)

The other option is feeding the linear_proj with a reshaped tensor of (W, N, 2, 21*5), but then we have other problems when reshaping to the shape it should be.

This is for next week. See you tomorrow :)

from multimodal-depression-from-video.