We need the evaluate.py script to load trained models and evaluate them on the test data.

We have some file with missing things from each modality. Read the file, get the ids that we sampled and construct an attention mask. Use the attention mask in the perceiver.

We're processing sequences in order. Use nn.Embedding() to map positions (i.e. numbers between 0 and sequence_length) to a vector. Add each vector to each corresponding latent vector.

Each hand should have a different embedding to differentiate between them. The embedding (nn.Embedding(...)) should be added to the projected coordinates. Similar to "token_type_ids" in huggingface transformers library.

Thanks for your great work! Can share the implement of the Attribution scores as shown in Fig 5? Thanks!

Read the new modalities and implement appropriate encoders for each one

Evaluating the baseline model with temporal evaluator only takes the final predictions into account.

Include the mean prediction over the whole video in TemporalEvaluator. Add a line to the .csv results file with kind = 'mean'. Add kind = 'last' to the current line.

Take windows in order, process all of them in order (with perceiver or not?). Take the majority vote or the last prediction.

We can use this to visualize the depression score over time, make nice plots for the paper.

After each modality is encoded, let's try to implement a simple baseline: just a simple transformer encoder that encodes all modalities and does the classification per window.

Evaluation is majority vote.

HI @david-gimeno.. I have downloaded all the videos from the D-vlog dataset.. should i split it based on the ids given in the test, train and validation.csv file?? or there is a separate file python3 ./scripts/feature_extraction/dvlog/extract_wavs.py --csv-path ./data/D-vlog/video_ids.csv --column-video-id video_id --video-dir $VIDEO_DIR --dest-dir $WAV_DIR video_id.csv which is missing in the repo?? please help me with this..

• Implement Time-Unaware Perceiver Architecture
• Discuss when modality embeddings are needed
• Discuss details & Implement Time-Aware Perceiver Architecture

If there are windows with no modality present, don't feed them into the model

Do a polar heatmap with the gaze angles (pitch / yaw).

Maybe do some statistics, find out what.

Use Poisson regression to compare between groups.

Plots with std and mean of blinking interval per video (binned by seconds), colored by group.

Do some statistical test and compute p-value.

Differentiate between modalities. Use nn.Embedding() to map a modality id to a vector representation. Add the vector to each latent vector in the corresponding modality.

Right now we concatenate everything. We need to process chunks of the video into windows. Maybe use torch.split()?

Decide on how to normalize the landmarks.

Use a batch-norm for the last dimension, probably, before applying the projection.

It's kind of wierd that we have different positional encodings for each modality. We should probably have the same ones for all of them. But how do we take into account the different framerates?

Maybe some framerate-aware positional encoding? (i.e. don't use learned positional encodings, but "classical" sin/cos P.E.)

Something like this (Fractional positional encoding): https://opus.bibliothek.uni-augsburg.de/opus4/frontdoor/deliver/index/docId/96742/file/96742.pdf

Repository: https://github.com/philm5/fpe-vtt/blob/master/vtt_transformer.py#L85

The implementation is in tensorflow, we need to convert it to torch (use ChatGPT?).

• Exploring different schedulers & learning rates for the baseline model
• Check if the perceiver is learning something, check if the same batch size works for it
• Ablation study on number of windows for perceiver, check if it works
• Small ablation on presence_threshold, maybe lower is better (more data), or maybe higher is better (more quality)
• Ablation on the modalities. Which combination works better?

Take N random windows from the video, make a prediction on all of them, take the majority prediction.

Hello, can you please help me with what value I should set for the variable in each file ?? for example PASE_CONFIG=
PASE_CHCK_PATH=
USE_AUTH_TOKEN= what should be the value for this??

Use this: https://github.com/hysts/pytorch_mpiigaze_demo to extract gaze for each frame of each video.

Gaze is a 3-tuple containing (angle_x, angle_y, angle_z).

If no gaze can be estimated, put (-1, -1, -1) so we can compute a mask.

As we have 21 landmarks of 5 coordinates for each N frame of each W window when dealing with hand landmarks:

• Original Shape: (W, N, 2, 21, 5)
• Input Transformer Shape: (N, embed_size)

So, we need a linear projection layer to transform the dimension of each modality to the embed_size of the Transformer. I am a bit confused how to preserve the distinction between both hands. Because the order in the forward pass would be:

1. x = linear_proj(x)
2. x += pos_emb
3. x += modality_emb
4. x[:, :, :embed_size//2] += left_hand_type_emb & x[:, :, embed_size//2:] += right_hand_type_emb
5. x = transformer(x)

The other option is feeding the linear_proj with a reshaped tensor of (W, N, 2, 21*5), but then we have other problems when reshaping to the shape it should be.

This is for next week. See you tomorrow :)

If we want ECIR, we need to add another dataset with some other particularity.

• Pre-process DAIC-WOZ dataset + Implement specific dataloader
• Pre-process Extended DAIC-WOZ dataset + Implement specific dataloader
• Experiments DAIC-WOZ
• Experiments E-DAIC-WOZ

Use this: https://github.com/wenzhengzeng/MPEblink

Extract blinking for each frame, for each video.

Blinking patterns = a list of 0s and 1s.

0 = not blinking
1 = blinking

If the eyes are not visible, put a -1 so we can compute a mask.

Re-run some experiments to check if it works better.