hey,
Two questions about the training data, hope you can help. Thanks in advance.

1. the Recola dataset that I downloaded doesn't include the same amount data that you have. Did you try to include additional data that collected by yourself?

In your code, you have:
train = [25, 15, 16 ,17 ,18 ,21 ,23 ,37 ,39 ,41 ,46 ,50 ,51 ,55 ,56, 60], # 25
valid = [14, 19, 24, 26, 28, 30, 34 ,40, 42, 43, 44, 45, 52, 64, 65],
test = [54, 53, 13, 20, 22, 32, 38, 47, 48, 49, 57, 58, 59, 62, 63] # 54, 53
What I got are:
Train = [ 16,21,23,25,37,39,41,46,56]
Valid = [19,26,28,30,34,42,45,64,65]
Test=[13,20,32,38,47,49,53,59,63]

Did I miss anything here?

2. Annotated data - are you using the annotation from the Recola database? Or the annotation has been post-processed by you?
   In your code, you have:
   def get_samples(subject_id):
   arousal_label_path = root_dir / 'Ratings_affective_behaviour_CCC_centred/arousal/{}.csv'.format(subject_id)
   valence_label_path = root_dir / 'Ratings_affective_behaviour_CCC_centred/valence/{}.csv'.format(subject_id)
   arousal = np.loadtxt(str(arousal_label_path), delimiter=',')[:, 1][1:] #this only reads the annotation from the 1st external annotator.

but what I got is something like /recola/AVEC_2016/ratings_individual/arousal/dev_1.csv, the format in the csv file is something as below:
time | FM1 | FM2 | FM3 | FF1 | FF2 | FF3
0 | 0 | 0 | 0 | 0 | 0 | 0
arousal = np.loadtxt(str(arousal_label_path), delimiter=',')[:, 1][1:] #this only reads the annotation from the 1st external annotator.

Greetings, I am very interested in your research, I would like to know please how figure 2 is drawn and how it should be analyzed, thank you

hi how can i solve it?
im new here

Hi,
I was trying to understand the code. If I am not wrong, in models.py, in 'audio model', is the network having first layer as dropout? I have only seen drop out being used in dense layers. I would like to know why it is done like this here.

Thanks,

Hi, I read about your work and it's great! I have one question related to the loss function that you used in your code, which is 1-ccc.
I have previous experience of using 1-ccc as loss function as well, but it turns out not a very good loss function to use. What I see is that the loss on train set drops gradually, but the loss on validation set keeps fluctuating, and it won't drop finally. In the end, I have to use mse instead.
So I am wondering, have you ever encountered this kind of problem during training with (1-ccc) loss function? If so , how do you solve it?

Hello, thanks for your work.

In file data_provider.py we prepare a tfrecords files, but we preprocess only audio data:
features=tf.train.Features(feature={
'sample_id': _int_feauture(i),
'subject_id': _int_feauture(subject_id),
'label': _bytes_feauture(label.tobytes()),
'raw_audio': _bytes_feauture(audio.tobytes()),
}))

But in fle data_generator.py we loading tfrecords with frames data:
features={
'raw_audio': tf.FixedLenFeature([], tf.string),
'label': tf.FixedLenFeature([], tf.string),
'subject_id': tf.FixedLenFeature([], tf.int64),
'frame': tf.FixedLenFeature([], tf.string),
}

The question is: when and where we encode frame data?

Hi,
I am a beginner of deep learning. And I downloaded the pretrained model from README, but I can't find checkpoint.
So, when I use saver.restore(sess, './emotion_recognition_model/')
The error below ocurred
ValueError: The passed save_path is not a valid checkpoint: ./emotion_recognition_model/

I hope this message find you well, Your answer will be of great help
Thanks

According to the paper, the cropped faces of subject's video are taken as input to the video network. My question is about the preprocessing pipeline. Which method or tool do you use to crop faces? How to align these faces? How to deal with the case where the facial landmark detection failed?
In data_provider.py, a sample is composed by the [frame, audio_sample, lable, subject_id] tensors, but in data_generator.py, there is no way to compute the frame tensor.