hi @Justin1904,

I am unable to install complete list of files, only following files are installed
CMU_MOSI_ModifiedTimestampedWords.csd
CMU_MOSI_Opinion_Labels.csd
CMU_MOSI_TimestampedPhones.csd
CMU_MOSI_TimestampedWords.csd
CMU_MOSI_TimestampedWordVectors_1.1.csd

First of all, thank you very much for your tutorial! However, when I ran the following code, I encountered an error...So, can you tell me how to solve it? Thank you.
Code:
words.append(word2id[word[0].decode('utf-8')]) # SDK stores strings as bytes, decode into strings here
Error:
AttributeError: 'numpy.float64' object has no attribute 'decode'

Hello,

first of all thanks for the great work on the tutorial. Could you explain how I could use the trained model to let it run on my own data ?(for example a regular video of myself)

Hi,

using the CMU-MOSI dataset:
the testing accuracy in the tutorial is around 71.6%.
I'm using the exactly same code from the tutorials, but I only get around 61.2% testing accuracy.

Any suggestion? or the author of CMU-MultimodalSDK has done something new to the dataset?

Thanks!

I'm trying to run through the experiments but i'm unable to get past the part where torch is imported.
I have Python: 3.7.4 installed. I'm currently running windows 10 on MacOs.
I have installed pytorch with the following command -> conda install pytorch torchvision cpuonly -c pytorch.
I get the following error running that segment.

Hi,

Thanks for this tutorials. When I downloaded dataset，I encountered this problem，which means the dataset wasn't be downloaded properly.

Here is the problem：
| Error | /data/mosei/CMU_MOSEI_VisualOpenFace2.csd resource is not a valid hdf5 computational sequence format ...

However，When I redownload the dataset without deleting the downloaded ones，I encountered：
“High-level features have been downloaded previously.”

Seems the downloader only checks the file name.

I wonder is there any way that I can download the proper dataset without deleting old ones?As u know,the dataset is enormous.

Hi,

Thank you for this tutorials. Is there anywhere a description of the different database that have been created and relaesed in the SDK.

Fo example
in the part of the tutorial to define modalities. The files below are chosen:

define your different modalities - refer to the filenames of the CSD files

visual_field = 'CMU_MOSI_VisualFacet_4.1'
acoustic_field = 'CMU_MOSI_COVAREP'
text_field = 'CMU_MOSI_ModifiedTimestampedWords'

But with the SDK we obtain the following list of CSD:
#Audio
CMU_MOSI_COVAREP.csd
CMU_MOSI_OpenSmile_EB10.csd
CMU_MOSI_openSMILE_IS09.csd

#Language
CMU_MOSI_TimestampedWords.csd
CMU_MOSI_TimestampedWordVectors.csd
CMU_MOSI_TimestampedWordVectors_1.1.csd
CMU_MOSI_TimestampedPhones.csd

#Video
CMU_MOSI_Visual_OpenFace_1.csd
CMU_MOSI_Visual_OpenFace_2.csd
CMU_MOSI_Visual_Facet_41.csd
CMU_MOSI_Visual_Facet_42.csd

is there any document where I could find what are the differences of those files/ How they were pre-processed?

Thank you in advance for your reply.

Regards,

Leticia Fernandez Moguel

Python exception :
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x89 in position 0: invalid start byte

Encountered in the definition of the load_emb function

We tried to overcome the issue by reading the file at WORD_EMB_PATH as a binary file but we just met another problem so we assume this is not the way to go.

thank you for the tutorial, very helpful insights!
The Jupiter notebook works fine,
I saw a model.std as the final output;
Can you guide us in some experiments:
how we can test this model with other inputs?, I didn't found documentation about methods and this phase of "toy use of the model.."
how we correctly preprocess external datasets for test or dev environments?
thank you for your attention.
Pedro Moya

Thanks for your repo! I've a question about labels.

How to get Emotion labels from CMU_MOSEI_LabelsEmotions.csd?
I've written the next code:

dataset=mmdatasdk.mmdataset(mmdatasdk.cmu_mosei.labels,'./cmumosei')

In debug mode I see the next picture:

When we look into data, we get something like this:

I can not see emotion labels here (like sadness, neutral etc.)

What have I to do to get them for each frame in videos?

Hi @Justin1904

Thank You for the great tutorial for the CMU-MultimodalSDK. Its very well explained.

I was successful in training the LSTM defined in this tutorial code with the CMU provided .csd files.

I created my own csd for visual feature from openface. But when instead of facet visual features csd file i used this csd . Rest features are unchanged.
The function of alignment with labels is throwing error this below error :-

<Status>: Unify was called ....
<Warning>: 9qR7uwkblbs entry is not shared among all sequences, removing it ...
<Warning>: POKffnXeBds entry is not shared among all sequences, removing it ...
<Warning>: BioHAh1qJAQ entry is not shared among all sequences, removing it ...

.....**same warning followed for other 88 videos**

Traceback (most recent call last):
  File "tutorial_interactive_demo.py", line 255, in <module>
    _visual = dataset[visual_field].data[segment]['features']
KeyError: ['2WGyTLYerpo[51]'[]

Please guide me for the resolution for this ? (I have also attached a log file of the training)
I have used features for 91 videos in the CMU-MOSI raw dataset instead of 23 available .
I aim to train the model on my own videos.
log_train.txt