soujanyaporia / multimodal-sentiment-analysis Goto Github PK

Thanks for sharing your works and codes! Would you please tell me which version of tensorflow you use? Because I am encountering too many issues due to the version incompatibility problem.

In the README.md:

MOSEI:
3 classes: happy/sad/neutral/angry/excited/frustrated 
Raw Features: (Pickle files) 
Audio: dataset/mosei/raw/audio_3way.pickle 
Text: dataset/mosei/raw/text_3way.pickle 
Video: dataset/mosei/raw/video_3way.pickle 

3 classes is happy/sad/neutral/angry/excited/frustrated ???
So the MOSEI dataset classes should be 3 or 6 ???

We noticed that unimodal_mosei_3way.pickle.zip has three classes, that is, positive, negative, and neutral. However, in the original CMU-MOSEI data, there are either five classes or binary classes.
So can I ask how these data are generated and are they features or raw data?

Because I might use the wrong versions of both python and tensorflow, I met many problems when I ran this project. Please tell me the Python version and the tensorflow version you used. Thanks in advanced.

Hi,
When I run the command:

python3 run.py --unimodal True --fusion True

the output is

Traceback (most recent call last):
File "run.py", line 384, in
unimodal(mode, args.data, args.classes)
File "run.py", line 276, in unimodal
test_feed_dict)
File "/Users/marcostexeira/tensorflow3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 905, in run
run_metadata_ptr)
File "/Users/marcostexeira/tensorflow3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1113, in _run
str(subfeed_t.get_shape())))

ValueError: Cannot feed value of shape (31, 63) for Tensor 'y:0', which has shape '(?, 63, 2)'

Is this related to running locally, rather than GPU?

Hi, I have tried some of the commands, and the results are:

Command1：
python run.py --unimodal True --fusion True --attention_2 True
Results：
Best epoch: 40
Best test accuracy: 0.7593085169792175
Best epoch loss: 22
Best test accuracy when loss is least: 0.7433510422706604



Command2：
python run.py --unimodal False --fusion True --attention_2 True
Results：
Best epoch: 33
Best test accuracy: 0.7619680762290955
Best epoch loss: 21
Best test accuracy when loss is least: 0.7380319237709045



Command3：
python run.py --unimodal True --fusion False
Results：
Best epoch: 12
Best test accuracy: 0.769946813583374
Best epoch loss: 11
Best test accuracy when loss is least: 0.7659574747085571



Command4:
python run.py --unimodal True --fusion True
Results:
Best epoch: 13
Best test accuracy: 0.7659574747085571
Best epoch loss: 12
Best test accuracy when loss is least: 0.7606382966041565



Command5:
python run.py --unimodal False --fusion True
Results:
Best epoch: 14
Best test accuracy: 0.769946813583374
Best epoch loss: 13
Best test accuracy when loss is least: 0.7686170339584351



Command6:
python run.py --unimodal False --fusion False
Results:
Best epoch: 12
Best test accuracy: 0.7659574747085571
Best epoch loss: 11
Best test accuracy when loss is least: 0.7606382966041565

Firstly, I think Command1 will be the best, but the result is not. The best result comes from Command3 and Command5(both without attention_2), so what's the point of doing attention_2? Besides, why do you use fscore instead of accuracy in your paper?

Secondly, the best accuracy of Command3(without fusion) is better than Command 4, so what's the point of fusion?

Hi @gangeshwark @soujanyaporia, I would like to ask you how the 100 OpenSmile features for IEMOCAP dataset were extracted. I have not been able to generate data similar to the data provided in the pkl files. Could you give a detailed explanation on how the feature was produces or provide the feature extraction scripts.

Kind regards

Hi~ I have read your paper "Multi-level Multiple Attentions for Contextual Multimodal Sentiment Analysis", the metric used in the paper is macro-fscore, but from the source code you provide, the output is only accuracy on the test set, and the f1 value on the training set is no more than 80%. So I want to know how the macro-fscore is calculated on the test set.
And... could you tell me how to achieve attention visualization? Thanks.

Hi,

Can you share the script used to create the pickle files from the original data files?

Thanks.

It will be very helpful if you can upload the data processing part.

Would you please tell me which version of tensorflow you use? Thank you very much !

When I first run “python run.py --unimodal True --fusion True” ,then " unimodal_mosi_2way.pickle" had been changed. Next I run "python run.py --unimodal False --fusion True --attention_2 True" ,the accuracy is only 0.76.
I can only use the "unimodal_mosi_2way.pickle" provided by the author to get the accuracy mentioned in the article. I want to ask if the author can provide the code for the training "unimodal_mosi_2way.pickle". Because the existing code is not achieving the effect mentioned by the author.
Thank you very much for looking at my problems while I am busy, I look forward to your reply！

when I run python run.py --unimodal True --fusion True

error:

Namespace(fusion=True, unimodal=True)
Training unimodals first
('starting unimodal ', 'text')
Traceback (most recent call last):
  File "run.py", line 415, in <module>
    unimodal(mode)
  File "run.py", line 240, in unimodal
    (train_data, train_label, _, _, test_data, test_label, _, train_length, _, test_length, _, _, _) = u.load()
ValueError: not enough values to unpack (expected 13, got 7)

how can I solve it...

Hi,
Could you share the script used to perform the dimension equalization ( d=300 ), cited on paper (Multi-level Multiple Attentions for Contextual Multimodal Sentiment Analysis) for multimodal fusion experiment ?

Hello,I found a performance issue in the definition of unimodal ,
run.py,
sess = tf.Session(config=session_conf) was repeatedly called in for mode in modality: and was not closed.
I think it will increase the efficiency and avoid out of memory if you close this session after using it.

Here are two files to support this issue,support1 and support2

Looking forward to your reply. Btw, I am very glad to create a PR to fix it if you are too busy.