anita-hu / msaf Goto Github PK

Offical implementation of paper "MSAF: Multimodal Split Attention Fusion"

License: MIT License

Python 99.22% Shell 0.78%

multimodal-learning multimodal-sentiment-analysis multimodal-emotion-recognition ntu-rgbd ravdess cmu-mosei pytorch action-recognition multimodal-deep-learning

msaf's Introduction

MSAF: Multimodal Split Attention Fusion

Code for the paper MSAF: Multimodal Split Attention Fusion. This is our implementation of the MSAF module and the three MSAF-powered multimodal networks.

If you use this code, please cite our paper:

@misc{su2020msaf,
      title={MSAF: Multimodal Split Attention Fusion}, 
      author={Lang Su and Chuqing Hu and Guofa Li and Dongpu Cao},
      year={2020},
      eprint={2012.07175},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

Installation

Clone this repo along with submodules

git clone --recurse-submodules https://github.com/anita-hu/MSAF.git

Install dependencies

Method 1: Using environment.yml (installs dependencies for all three datasets)

With this method, you can skip dependency installation steps from the dataset specific README files

conda env create -f environment.yml

Method 2: Without environment.yml

This code was developed with Python 3.6, PyTorch 1.7.1 in Ubuntu 20.04.

Basic dependencies (needed for all datasets): Pytorch, Tensorboard
Dataset specific dependencies: see README file in each dataset folder

Usage

The MSAF module is implemented in MSAF.py
The README file in each dataset folder has details on data preprocessing, training and evaluation (pretrained weights are available)

msaf's People

Contributors

Stargazers

Watchers

Forkers

jianwang-scu yuanqinglee qianfan1996 gonepiao rongfei-chen amena6490 jack960317 sun1775 binnchow

msaf's Issues

need 2 optimizers?

Hello.
There are 2 optimizers in the training phase, and it seems to learn the pose stream and apperance stream alternately.
Is there a reason why you didn't learn to cross-entropy loss using only one optimizer after combining the results of the pose stream and appearance stream?
Thank you for always.

error

CMU-MultimodalSDK @ fd18d71 is error

hi,i have some question about code

not run

CMU_MOSEI not run.error：ModuleNotFoundError: No module named ‘mmsdk‘

problem

When i run "ravdess",the error is "Expected object of scalar type Double but got scalar type Float for argument #3 'mat1' in call to th_addmm"

About code issues

Hello, can the code in Figure 3 in the paper be published? I want to see a section of that code. Thank you

Eval accuracy slightly different

Hi. Thank you so much for providing useful code.

When I tested NTU-RGBD datset through the command provided
'python main_msaf.py --datadir <path/to/NTU>
--checkpointdir checkpoints
--test_cp msaf_ntu_epoch12_92.24.checkpoint
--no_bad_skel'

, the performance was slightly different and 91.38 instead of 92.24.
Am I missing something? Thank you.

AttributeError: 'MSAFLSTMNet' object has no attribute 'audio_id'

i3d_32frames_85.63.checkpoint

Hi. Thanks for sharing your work.
May I ask how you got the i3d_32frames_85.63.checkpoint?

When I downloaded i3d checkpoint pre-trained on imagenet/kinetics from https://github.com/rimchang/kinetics-i3d-Pytorch/tree/master/data/pytorch_checkpoints and fine-tuned the network on NTU DB, the accuracy was about 82.56.

And also when I downloaed i3d_32frames_85.63.checkpoint from your repo and fine-tuned the I3D network on NTU, the accuracy wen up to 88.41.

Thank you.

about code

Hello, can you give me a copy of the complete code of Resnext，thank you

error

With this address,I have the problem:
Traceback (most recent call last):
File "dataset_prep.py", line 130, in
dataset = md.mmdataset(recipe)
File "/root/autodl-tmp/MSAF-master/cmu_mosei/mmsdk/mmdatasdk/dataset/dataset.py", line 28, in init
self.computational_sequences[entry]=computational_sequence(address,destination)
File "/root/autodl-tmp/MSAF-master/cmu_mosei/mmsdk/mmdatasdk/computational_sequence/computational_sequence.py", line 32, in init
h5handle,data,metadata=self.initialize(resource,destination)
File "/root/autodl-tmp/MSAF-master/cmu_mosei/mmsdk/mmdatasdk/computational_sequence/computational_sequence.py", line 92, in initialize
return readCSD(self.mainFile)
File "/root/autodl-tmp/MSAF-master/cmu_mosei/mmsdk/mmdatasdk/computational_sequence/file_ops.py", line 14, in readCSD
log.error("%s file not found, please check the path ..."%resource,error=True)
File "/root/autodl-tmp/MSAF-master/cmu_mosei/mmsdk/mmdatasdk/log/log.py", line 18, in error
action("%s%s"%(msgstart,msgstring))
File "/root/autodl-tmp/MSAF-master/cmu_mosei/mmsdk/mmdatasdk/log/log.py", line 17, in
action,msgstart=(lambda x:( for _ in ()).throw(errorType(x)),": ") if error else (lambda x:sys.stdout.write(str(x)+'\n'),bcolors.WARNING+bcolors.BOLD+": "+bcolors.ENDC)
File "/root/autodl-tmp/MSAF-master/cmu_mosei/mmsdk/mmdatasdk/log/log.py", line 17, in
action,msgstart=(lambda x:( for _ in ()).throw(errorType(x)),": ") if error else (lambda x:sys.stdout.write(str(x)+'\n'),bcolors.WARNING+bcolors.BOLD+": "+bcolors.ENDC)
RuntimeError: : ./CMU_MOSEI/csd/CMU_MOSEI_VisualFacet42.csd file not found, please check the path ...

anita-hu / msaf Goto Github PK

msaf's Introduction

MSAF: Multimodal Split Attention Fusion

Installation

Usage

msaf's People

Contributors

Stargazers

Watchers

Forkers

msaf's Issues

Recommend Projects

Recommend Topics

Recommend Org