githwd2016 / mate Goto Github PK

Thanks a lot for your instruction in the dev-branch.

May I know the "requirements.txt", which is your environment configure?
At least, it will be much helpful to tell us your PyTorch version.

Hi,
When will the code be available
Thx

When I run your training after filling some parameters for the configure-json file, there is an issue in loading image features, which leads to inconsistent dimensions, as below,

Traceback (most recent call last):
File "/home/jfhe/Documents/daily software/pycharm/pycharm-professional-anaconda-2019.1.1/pycharm-anaconda-2019.1.1/helpers/pydev/pydevd.py", line 1741, in
main()
File "/home/jfhe/Documents/daily software/pycharm/pycharm-professional-anaconda-2019.1.1/pycharm-anaconda-2019.1.1/helpers/pydev/pydevd.py", line 1735, in main
globals = debugger.run(setup['file'], None, None, is_module)
File "/home/jfhe/Documents/daily software/pycharm/pycharm-professional-anaconda-2019.1.1/pycharm-anaconda-2019.1.1/helpers/pydev/pydevd.py", line 1135, in run
pydev_imports.execfile(file, globals, locals) # execute the script
File "/home/jfhe/Documents/daily software/pycharm/pycharm-professional-anaconda-2019.1.1/pycharm-anaconda-2019.1.1/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "/home/jfhe/Documents/MountHe/jfhe/mm_dialogue/MM_Dialogue/train.py", line 268, in
exit(main(_args))
File "/home/jfhe/Documents/MountHe/jfhe/mm_dialogue/MM_Dialogue/train.py", line 167, in main
dec_output_prob = model_p((text_input, text_pos, text_turn, text_speaker,
File "/home/jfhe/anaconda3/envs/MM20/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/jfhe/anaconda3/envs/MM20/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py", line 165, in forward
return self.module(*inputs[0], **kwargs[0])
File "/home/jfhe/anaconda3/envs/MM20/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/jfhe/Documents/MountHe/jfhe/mm_dialogue/MM_Dialogue/widget/model.py", line 83, in forward
context_embs, context_seq = self.context_encode(context)
File "/home/jfhe/Documents/MountHe/jfhe/mm_dialogue/MM_Dialogue/widget/model.py", line 88, in context_encode
text_embs, image_embs = self.emb(text_input, text_pos, text_turn, text_speaker,
File "/home/jfhe/anaconda3/envs/MM20/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/jfhe/Documents/MountHe/jfhe/mm_dialogue/MM_Dialogue/widget/module.py", line 60, in forward
image_outputs = self.image_layer(image_input) + self.src_token_emb(image_turn) +
RuntimeError: The size of tensor a (12) must match the size of tensor b (24) at non-singleton dimension 1
import sys; print('Python %s on %s' % (sys.version, sys.platform))
Python 3.8.10 (default, May 19 2021, 18:05:58)
[GCC 7.3.0] on linux
image_input.shape
torch.Size([1, 12, 4096])
image_turn.shape
torch.Size([1, 24])
image_speaker.shape
torch.Size([1, 24])
image_pos.shape
torch.Size([1, 24])

The code is from (widget/module.py/class Eb) as below [ def forward(self, src_input, src_pos, src_turn, src_speaker, image_input, image_pos, image_turn, image_speaker): # text_input, text_pos, text_turn, text_speaker, image_turn, image_speaker = (bs, text_len) # image_input = (bs, image_len, image_in_size) text_outputs = self.src_token_emb(src_input) + self.src_token_emb(src_turn) + \ self.src_token_emb(src_speaker) + self.position_enc(src_pos) # (bs, text_len, embedding_size) image_outputs = self.image_layer(image_input) + self.src_token_emb(image_turn) + \ self.src_token_emb(image_speaker) + self.src_token_emb(image_pos) # (bs, image_len, embedding_size) return text_outputs, image_outputs](url), where the four items cannot sum up due to in consistent feature dimensions.

May I ask where the second-stage decoder (contains style-tips and cel_preference) is, because it seems that current code dosen't contain it.