This is the official code for the paper Collaborative Reasoning on Multi-Modal Semantic Graphs for Video-Grounded Dialogue Generation (EMNLP 2022 Findings)

We study video-grounded dialogue generation, where a response is generated based on the dialogue context and the associated video. The primary challenges of this task lie in (1) the difficulty of integrating video data into pre-trained language models (PLMs) which presents obstacles to exploiting the power of large-scale pre-training; and (2) the necessity of taking into account the complementarity of various modalities throughout the reasoning process. Although having made remarkable progress in video-grounded dialogue generation, existing methods still fall short when it comes to integrating with PLMs in a way that allows information from different modalities to complement each other. To alleviate these issues, we first propose extracting pertinent information from videos and turning it into reasoning paths that are acceptable to PLMs. Additionally, we propose a multi-agent reinforcement learning method to collaboratively perform reasoning on different modalities (i.e., video and dialogue context). Empirical experiment results on two public datasets indicate that the proposed model can significantly outperform state-of-the-art models by large margins on both automatic and human evaluations.

we first propose extracting pertinent information from videos and turning it into reasoning paths that are acceptable to PLMs. Additionally, we propose a multi-agent reinforcement learning method to collaboratively perform reasoning on different modalities (i.e., video and dialogue context)

pip install requirements.txt

We will maintain the code for creating the multi-modal semantic graph, but you can reproduce it following the details in the paper.

• Download DSTC7-AVSD
• Download Twitch-FIFA
• Install word2vec
• Coreference Resolution: AllenNLP | Huggingface | references | ChatGPT
• Open IE: OPENIE 5.1 | OPENIE6 | Stanford OpenIE | ChatGPT
• Video Action Extraction: PytorchVideo | SLOWFAST
• The data structure should look like the following (default)

data/
├── avsd/ # avsd dataset
    ├── vggish
    ├── i3d_flow
    ├── i3d_rgb
    ├── train.json
    ├── val.json
    └── test.json
├── mm-graph/semantic_graph/ # multi-modal semantic graph
    ├── avsd_context/traj.json
    ├── avsd_caption/traj.json
    └── avsd_video/traj.json

It is worth noting that we add the implementation of random reward (Discovering Diverse Multi-Agent Strategic Behavior via Reward Randomization) and simulated annealing (Generating Informative Dialogue Responses with Keywords-Guided Networks) in this version of code for further study.

python marl_vt_train_rr.py \
	--train_path data/avsd/train.json \
	--valid_path data/avsd/valid.json \
	--fea_path data/avsd/ \
	--context_traj_path data/mm-graph/semantic_graph/avsd_context/traj.json \
	--caption_traj_path data/mm-graph/semantic_graph/avsd_caption/traj.json \
	--video_traj_path data/mm-graph/semantic_graph/avsd_video/traj.json \
	--train_batch_size 4 \
	--valid_batch_size 4 \

python marl_vt_generate.py \
	--test_set data/avsd/test.json \
	--context_traj_path data/mm-graph/semantic_graph/avsd_context/traj.json \
	--caption_traj_path data/mm-graph/semantic_graph/avsd_caption/traj.json \
	--video_traj_path data/mm-graph/semantic_graph/avsd_video/traj.json \
	--beam_size 5 \
	--max_length 18 \
	--min_length 1 \
	--penalty 0.4 \
	--ckptid SELECTED_CKPT

@inproceedings{zhao-etal-2022-collaborative,
    title = "Collaborative Reasoning on Multi-Modal Semantic Graphs for Video-Grounded Dialogue Generation",
    author = "Zhao, Xueliang  and
      Wang, Yuxuan  and
      Tao, Chongyang  and
      Wang, Chenshuo  and
      Zhao, Dongyan",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2022",
    month = dec,
    year = "2022",
    address = "Abu Dhabi, United Arab Emirates",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2022.findings-emnlp.442",
    pages = "5988--5998","",
}