- Description
- Dependencies
- Manual
- Data
- Results
- Results and Data for the Final Version of the Paper
- Citation
This repo contains the code of Semantics-Assisted Video Captioning Model, based on the paper "Semantics-Assisted Video Captioning Model Trained with Scheduled Sampling Strategy". It is under review at Frontiers in Robotics and AI.
We propose three ways to improve the video captioning model. First of all, we utilize both spatial features and dynamic spatio-temporal features as inputs for semantic detection network in order to generate meaningful semantic features for videos. Then, we propose a scheduled sampling strategy which gradually transfers the training phase from a teacher guiding manner towards a more self teaching manner. At last, the ordinary logarithm probability loss function is leveraged by sentence length so that short sentence inclination is alleviated. Our model achieves state-of-the-art results on the Youtube2Text dataset and is competitive with the state-of-the-art models on the MSR-VTT dataset.
The overall structure of our model looks like this 
.
Here is some captions generated by our model.
If you need a newer and more powerful model, please refer to Delving-Deeper-into-the-Decoder-for-Video-Captioning.
- Python3.6
- TensorFlow 1.13
- NumPy
- sklearn
- pycocoevalcap(Python3)
- Make sure you have installed all the required packages.
- Download pycocoevalcap and put it along with
msrvttt,msvd,taggingfolders. - Download files in the Data section.
cd path_to_directory_of_model; mkdir savesrun_model.shis used for training models andtest_model.shis used for testing models. Specify the GPU you want to use by modifyingCUDA_VISIBLE_DEVICESvalue. Specify the needed data paths by modifyingcorpus,ecores,tagandrefvalues. The words will be sampled by argmax strategy ifargmaxis 1 and they will be sampled by multinomial strategy ifargmaxis 0.nameis the name which you give to the model.testrefers to the path of the saved model which is to be tested. Do not give a parameter totestif you want to train a model.- After completing the configuration of the bash file, then
bash run_model.shfor training,bash test_model.shfor testing.
| Model | B-4 | C | M | R | Overall |
|---|---|---|---|---|---|
| LSTM-E | 45.3 | 31.0 | |||
| h-RNN | 49.9 | 65.8 | 32.6 | ||
| aLSTMs | 50.8 | 74.8 | 33.3 | ||
| SCN | 51.1 | 77.7 | 33.5 | ||
| MTVC | 54.5 | 92.4 | 36.0 | 72.8 | 0.9198 |
| ECO | 53.5 | 85.8 | 35.0 | ||
| SibNet | 54.2 | 88.2 | 34.8 | 71.7 | 0.8969 |
| Our Model | 61.8 | 103.0 | 37.8 | 76.8 | 1.0000 |
| Model | B-4 | C | M | R | Overall |
|---|---|---|---|---|---|
| v2t_navigator | 40.8 | 44.8 | 28.2 | 60.9 | 0.9325 |
| Aalto | 39.8 | 45.7 | 26.9 | 59.8 | 0.9157 |
| VideoLAB | 39.1 | 44.1 | 27.7 | 60.6 | 0.9140 |
| MTVC | 40.8 | 47.1 | 28.8 | 60.2 | 0.9459 |
| CIDEnt-RL | 40.5 | 51.7 | 28.4 | 61.4 | 0.9678 |
| SibNet | 40.9 | 47.5 | 27.5 | 60.2 | 0.9374 |
| HACA | 43.4 | 49.7 | 29.5 | 61.8 | 0.9856 |
| TAMoE | 42.2 | 48.9 | 29.4 | 62.0 | 0.9749 |
| Our Model | 43.8 | 51.4 | 28.9 | 62.4 | 0.9935 |
- MSVD tag index2word and word2index mappings(ExternalRepo)
- We use the same word-index mapping in semantic tag to the code in this link.
-
MSR-VTT Dataset:
- train_val_test_annotation.zip (GoogleDrive)
- SHA-256: ce2d97dd82d03e018c6f9ee69c96eb784397d1c83f734fdb8c17aafa5e27da31
- msr-vtt-v1.part1.rar (GoogleDrive)
- SHA-256: 3445e0d1bffda3739110dfcf14182b63222731af8a4d7153f0ac09dbec39a0d3
- msr-vtt-v1.part2.rar (GoogleDrive)
- SHA-256: b550997526272ab68a42f1bd93315aa2bbb521c71f33d0cb922fbbfb86f15aae
- msr-vtt-v1.part3.rar (GoogleDrive)
- SHA-256: debbd0e535e77d9927ffb375299c08990519e22ba7dac542b464b70d440ef515
- train_val_test_annotation.zip (GoogleDrive)
-
Data and Models for both MSVD and MSR-VTT
- data.zip
- SHA-256: fadd721eaa0f13aff7c3505e4784a003514c33ffa5a934a9dcf13955285df11f
- Source Code: GitHub.
- ECO_full_kinetics.caffemodel (GoogleDrive)
- MD5 31ed18d5eadfd59cb65b7dcdadc310b4
- SHA-1 b749384d2dac102b8035965566e3030fce465c20
-
MSVD Results
-
MSR-VTT Results
- SHA256: d2a731794ef1bc90c9ccd6c7fe5e92fa7ad104f9e9188ac751c984b23d3a939b
@ARTICLE{10.3389/frobt.2020.475767,
AUTHOR={Chen, Haoran and Lin, Ke and Maye, Alexander and Li, Jianmin and Hu, Xiaolin},
TITLE={A Semantics-Assisted Video Captioning Model Trained With Scheduled Sampling},
JOURNAL={Frontiers in Robotics and AI},
VOLUME={7},
PAGES={129},
YEAR={2020},
URL={https://www.frontiersin.org/article/10.3389/frobt.2020.475767},
DOI={10.3389/frobt.2020.475767},
ISSN={2296-9144},
ABSTRACT={Given the features of a video, recurrent neural networks can be used to automatically generate a caption for the video. Existing methods for video captioning have at least three limitations. First, semantic information has been widely applied to boost the performance of video captioning models, but existing networks often fail to provide meaningful semantic features. Second, the Teacher Forcing algorithm is often utilized to optimize video captioning models, but during training and inference, different strategies are applied to guide word generation, leading to poor performance. Third, current video captioning models are prone to generate relatively short captions that express video contents inappropriately. Toward resolving these three problems, we suggest three corresponding improvements. First of all, we propose a metric to compare the quality of semantic features, and utilize appropriate features as input for a semantic detection network (SDN) with adequate complexity in order to generate meaningful semantic features for videos. Then, we apply a scheduled sampling strategy that gradually transfers the training phase from a teacher-guided manner toward a more self-teaching manner. Finally, the ordinary logarithm probability loss function is leveraged by sentence length so that the inclination of generating short sentences is alleviated. Our model achieves better results than previous models on the YouTube2Text dataset and is competitive with the previous best model on the MSR-VTT dataset.}
}
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent