In this project, we developed basic Encoder-Decoder model, and S2VT model to generate video captions. In addition, we also applied Attention machnism to improve performance.

The following instructions will get you a copy of the project and running on your local machine for testing purposes.

The following are some toolkits and their version you need to install for running this project

• Python 3.6 - The Python version used
• Pytorch 0.3 - Deep Learning for Python
• Pandas 0.21.0 - Data Analysis Library for Python

In addition, it is required to use GPU to run this project.

The following are the model structures we implemented in Pytorch from scratch:

• [Baseline Model]
• [S2VT Model] In order to improve performance, we also implemented Bahdanau Attention and Luong Attention

[1] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv:1409.0473
[2] Minh-Thang Luong, Hieu Pham, Christopher D. Manning. 2015. Effective Approaches to Attention-based Neural Machine Translation
[3] Samy Bengio, Oriol Vinyals, Navdeep Jaitly, Noam Shazeer. 2015. Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
[4] Natsuda Laokulrat, Sang Phan, Noriki Nishida. 2016. Generating Video Description using Sequence-to-sequence Model with Temporal Attention