Implement ConvLSTM/ConvGRU cell with Pytorch. This idea has been proposed in this paper: Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting

Encoder-decoder structure. Takes in a sequence of 10 movingMNIST fames and attempts to output the remaining frames.

Requires Pytorch v1.1 or later (and GPUs)

Clone repository

git clone https://github.com/jhhuang96/ConvLSTM-PyTorch.git

To run endoder-decoder network for prediction moving-mnist:

python main.py

The script data/mm.py is the script to generate customized Moving Mnist based on MNIST.

MovingMNIST(is_train=True,
            root='data/',
            n_frames_input=args.frames_input,
            n_frames_output=args.frames_output,
            num_objects=[3])

• is_train: If True, use script to generate data. If False, directly use Moving Mnist data downloaded from http://www.cs.toronto.edu/~nitish/unsupervised_video/
• root: The path of MNIST data
• n_frames_input: Number of input frames (int)
• n_frames_output: Number of output frames (int)
• num_objects: Number of digits in a frame (List) . [3] means there are 3 digits in each frame

• The first line is the real data for the first 10 frames
• The second line is prediction of the model for the last 10 frames

@inproceedings{xingjian2015convolutional,
  title={Convolutional LSTM network: A machine learning approach for precipitation nowcasting},
  author={Xingjian, SHI and Chen, Zhourong and Wang, Hao and Yeung, Dit-Yan and Wong, Wai-Kin and Woo, Wang-chun},
  booktitle={Advances in neural information processing systems},
  pages={802--810},
  year={2015}
}
@inproceedings{xingjian2017deep,
    title={Deep learning for precipitation nowcasting: a benchmark and a new model},
    author={Shi, Xingjian and Gao, Zhihan and Lausen, Leonard and Wang, Hao and Yeung, Dit-Yan and Wong, Wai-kin and Woo, Wang-chun},
    booktitle={Advances in Neural Information Processing Systems},
    year={2017}
}