PyTorch implementation of "Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation" by Jiang H., Sun D., Jampani V., Yang M., Learned-Miller E. and Kautz J. [Project] [Paper]
Results
Results on UCF101 dataset using the evaluation script provided by paper's author. The get_results_bug_fixed.sh
script was used. It uses motions masks when calculating PSNR, SSIM and IE.
Method | PSNR | SSIM | IE |
---|---|---|---|
DVF | 29.37 | 0.861 | 16.37 |
SepConv - L_1 | 30.18 | 0.875 | 15.54 |
SepConv - L_F | 30.03 | 0.869 | 15.78 |
SuperSloMo_Adobe240fps | 29.80 | 0.870 | 15.68 |
pretrained mine | 29.77 | 0.874 | 15.58 |
SuperSloMo | 30.22 | 0.880 | 15.18 |
Prerequisites
This codebase was developed and tested with pytorch 0.4.1 and CUDA 9.2.
Install:
For GPU, run
conda install pytorch=0.4.1 cuda92 -c pytorch
pip install torchvision
For CPU, run
conda install pytorch-cpu=0.4.1 -c pytorch
pip install torchvision
- TensorboardX for training visualization
- tensorflow for tensorboard
- matplotlib for training graph in notebook.
- tqdm for progress bar in video_to_slomo.py
- numpy
Training
Preparing training data
In order to train the model using the provided code, the data needs to be formatted in a certain manner.
The create_dataset.py script uses ffmpeg to extract frames from videos.
Adobe240fps
For adobe240fps, download the dataset, unzip it and then run the following command
python data\create_dataset.py --ffmpeg_dir path\to\folder\containing\ffmpeg --videos_folder path\to\adobe240fps\videoFolder --dataset_folder path\to\dataset --dataset adobe240fps
Custom
For custom dataset, run the following command
python data\create_dataset.py --ffmpeg_dir path\to\folder\containing\ffmpeg --videos_folder path\to\adobe240fps\videoFolder --dataset_folder path\to\dataset
The default train-test split is 90-10. You can change that using command line argument --train_test_split
.
Run the following commmand for help / more info
python data\create_dataset.py --h
Training
In the train.ipynb, set the parameters (dataset path, checkpoint directory, etc.) and run all the cells.
or to train from terminal, run:
python train.py --dataset_root path\to\dataset --checkpoint_dir path\to\save\checkpoints
Run the following commmand for help / more options like continue from checkpoint, progress frequency etc.
python train.py --h
Tensorboard
To get visualization of the training, you can run tensorboard from the project directory using the command:
tensorboard --logdir log --port 6007
and then go to https://localhost:6007.
Evaluation
Pretrained model
You can download the pretrained model trained on adobe240fps dataset here.
Video Converter
You can convert any video to a slomo or high fps video (or both) using video_to_slomo.py. Use the command
python video_to_slomo.py --ffmpeg path\to\folder\containing\ffmpeg --video path\to\video.mp4 --sf N --checkpoint path\to\checkpoint.ckpt --fps M --output path\to\output.mp4
If you want to convert a video from 30fps to 90fps set fps
to 90 and sf
to 3 (to get 3x frames than the original video).
Run the following commmand for help / more info
python video_to_slomo.py --h
You can also use eval.py
if you do not want to use ffmpeg. You will instead need to install opencv-python
using pip for video IO.
A sample usage would be:
python eval.py data/input.mp4 --checkpoint=data/SuperSloMo.ckpt --output=data/output.mp4 --scale=4
Use python eval.py --help
for more details
More info TBA
References:
Parts of the code is based on TheFairBear/Super-SlowMo