alexadlu / matnet Goto Github PK

UPDATES:

• [2020/03/04] Update results for DAVIS-17 validation set!
• [2019/11/17] Codes released!

This is a PyTorch implementation of our MATNet for unsupervised video object segmentation.

Motion-Attentive Transition for Zero-Shot Video Object Segmentation. Paper

Tianfei Zhou, Shunzhou Wang, Yi Zhou, Yazhou Yao, Jianwu Li, Ling Shao, AAAI 2020, New York, USA.

The training and testing experiments are conducted using PyTorch 1.0.1 with a single GeForce RTX 2080Ti GPU with 11GB Memory.

• PyTorch 1.0.1

Other minor Python modules can be installed by running

pip install -r requirements.txt

git clone --recursive https://github.com/tfzhou/MATNet.git

In the paper, we use the following two public available dataset for training. Here are some steps to prepare the data:

• DAVIS-17: we use all the data in the train subset of DAVIS-16. However, please download DAVIS-17 to fit the code. It will automatically choose the subset of DAVIS-16 for training.

• YoutubeVOS-2018: we sample the training data every 10 frames in YoutubeVOS-2018.

• Create soft links:

  cd data; ln -s your/davis17/path DAVIS2017; ln -s your/youtubevos/path YouTubeVOS_2018;

I have provided some matlab scripts to generate edge annotations from mask. Please run data/run_davis2017.m and data/run_youtube.m.

I have provided the pytorch codes to generate HED results for the two datasets (see 3rdparty/pytorch-hed). Please run run_davis.py and run_youtube.py.

The codes are borrowed from https://github.com/sniklaus/pytorch-hed.

I have provided the pytorch codes to generate optical flow results for the two datasets (see 3rdparty/pytorch-pwc). Please run run_davis2017.py and run_youtubevos.py.

The codes are borrowed from https://github.com/sniklaus/pytorch-pwc. Please follow the setup section to install cupy.

warning: Total size of optical flow results of Youtube-VOS is more than 30GB.

Once all data is prepared, please run python train_MATNet.py for training.

1. Run python test_MATNet.py to obtain the saliency results on DAVIS-16 val set.
2. Run python apply_densecrf_davis.py for binary segmentation results.

1. The segmentation results on DAVIS-16 and Youtube-objects can be downloaded from Google Drive.
2. The segmentation results on DAVIS-17 can be downloaded from Google Drive. We achieved 58.6 in terms of Mean J&F.

The pre-trained model can be downloaded from Google Drive.

If you find MATNet useful for your research, please consider citing the following paper:

@inproceedings{zhou2020motion,
  title={Motion-Attentive Transition for Zero-Shot Video Object Segmentation},
  author={Zhou, Tianfei and Wang, Shunzhou and Zhou, Yi and Yao, Yazhou and Li, Jianwu and Shao, Ling},
  booktitle={Proceedings of the 34th AAAI Conference on Artificial Intelligence (AAAI)},
  year={2020},
  organization={AAAI}
}