A PyTorch implementation of our paper DSNet: A Flexible Detect-to-Summarize Network for Video Summarization by Wencheng Zhu, Jiwen Lu, Jiahao Li, and Jie Zhou. Published in IEEE Transactions on Image Processing.

This project is developed on Ubuntu 16.04 with CUDA 9.0.176.

First, clone this project to your local environment.

git clone https://github.com/li-plus/DSNet.git

Create a virtual environment with python >= 3.6.

virtualenv -p /usr/bin/python3.8 dsnet
. dsnet/bin/activate

Install python dependencies.

pip install -r requirements.txt

Download the pre-processed datasets into datasets/ folder, including TVSum, SumMe, OVP, and YouTube datasets.

mkdir -p datasets/ && cd datasets/
wget https://www.dropbox.com/s/tdknvkpz1jp6iuz/dsnet_datasets.zip
unzip dsnet_datasets.zip

If the Dropbox link is unavailable to you, try downloading from below links.

• (Baidu Cloud) Link: https://pan.baidu.com/s/1LUK2aZzLvgNwbK07BUAQRQ Extraction Code: x09b
• (Google Drive) https://drive.google.com/file/d/11ulsvk1MZI7iDqymw9cfL7csAYS0cDYH/view?usp=sharing

Now the datasets structure should look like

DSNet
└── datasets/
    ├── eccv16_dataset_ovp_google_pool5.h5
    ├── eccv16_dataset_summe_google_pool5.h5
    ├── eccv16_dataset_tvsum_google_pool5.h5
    ├── eccv16_dataset_youtube_google_pool5.h5
    └── readme.txt

To train anchor-based attention model on TVSum and SumMe datasets with canonical settings, run

python train.py anchor-based --model-dir ../models/ab_basic --splits ../splits/tvsum.yml ../splits/summe.yml

To train on augmented and transfer datasets, run

python train.py anchor-based --model-dir ../models/ab_tvsum_aug/ --splits ../splits/tvsum_aug.yml
python train.py anchor-based --model-dir ../models/ab_summe_aug/ --splits ../splits/summe_aug.yml
python train.py anchor-based --model-dir ../models/ab_tvsum_trans/ --splits ../splits/tvsum_trans.yml
python train.py anchor-based --model-dir ../models/ab_summe_trans/ --splits ../splits/summe_trans.yml

To train with LSTM, Bi-LSTM or GCN feature extractor, specify the --base-model argument as lstm, bilstm, or gcn. For example,

python train.py anchor-based --model-dir ../models/ab_basic --splits ../splits/tvsum.yml ../splits/summe.yml --base-model lstm

Much similar to anchor-based models, to train on canonical TVSum and SumMe, run

python train.py anchor-free --model-dir ../models/af_basic --splits ../splits/tvsum.yml ../splits/summe.yml --nms-thresh 0.4

Note that NMS threshold is set to 0.4 for anchor-free models.

To evaluate your anchor-based models, run

python evaluate.py anchor-based --model-dir ../models/ab_basic/ --splits ../splits/tvsum.yml ../splits/summe.yml

For anchor-free models, remember to specify NMS threshold as 0.4.

python evaluate.py anchor-free --model-dir ../models/af_basic/ --splits ../splits/tvsum.yml ../splits/summe.yml --nms-thresh 0.4

Based on the public datasets provided by DR-DSN, we apply KTS algorithm to generate video shots for OVP and YouTube datasets. Note that the pre-processed datasets already contain these video shots. To re-generate video shots, run

python make_shots.py --dataset ../datasets/eccv16_dataset_ovp_google_pool5.h5
python make_shots.py --dataset ../datasets/eccv16_dataset_youtube_google_pool5.h5

We gratefully thank the below open-source repo, which greatly boost our research.

• Thank KTS for the effective shot generation algorithm.
• Thank DR-DSN for the pre-processed public datasets.
• Thank VASNet for the training and evaluation pipeline.

If you find our codes or paper helpful, please consider citing.

@article{zhu2020dsnet,
  title={DSNet: A Flexible Detect-to-Summarize Network for Video Summarization},
  author={Zhu, Wencheng and Lu, Jiwen and Li, Jiahao and Zhou, Jie},
  journal={IEEE Transactions on Image Processing},
  volume={30},
  pages={948--962},
  year={2020}
}