Stephane Vujasinovic, Sebastian Bullinger, Stefan Becker, Norbert Scherer-Negenborn, Michael Arens, Rainer Stiefelhagen

ICIP 2022

📰 New Project (18/09/2023):
READMem: Robust Embedding Association for a Diverse Memory in Unconstrained Video Object Segmentation

TL;DR: We manage the memory of STM like sVOS methods to better deal with long video. To attain long-term performance we estimate the inter-frame diversity of the base memory and integrate the embeddings of an incoming frame into the memory if it enhances the diversity. In return, we are able to limit the number of memory slots and deal with unconstrained video sequences without hindering the performance on short sequences and alleviate the need for a sampling interval.

Abstract

While current methods for interactive Video Object Segmentation (iVOS) rely on scribble-based interactions to generate precise object masks, we propose a Click-based interactive Video Object Segmentation (CiVOS) framework to simplify the required user workload as much as possible. CiVOS builds on de-coupled modules reflecting user interaction and mask propagation. The interaction module converts click-based interactions into an object mask, which is then inferred to the remaining frames by the propagation module. Additional user interactions allow for a refinement of the object mask. The approach is extensively evaluated on the popular interactive DAVIS dataset, but with an inevitable adaptation of scribble-based interactions with click-based counterparts. We consider several strategies for generating clicks during our evaluation to reflect various user inputs and adjust the DAVIS performance metric to perform a hardware-independent comparison. The presented CiVOS pipeline achieves competitive results, although requiring a lower user workload.

[Paper] [ArXiv] [PDF]

@INPROCEEDINGS{Vujasinović_2021_ICIP,
  author={Vujasinović, Stéphane and Bullinger, Sebastian and Becker, Stefan and Scherer-Negenborn, Norbert and Arens, Michael and Stiefelhagen, Rainer},
  booktitle={2022 IEEE International Conference on Image Processing (ICIP)}, 
  title={Revisiting Click-Based Interactive Video Object Segmentation}, 
  year={2022},
  pages={2756-2760},
  doi={10.1109/ICIP46576.2022.9897460}}

1. The framework is built with Python 3.7 and relies on the following packages:
   • NumPy 1.21.4
   • SciPy 1.7.2
   • PyTorch 1.10.0
   • torchvision 0.11.1
   • OpenCV 4.5.4 (opencv-python-headless if you don't want to use the demo)
   • Cython 0.29.24
   • scikit-learn 0.20.3
   • scikit-image 0.18.3
   • scipy 1.7.2
   • Pillow 8.4.0
   • imgaug 0.4.0
   • albumentations 1.10
   • tqdm 4.62.3
   • PyYaml 6.0
   • easydict 1.9
   • future 0.18.2
   • cffi 1.15.0
   • davis-interactive 1.0.4
   • networkx 2.6.3 for DAVIS
   • gdown 4.2.0 for downloading pretrained models
   • tensorboard 2.4.1
2. Download the DAVIS dataset download_datasets.py
3. Download the pretrained models download_models.py

1. Adapt the paths and variables in Demo.yml
2. Launch CiVOS_Demo.py (Nota bene: only 1 object can be segmented in the Demo)
3. Mouse and keyboard bindings:
   • Positive interaction: left mouse click
   • Negative interaction: right mouse click
   • Predict a mask of the object of interest for the video sequence: space bar
   • Visualize the results with the keys x(forward direction) and y(backward direction)
   • Quit the demo with key q

1. Adapt the paths and variables of EXAMPLE_DEBUGGING.yml
2. Adapt and lauch the bash file CiVOS_evaluation_script_example.sh
3. Read .csv files results with Summarize_with_DAVIS_arbitrary_report.py

Quantitative evaluation on the interactive DAVIS 2017 validation set.

R-AUC-J&F results on the DAVIS 2017 validation set for CiVOS by generating clicks in three different ways.

RiTM: GitHub, Paper

MiVOS: GitHub, Paper

DeepLabV3Plus: GitHub, Paper

DAVIS-interactive: GitHub, Project