Code for ACCV 2020 Synthesizing the Unseen for Zero-shot Object Detection
Zero Shot Detection (ZSD) is a recently introduced paradigm which enables simultaneous localization and classification of previously unseen objects. It is arguably the most extreme case of learning with minimal supervision. we propose a symantically driven conditional feature generation module to synthesize visual features for unseen objects.
- mmdetection we recommend using Docker 2.0. Please use the mmdetection codes from this repo.
- conda env create -f environment.yml
The following scripts are for dfferents steps in the pipeline on MSCOCO dataset, please see the respective files for more arguments. Before running the scripts, please set the datasets and backbone paths in the config files. Weights of ResNet101 trained excluding overlapping unseen classes from ImageNet. Weights of FasterRCNN trained on MSCOCO2014, PASCALVOC, ILSVRC
cd mmdetection
./tools/dist_train.sh configs/faster_rcnn_r101_fpn_1x.py 8 --validate
cd mmdetection
# extract seen classes features to train Synthesizer and unseen class features for cross validation
python tools/zero_shot_utils.py configs/faster_rcnn_r101_fpn_1x.py --classes [seen, unseen] --load_from [detector checkpoint path] --save_dir [path to save features] --data_split [train, test]
# example to extract training features for seen classes
python tools/zero_shot_utils.py configs/faster_rcnn_r101_fpn_1x.py --classes seen --load_from ./work_dir/coco2014/epoch_12.pth --save_dir ../../data/coco --data_split train
# example to extract test features for unseen classes
python tools/zero_shot_utils.py configs/faster_rcnn_r101_fpn_1x.py --classes unseen --load_from ./work_dir/coco2014/epoch_12.pth --save_dir ../../data/coco --data_split test
# modify the paths to extracted features, labels and model checkpoints.
./script/train_coco_generator_65_15.sh
cd mmdetection
./tools/dist_test.sh configs/faster_rcnn_r101_fpn_1x.py [detector checkpoint path for seen classes] [num of gpus] --dataset [coco, voc, imagenet] --out [file name to save detection results] [--zsd, --gzsd] --syn_weights [path to synthesized classifier checkpoint]
# example
./tools/dist_test.sh configs/faster_rcnn_r101_fpn_1x.py work_dir/coco2014/epoch_12.pth 8 --dataset coco --out coco_results.pkl --zsd --syn_weights ../checkpoints/coco_65_15/classifier_best_137.pth
If you use this code for your research, please consider citing:
@misc{hayat2020synthesizing,
title={Synthesizing the Unseen for Zero-shot Object Detection},
author={Nasir Hayat and Munawar Hayat and Shafin Rahman and Salman Khan and Syed Waqas Zamir and Fahad Shahbaz Khan},
year={2020},
eprint={2010.09425},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
-
MSCOCO
-
MSCOCO Class Wise AP
-
ILSVRC
-
PASCAL VOC
-
Zero Shot Detections
-
Generalized Zero Shot Detections
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent