- Task: Zero-shot image captioning
- Chitrang Goyani - [email protected]
- Paper: https://openaccess.thecvf.com/content/ICCV2023/html/Fei_Transferable_Decoding_with_Visual_Entities_for_Zero-Shot_Image_Captioning_ICCV_2023_paper.html
- Original repo: https://github.com/FeiElysia/ViECap
demo- contains the figures from the paperimages- contains images for inferencelogs- train and eval logsscripts- train and eval shell scriptstutorial- tutorialsCaptionsDataset.py- process datasets used for training ref:main.pyClipCap.py- translate images into descriptive relevant text (CLIP prefix for image captioning) ref:main.pyentities_extraction.py- extract the entities from each caption within the chosen datasetgenerating_prompt_ensemble.py- To evaluate the trained ViECap, first construct the vocabulary and extract the embeddings of each category in the vocabularyimages_features_extraction.py- download the image source files for the COCO and Flickr30k dataset from their official websites. Afterwards, place these files into the 'ViECap/annotations/coco/val2014' directory for COCO images and the 'ViECap/annotations/flickr30k/flickr30k-images' directory for Flickr30k images.infer_by_batch.py- utilload_annotations.py- utilmain.py- training codemodelling_opt.py- utilrequirements.txt- to install the requirements for running the coderetrieval_categories.py- utilsearch.py- utiltexts_features_extraction.py- pre-extract the training text featuresutils.py- utilvalidation.py- validation code
- Clone the repo
- Setup a conda environment and install requirements from
requirements.txtfile. Refer CapDec repo if don't know how to setup conda environment: https://github.com/DavidHuji/CapDec - Activate the conda environment and download the
annotations,evaluationandcheckpoints(optional, only required if you want to evaluate on pre-trained weights) - Dataset preparation:
- Download the karpathy splits: https://www.kaggle.com/datasets/shtvkumar/karpathy-splits/tasks
- We will be using only flickr30k for this tutorial.
- Place the
dataset_{name_of_dataset}.jsonin ./annotations/{dataset}/karpathy_split asdataset.json. Convert to dataset to list and generate train, test and validation splits by using theparse_karpathy.py - run
python entities_extraction.py - run
python texts_features_extraction.py - run
python generate_prompt_ensemble.pyafter downloading the vocabulary from the Releases of the original repo. - run (optional)
python images_features_extraction.py
- Training:
- run
bash train_flickr30k.sh 0to start training - can change hyperparameters in the shell script file
- run
- Evalution:
- Cross-domain captioning:
bash eval_{to_dataset}.sh train_{from_dataset} 0 '--top_k 3 --threshold 0.2' {14/29} - In-domain captioning:
bash eval_{dataset}.sh train_{dataset} 0 '' {14/29}
- Cross-domain captioning:
- Inference:
python infer_by_instance.py --prompt_ensemble --using_hard_prompt --soft_prompt_first --image_path ./images/instance1.jpg
- ViECap
@InProceedings{Fei_2023_ICCV,
author = {Fei, Junjie and Wang, Teng and Zhang, Jinrui and He, Zhenyu and Wang, Chengjie and Zheng, Feng},
title = {Transferable Decoding with Visual Entities for Zero-Shot Image Captioning},
booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
month = {October},
year = {2023},
pages = {3136-3146}
}
@article{fei2023transferable,
title={Transferable Decoding with Visual Entities for Zero-Shot Image Captioning},
author={Fei, Junjie and Wang, Teng and Zhang, Jinrui and He, Zhenyu and Wang, Chengjie and Zheng, Feng},
journal={arXiv preprint arXiv:2307.16525},
year={2023}
}
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent