This repository is the official implementation of the Prophet, a two stage framework designed to prompt GPT-3 with answer heuristics for knowledge-based VQA. In stage one, we train a vanilla VQA model on a specific knowledge-based VQA dataset and extract two types of complementary answer heuristics from the model: answer candidates and answer-aware examples. In stage two, answer heuristics are used to prompt GPT-3 to generate better answers. Prophet significantly outperforms existing state-of-the-art methods on two datasets, delivering 61.1% on OK-VQA and 55.7% on A-OKVQA. Please refer to our paper for details.
April 28, 2023
- Add pretrained and finetuned models on A-OKVOA.
March 10, 2023
- Training and testing codes of the two-stages Prophet framework.
- Pretrained and finetuned models on OK-VOA.
To conduct the following experiments, a machine with at least 1 RTX 3090 GPU, 50GB memory, and 300GB free disk space is recommended. We strongly recommend using an SSD drive to guarantee high-speed I/O.
Following software is needed:
- Python >= 3.9
- Cuda >= 11.3
- Pytorch >= 12.0
- what you can find in environment.yml
We recommend downloading Anaconda first and then creating a new environment with the following command:
$ conda env create -f environment.ymlThis command will create a new environment named prophet with all the required packages. To activate the environment, run:
$ conda activate prophetBefore running the code, prepare two folders: datasets and assets. The datasets folder contains all the datasets and features used in this project, and the assets folder contains the pre-computed resources and other intermediate files (you can use them to skip some early experiment steps and save time).
First, download the datasets and assets. Then put the datasets and assets folder in the root directory of this project. Download MSCOCO 2014 and 2017 images from here (you can skip MSCOCO 2017 if you only experiments on OK-VQA) and put them in the datasets folder. Run the following command to extract the features of the images:
$ bash scripts/extract_img_feats.shAfter that, the datasets and assets folder will have the following structure:
Click to expand
datasets
├── aokvqa
│ ├── aokvqa_v1p0_test.json
│ ├── aokvqa_v1p0_train.json
│ └── aokvqa_v1p0_val.json
├── coco2014
│ ├── train2014
│ └── val2014
├── coco2014_feats
│ ├── train2014
│ └── val2014
├── coco2017
│ ├── test2017
│ ├── train2017
│ └── val2017
├── coco2017_feats
│ ├── test2017
│ ├── train2017
│ └── val2017
├── okvqa
│ ├── mscoco_train2014_annotations.json
│ ├── mscoco_val2014_annotations.json
│ ├── OpenEnded_mscoco_train2014_questions.json
│ └── OpenEnded_mscoco_val2014_questions.json
└── vqav2
├── v2_mscoco_train2014_annotations.json
├── v2_mscoco_val2014_annotations.json
├── v2_OpenEnded_mscoco_train2014_questions.json
├── v2_OpenEnded_mscoco_val2014_questions.json
├── v2valvg_no_ok_annotations.json
├── v2valvg_no_ok_questions.json
├── vg_annotations.json
└── vg_questions.json
We've also provided a tree structure of the entire project in misc/tree.txt.
We provide bash scripts for each stage of the Prophet framework. You can find them in the scripts directory. There are two common arguments you should take care of when running each script:
--task: specify the task (i.e., the target dataset) you want to deal with. The available options areok(training ontrainset of OK-VQA and evaluating on thetestset of OK-VQA),aok_val(training ontrainset of A-OKVQA and evaluating on thevalset of A-OKVQA) andaok_test(training ontrainset andvalset of A-OKVQA and evaluating on thetestset of A-OKVQA);
Note that although Prophet uses VQA v2 datasets for pre-training, there are slight differences in how the datasets are used for different tasks (ok, aok_val, and aok_test), as detailed in configs/task_to_split.py. This means that different pre-training commands need to be followed for each task.
--version: specify the version name of this run. This name will be used to create a new folder in theoutputsdirectory to store the results of this run.
Notice that you can omit any arguments when invoking following scripts, it will then use the default arguments written in the script files.
Before running any script, you can also update the configuration files (*.yml) in the configs directory to change hyperparameters.
Take OK-VQA for example, Propht consists of two phases, stage one for training a vanilla VQA model and extracting answer heuristics, and stage two for prompting GPT-3 with answer heuristics.
At this stage, we train an improved MCAN model (check the paper for detail description) through pretraning on VQA v2 and finetuning on target dataset. Multiple GPUs are supported by setting --gpu 0,1,2,3 (for example). Run pretraining step with commands:
$ bash scripts/pretrain.sh \
--task ok --version okvqa_pretrain_1 --gpu 0We've provided a pretrained model for OK-VQA here. Then, run finetuning step with commands:
$ bash scripts/finetune.sh \
--task ok --version okvqa_finetune_1 --gpu 0 \
--pretrained_model outputs/okvqa_pretrain_1/ckpts/epoch_13.pklAll epoch checkpoints are saved in outputs/ckpts/{your_version_name}. We've also provided a finetuned model for OK-VQA here. You may pick one to generate answer heuristics by run following command:
$ bash scripts/heuristics_gen.sh \
--task ok --version okvqa_heuristics_1
--gpu 0 --ckpt_path outputs/okvqa_finetune_1/ckpts/epoch_6.pkl
--candidate_num 10 --example_num 100The extracted answer heuristics will be stored as candidates.json and examples.json in outputs/results/{your_version_name} directory.
You may need the candidates.json and examples.json files generated in the former stage to step into this stage. Or you can just skip stage one, and use the files of answer heuristics we provided in assets. Especially, the candidates.json and examples.json files for OK-VQA are answer_aware_examples_okvqa.json and candidates_okvqa.json. To prompt GPT-3 with answer heuristics and generate better answers, run the following command:
$ bash scripts/prompt.sh \
--task ok --version okvqa_prompt_1 \
--examples_path outputs/results/okvqa_heuristics_1/examples.json \
--candidates_path outputs/results/okvqa_heuristics_1/candidates.json \
--openai_key sk-xxxxxxxxxxxxxxxxxxxxxxThe result file will be stored as result.json in outputs/results/{your_version_name} directory.
We also provide example scripts for the aok_val and aok_test modes on A-OKVQA.
Click to expand
Similary, for task of aok_val, run pretraining step with commands:
$ bash scripts/pretrain.sh \
--task aok_val --version aokvqa_val_pretrain_1 --gpu 0We've provided a pretrained model for aok_val here.Then, run finetuning step with commands:
$ bash scripts/finetune.sh \
--task aok_val --version aokvqa_val_finetune_1 --gpu 0 \
--pretrained_model outputs/aokvqa_val_pretrain_1/ckpts/epoch_13.pklAll epoch checkpoints are saved in outputs/ckpts/{your_version_name}.We've also provided a finetuned model for aok_val here. You may pick one to generate answer heuristics by run following command:
$ bash scripts/heuristics_gen.sh \
--task aok_val --version aokvqa_val_heuristics_1
--gpu 0 --ckpt_path outputs/aokvqa_val_finetune_1/ckpts/epoch_6.pkl
--candidate_num 10 --example_num 100The extracted answer heuristics will be stored as candidates.json and examples.json in outputs/results/{your_version_name} directory.
You may need the candidates.json and examples.json files generated in the former stage to step into this stage. Or you can just skip stage one, and use the files of answer heuristics we provided in assets. Especially, the candidates.json and examples.json files for aok_val are examples_aokvqa_val.json and candidates_aokvqa_val.json. To prompt GPT-3 with answer heuristics and generate better answers, run the following command:
$ bash scripts/prompt.sh \
--task ok --version okvqa_val_prompt_1 \
--examples_path outputs/results/aokvqa_val_heuristics_1/examples.json \
--candidates_path outputs/results/aokvqa_val_heuristics_1/candidates.json \
--captions_path assets/captions_aokvqa.json \
--openai_key sk-xxxxxxxxxxxxxxxxxxxxxxThe result file will be stored as result.json in outputs/results/{your_version_name} directory.
For task of aok_val, run pretraining step with commands:
$ bash scripts/pretrain.sh \
--task aok_test --version aokvqa_test_pretrain_1 --gpu 0We've provided a pretrained model for aok_test here. Then, run finetuning step with commands:
$ bash scripts/finetune.sh \
--task aok_test --version aokvqa_test_finetune_1 --gpu 0 \
--pretrained_model outputs/aokvqa_test_pretrain_1/ckpts/epoch_13.pklAll epoch checkpoints are saved in outputs/ckptss/{your_version_name}.We've also provided a finetuned model for aok_test here. You may pick one to generate answer heuristics by run following command:
$ bash scripts/heuristics_gen.sh \
--task aok_test --version aokvqa_test_heuristics_1
--gpu 0 --ckpt_path outputs/aokvqa_test_finetune_1/ckpts/epoch_6.pkl
--candidate_num 10 --example_num 100The extracted answer heuristics will be stored as candidates.json and examples.json in outputs/results/{your_version_name} directory.
You may need the candidates.json and examples.json files generated in the former stage to step into this stage. Or you can just skip stage one, and use the files of answer heuristics we provided in assets. Especially, the candidates.json and examples.json files for aok_test are examples_aokvqa_test.json and candidates_aokvqa_test.json. To prompt GPT-3 with answer heuristics and generate better answers, run the following command:
$ bash scripts/prompt.sh \
--task ok --version okvqa_test_prompt_1 \
--examples_path outputs/results/aokvqa_test_heuristics_1/examples.json \
--candidates_path outputs/results/aokvqa_test_heuristics_1/candidates.json \
--captions_path assets/captions_aokvqa.json \
--openai_key sk-xxxxxxxxxxxxxxxxxxxxxxThe result file will be stored as result.json in outputs/results/{your_version_name} directory.
For the task of ok and aok_val whose annotations are available, the scores are automatically computed after finetuning and prompting. You can also evaluate the result files that outputted after finetuning or prompting, by run
$ bash scripts/evaluate_file.sh \
--task ok --result_path outputs/results/okvqa_prompt_1/result.jsonUsing the corresponding result files and evaluation script above, we obtain the accuracies in the following table, respectively.
| OK-VQA | A-OKVQA (val) | A-OKVQA (test) | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
|
For the task of aok_test, you need to submit the result file to the A-OKVQA Leaderboard to evaluate the result.
If you use this code in your research, please cite our paper:
@inproceedings{shao2023prompting,
title={Prompting Large Language Models with Answer Heuristics for Knowledge-based Visual Question Answering},
author={Shao, Zhenwei and Yu, Zhou and Wang, Meng and Yu, Jun},
booktitle={Computer Vision and Pattern Recognition (CVPR)},
pages={14974--14983},
year={2023}
}This project is licensed under the Apache License 2.0 - see the LICENSE file for details.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent