Compared models without using Visual Genome about ban-vqa HOT 1 CLOSED

Following the corresponding papers, they both use Visual Genome for data augmentation (bottom-up) or feature learning (both).

In Section 4.1.1 of Anderson et al. (CVPR 2018), they mentioned as:

When training the VQA model, we augment the VQA v2.0 training data with Visual Genome question and answer pairs provided the correct answer is present in model’s answer vocabulary. This represents about 30% of the available data, or 485K questions.

For the image feature learning, both approaches use bottom-up attention image features which are pretrained using Visual Genome (as we did). Bottom-up (Anderson et al., 2018) uses Visual Genome for feature learning and training, while Counter is for feature learning. Therefore we cannot say that Counter is not available to the external data, Visual Genome. Besides this aspect, the Counter model is mainly to compare with Number score (Although we can show that BAN with Counter outperforms Original Counter without the augmentation of Visual Genome. See the table below).

In the 30% of Visual Genome, the ratio of counting question is approximately 11.5%.

Without Visual Genome augmentation in training and evaluate on test-dev split,
BAN (69.52 -> 69.14), BAN+Glove (69.66 -> 69.27), BAN+Glove+Counter (70.04 -> 69.58).

VQA 2.0 Test-dev results without Visual Genome augmentation:

                  | Overall | Yes/no  |  Number |  Other
------------------+---------+---------+---------+--------
BAN               |  69.14  |  85.03  |  49.80  |  59.97
BAN+Glove         |  69.27  |  85.19  |  49.94  |  60.08
BAN+Glove+Counter |  69.58  |  85.18  |  52.76  |  60.06

from ban-vqa.