Being Negative but Constructively: Lessons Learnt from Creating Better Visual Question Answering Datasets
Table 9: Test accuracy (%) on VQA2-2017val.
['Method', 'IoU', 'QoU', 'IoU +QoU']
For both datasets, we conduct the same experiments as in Sect. 5.3 of the main text using the MLP-based models. Meanwhile, models that can observe either images and answers (MLP-IA) or questions and answers (MLP-QA) fail to predict as good as the model that observe all three sources of information. These empirical observations meet our expectation and again verify the effectiveness of our proposed methods for creating decoys. Besides, we also perform a more in-depth experiment on VQA2, removing triplets with Yes /No as the target. , we see that the overall performance for each model decreases as the dataset becomes more challenging on average. Specifically, the model that observes question and answer on VQA2− performs much worse than that on VQA2 (37.2% vs. 48.1%).