VQA-LOL: Visual Question Answering under the Lens of Logic
Table 10: Training dataset distribution and sizes, for explicit training with new data. Note that training dataset sizes are consistent with the VQA dataset.
['Training Datasets', 'Proportion of datasets (%) VQA-Other', 'Proportion of datasets (%) VQA-Number', 'Proportion of datasets (%) VQA-YesNo', 'Proportion of datasets (%) Comp', 'Proportion of datasets (%) Supp', 'Training Samples']
The first question is: can models pre-trained on the VQA dataset answer logically composed questions? The question then arises: can the model answer such questions, if we explicitly train it with data that also contains logical compositions of questions? For this investigation, we construct two datasets, VQA-Compose and VQA-Supplement, by utilizing annotations from the VQA dataset and the object and caption annotations from the COCO dataset [lin2014microsoft]. We train the state-of-the-art VQA model LXMERT [tan2019lxmert] on these datasets and perform multiple experiments to test for robustness to logically composed questions. Our results show an improvement in performance on logical questions but a reduction on the VQA test set. We suspect that logical questions add another dimension to the diversity of the dataset, thereby making the overall task more challenging.