VQA-LOL: Visual Question Answering under the Lens of Logic
Table 5: Test-set results. VQA results are reported on the ‘test-standard’ splits. Note that our model performance is close to SOTA, but significantly better at logical robustness 10. * Since the MCAN code uses a fixed vocabulary for evaluation, it is unable to evaluate our VQA-Supplement questions which contain questions created from COCO captions. These are test-dev scores, # MCAN does not report their test-standard single model scores. 44footnotemark: 4
['Model (Training Data)', 'Test-Standard Accuracy (%) ↑ Yes-No', 'Test-Standard Accuracy (%) ↑ Number', 'Test-Standard Accuracy (%) ↑ Other', 'Test-Standard Accuracy (%) ↑ Overall', 'Validation Accuracy (%) ↑ Comp', 'Validation Accuracy (%) ↑ Supp']
Finally, we evaluate our models on the VQA Test-Standard dataset. This shows that logical connectives in questions can be learned while not degrading the overall performance on the original VQA test set. We would also like to highlight that our best accuracy on the VQA test set (71.19%) is within ∼1.5% of the state-of-the-art on all three types of questions.