A negative case analysis of visual grounding methods for VQA
Table A4: VQA accuracy per answer-type on VQACPv2 test set.
['[EMPTY]', 'Overall', 'Yes/No', 'Num', 'Other']
HINT/SCR and our regularizer show large gains in ‘Yes/No’ questions. We hypothesize that the methods help forget linguistic priors, which improves test accuracy of such questions. In the train set of VQACPv2, the answer ‘ no’ is more frequent than the answer ‘ yes’, tempting the baseline model to answer ‘yes/no’ questions with ‘no’. However, in the test set, answer ‘yes’ is more frequent. Regularization effects caused by HINT/SCR and our method cause the models to weaken this prior i.e., reduce the tendency to just predict ‘no’, which would increase accuracy at test because ‘yes’ is more frequent in the test set. Next, all of the methods perform poorly on ‘Number (Num)’ answer type, showing that methods find it difficult to answer questions that are most reliant on correct visual grounding such as: localizing and counting objects. Finally, we do not observe large improvements in ‘Other’ question type, most likely due to the large number of answers present under this answer type.