A negative case analysis of visual grounding methods for VQA
Table 1: Results on VQA-CPv2 and VQAv2 datasets for the baseline UpDn, visual sensitivity enhancement methods (HINT and SCR) and our own regularization method, including the published (pub.) numbers.
['[EMPTY]', '[BOLD] VQA-CPv2 Train', '[BOLD] VQA-CPv2 Test', '[BOLD] VQAv2 Train', '[BOLD] VQAv2 Val']
In our first experiment we studied how irrelevant visual cues performed compared to relevant ones. We fine-tune the model with irrelevant cues defined as: Sirrelevant:=(1−Sh), where, Sh represents the human-based importance scores. are not necessarily from looking at relevant regions. In our next experiment we studied how random visual cues performed with HINT and SCR. We assign random importance scores to the visual regions: Srand∼uniform(0,1). We test two variants of randomness: Fixed random regions, where Srand are fixed once chosen, and Variable random regions, where Srand are regenerated every epoch. The performance improves even when the importance scores are changed every epoch, indicating that it is not even necessary to look at the same visual regions. We compare the training accuracies to analyze the regularization effects. We hypothesize that degrading performance on the train set helps forget linguistic biases, which in turn helps accuracy on VQA-CPv2’s test set but hurts accuracy on VQAv2’s val set. 100% of the dataset. Confirming our hypothesis, all variants of our model achieve near state-of-the-art results, solidifying our claim that the performance gains for recent methods come from regularization effects.