SQuINTing at VQA Models: Introspecting VQA Models with Sub-Questions
Table 1: Results on held out VQA-introspect val set for (1) Consistency metrics along the four quadrants described in Section 5 and Consistency and Attention Correlation metrics as described in Section 5 (metrics), and (2) Overall and Reasoning accuracy. The Reasoning accuracy is obtained by only looking at the number of times the main question is correct (M✓ S✓ + M✓ S✗) ignoring repetitions of the main question due to multiple sub-questions.
['Method', 'Consistency Metric M✓\xa0S✓ ↑', 'Consistency Metric M✓\xa0S✗ ↓', 'Consistency Metric M✗\xa0S✓ ↓', 'Consistency Metric M✗\xa0S✗ ↓', 'Consistency Metric Consistency% ↑', 'Consistency Metric Consistency% (balanced) ↑', 'Consistency Metric Attn Corr ↑', 'VQA Accuracy Overall ↑', 'VQA Accuracy Reasoning (M✓\xa0S✓ + M✓\xa0S✗) ↑']
(SOTA as of 2018) The overall accuracy of the model is 64.95%, while accuracy on Reasoning questions is 69.61%. We note that for 28.27% of the cases, the model is inconsistent, i.e., it answered the main question correctly, but got the sub question wrong. Further, we observe that 9.02% of the times the Pythia model gets all the sub questions wrong when the main question is right – i.e., it seems to be severely wrong on its perception and using other paths (shortcuts or biases) to get the Reasoning question right . such that it now attends to the same regions for main questions and associated sub-questions. We also report a few additional metrics: Consistency refers to how often the model predicts the sub-question correctly given that it answered the main question correctly, while Consistency (balanced) reports the same metric on a balanced version of the sub-questions (to make sure models are not exploiting biases to gain consistency). Attention Correlation refers to the correlation between the attention embeddings of the main and sub-question. Finally, we report Overall accuracy (on the whole evaluation dataset), and accuracy on the Reasoning split (Reasoning Accuracy). Note that our approach does not require sub-questions at test time. We use λ1=0.1 and λ2=1 with a learning rate of 0.01 in our experiments. Correspondingly, our confidence that it actually learned the necessary concepts when it answered Reasoning questions correctly should increase.