Counterfactual VQA: A Cause-Effect Look at Language Bias
Table 3: Experimental results on VQA-CP v2 and VQA v2. Best and second best numbers are highlighted in each column. The results on VQA v2 test-std are obtained via our reimplementation.
['Training set Test set', '[EMPTY]', 'VQA-CP v2 train VQA-CP v2 test', 'VQA-CP v2 train VQA-CP v2 test', 'VQA-CP v2 train VQA-CP v2 test', 'VQA-CP v2 train VQA-CP v2 test', 'VQA-CP v2 train VQA v2 test-std', 'VQA-CP v2 train VQA v2 test-std', 'VQA-CP v2 train VQA v2 test-std', 'VQA-CP v2 train VQA v2 test-std', 'VQA v2 train VQA v2 val', 'VQA v2 train VQA v2 val', 'VQA v2 train VQA v2 val', 'VQA v2 train VQA v2 val']
Can CF-VQA overcome language bias? The out-distribution setting aims to evaluate whether VQA models effectively reduce language bias. With a deep look at the question type, we find that the improvement on “Yes/No” questions is extremely large (from ∼70% to ∼90%), which indicates that the language bias affects variously on different types of questions. Is CF-VQA robust? The in-distribution and independent-distribution settings are conducted to evaluate whether VQA models over-correct language bias. Note that LM clark2019don achieves a competitive performance on VQA-CP v2 test split (∼52%) with an additional language entropy penalty (LM+H). However, the accuracy drops significantly by ∼7% on the in-domain setting (i.e., VQA v2 val split), which indicates that the entropy penalty forces the model to over-correct the language bias, especially on “Yes/No” questions.