Reducing Language Biases in Visual Question Answering with Visually-Grounded Question Encoder
Figure 4: Performance comparison between the baseline and VGQE corresponding to some of the question-types in the VQA-CPv2 test set. VGQE reduces the learning of unwanted question pattern-answer correlations and improves the performance. All numbers are VQA accuracy values.
['Question Type', 'Baseline', '+VGQE']
The baseline model shows comparatively poor performance than the model with VGQE. Since most of the time, the baseline model with a traditional question encoder tries to memorize the language biases in the train set. We can see that, in all the question-types, the incorporation of VGQE reduces such biases and improves performance. To further clarify this, in Fig. It is clear from the visualizations that the baseline model learns the language biases in the train set and predicts the answer without considering the image. The incorporation of VGQE helps to reduce such biases and to answer by grounding the question onto the image. E.g., from Fig. Most of such questions are linked to the answer “no” in the train set; in other words, there is a strong correlation between the “Do you” language pattern and the answer “no.” We can see that most of the time, the baseline model predicts the answer ”no” upon seeing the pattern “Do you,” without considering the image. This indicates that the baseline model relies on the strong correlation between the language pattern “Do you” and the answer “no,” rather than looking at the image. In contrast, the incorporation of VGQE helps to reduce such learning of biases and to predict the answer by considering the image as well.