Weak Supervision helps Emergence of Word-Object Alignment and improves Vision-Language Tasks
Table 2: Impact of the proposed object-word alignment weak supervision on the VQA task. The presented results are calculated on the GQA [14] test-std split.
['Models', 'Consistency', 'Accuracy']
In order to quantify the impact of the our object-word alignment weak supervision on the VQA task, we evaluate the two versions of our model, with and without the proposed loss, on the GQA dataset. One may observe that the proposed weak supervision boosts the accuracy with +5.6 points. Moreover, when we focus on the metric which explicitly measures the reasoning capacity of the model, namely the consistency, our weakly-supervised alignment allows to gain more than +10 points. This demonstrates that, by enforcing the model to explicitly align words with visual objects, we obtained a finer multimodal representation.