Sparse and Structured Visual Attention
Table 3: Automatic evaluation of VQA on VQA-2.0. Sparse-TVmax and soft-TVmax correspond to using sparsemax or softmax on the image self-attention and TVmax on the output attention. Other models use softmax or sparsemax on self-attention and output attention.
['[EMPTY]', 'Att. to image', 'Att. to bounding boxes', 'Test-Dev Yes/No', 'Test-Dev Number', 'Test-Dev Other', 'Test-Dev Overall', 'Test-Standard Yes/No', 'Test-Standard Number', 'Test-Standard Other', 'Test-Standard Overall']
As can be seen in the results presented in Table 3 the models using TVMAX in the output attention layer outperform the models using softmax and sparsemax. Moreover, the results are slightly superior when the sparsemax transformation is used in the self-attention layers of the decoder. It can also be observed that, when using sparsemax both in the self-attention layers of the decoder and in the output attention mechanism, the accuracy is superior than when using softmax. Thus, having sparse attention mechanisms in the self-attention layers is beneficial, but the biggest improvement is obtained when using TVMAX in the output attention. This corroborates our intuition that selecting the features of contiguous regions of the image leads to a better understanding of the image, and, consequently, better accuracy. Additionally, when using bounding box features, sparsemax outperforms softmax, showing that selecting only the bounding boxes of the relevant objects leads to a better answering capability. We can also see that combining ResNet features with bounding box features improves results. Moreover, the model using TVMAX in the final attention layer achieves the highest accuracy, showing that features obtained using the TVMAX transformation are a better complement to bounding box features.