Inverse Visual Question Answering with Multi-Level Attentions
Table 2: Performance comparison of our main model variants.
['Method', 'BLEU-1', 'BLEU-2', 'BLEU-3', 'BLEU-4', 'METEOR', 'ROUGE-L', 'CIDEr']
The baseline is the powerful Show Attend and Tell The model is compared to ours in the context of using dynamic attention to focus on different salient regions at each time step. The proposed model incorporates additional semantic information S based on the object and attribute labels. It uses S to produce enhanced visual features Ev and obtain att0 through dual guiding attention. To investigate whether and how much this semantic information contributes to the major performance gain for the proposed model, we create a variant of the proposed model by dropping the semantic information S and dual guiding attention features att0. It actually outperforms the iVQA model in terms of BLEU-1, BLEU-2, BLEU-3 and METEOR. This validates the visual features extracted at the object-level salient regions more suitable for the task of iVQA than the visual features extracted with conventional deep networks. Nevertheless, the full model still beats the variant in terms of all evaluation metrics. This suggests that the use of semantic information and dual guiding attention is beneficial for iVQA. Enhancing visual features with semantic information that has the same textual modality with the answer and generated partial question is useful for our model to precisely attend to corresponding objects, and consequently improve the overall performance.