Aligning Visual Regions and Textual Concepts for Semantic-Grounded Image Representations
Table 4: Ablation analysis of the proposed approach. As we can see, incorporating MIA-refined image representation from a single modality can also lead to overall improvements.
['Methods', 'BLEU-1', 'BLEU-2', 'BLEU-3', 'BLEU-4', 'METEOR', 'ROUGE', 'CIDEr', 'SPICE']
Ablation Study. As the deployment of MIA inevitably introduces information from the other modality, we conduct ablation studies to investigate whether the improvement is derived from the well-aligned and integrated image representations or the additional source information. Thanks to the mutual iterative attention process, “Visual Attention w/ IN” and “Concept Attention w/ IN” can pay attention to integrated visual features and textual concepts, respectively. This frees the decoder from associating the unrelated original features in each domain, which may explain for the improvements. The performance in terms of SPICE and CIDEr is further elevated when TN and IN are combined. The progressively increased scores demonstrate that the improvements indeed come from the refined semantic-grounded image representations produced by MIA, rather than the introduction of additional information.