Aligning Visual Regions and Textual Concepts for Semantic-Grounded Image Representations
Table 7: Variation of model performance under the breakdown of SPICE F-scores. We can find that the w/ TN has a higher Object scores than the baselines, and the w/ IN reaches better scores in Count and Size. As we can see, incorporating Mutual Iterative Attention (MIA) directly on the baselines, leads to overall improvements.
['Methods', 'SPICE All', 'SPICE Objects', 'SPICE Attributes', 'SPICE Relations', 'SPICE Color', 'SPICE Count', 'SPICE Size']
As we can see, the IN, TN and MIA promotes the baselines over almost all sub-categories. Especially, the IN is good at associating related parts in the image, which is demonstrated by the increased scores in Count and Size. and the TN collects relevant textual concepts, providing comprehensive context that is detailed in objects. Encouragingly, when incorporating IN and TN at the same time, i.e., w/ MIA, the advantages of the IN and TN are united to produce a balanced improvement. It proves the effectiveness of our approach.