Large-Scale Adversarial Training for Vision-and-Language Representation Learning
Table 4: Probing analysis of the attention heads in pre-trained UNITER and Villa models.
['Model', 'Visual Coreference (Flickr30k) scene', 'Visual Coreference (Flickr30k) clothing', 'Visual Coreference (Flickr30k) animals', 'Visual Coreference (Flickr30k) instruments', 'Visual Coreference (Flickr30k) vehicles', 'Visual Relation (Visual Genome) on', 'Visual Relation (Visual Genome) standing in', 'Visual Relation (Visual Genome) wearing', 'Visual Relation (Visual Genome) holding', 'Visual Relation (Visual Genome) covering', 'Ave.']
Probing Analysis Pre-trained models are expected to learn intricate knowledge about multimodality correlations, such as visual coreference (i.e., region-phrase alignment) and visual relation (i.e., region-region interaction). To provide a more direct measurement on how well our adversarial pre-trained model captures such multimodal signals, we conduct a probing analysis following Cao et al. We consider five most common visual coreference types in Flickr30k Entities Plummer et al. learned by pre-trained models. Results show that Villa presents higher attention weights across all the ten categories (0.223 vs. 0.195 on average), indicating a higher probability of identifying those relations.