Bilinear Attention Networks
Table 4: Test split results for Flickr30k Entities. We report the average performance of our three randomly-initialized models (the standard deviation of R@1 is 0.17). Upper Bound of performance asserted by object detector is shown. † box size and color information are used as additional features. ‡ semantic segmentation, object detection, and pose-estimation is used as additional features. Notice that the detectors of Hinami and Satoh (2017) and ours (Anderson et al., 2017) are based on Faster RCNN (Ren et al., 2017), pre-trained using Visual Genome dataset (Krishna et al., 2016).
['Model', 'Detector', 'R@1', 'R@5', 'R@10', 'Upper Bound']
Flickr30k Entities. The task is to localize a corresponding box for each entity. In this way, visual grounding of textual information is quantitatively measured. This metric is called Recall@1. If K predictions are permitted to find at least one correction, it is called Recall@K. The upper bound of performance depends on the performance of object detection if the detector proposes candidate boxes for the prediction. Performance. Our bilinear attention map to predict the boxes for the phrase entities in a sentence achieves new state-of-the-art with 69.69% for Recall@1. Compared to Query-Adaptive RCNN, the parameters of our object detector are fixed and only used to extract 10-100 visual features and the corresponding box proposals.