Learning Deep Structure-Preserving Image-Text Embeddings
Table 3: Phrase localization results on Flickr30K Entities using Fast-RCNN features. We use 100 EdgeBox proposals, for which the recall upper bound is R@100=76.91.
['Methods', 'R@1', 'R@5', 'R@10', 'mAP(all)']
First, we treat phrase localization as the problem of retrieving instances of a query phrase from a set of region proposals extracted from test images, and report Recall@K, or the percentage of queries for which a correct match has rank of at most K (a region proposal is considered to be a correct match if it has IOU of at least 0.5 with the ground-truth bounding box for that phrase). Second, we report average precision (AP) of ranking bounding boxes for each phrase in the test images that contain that phrase, following nonmaximum suppression. We can see that including the structure terms generally gives better results than excluding them, though the effects of turning on each term separately do not differ too much. In large part, this is because of the limited number of structure-preserving constraint triples for each view. In the Flickr30K Entities training set, for all 130K pairs, there are around 70K unique phrases and 80K regions described by a single phrase. This means, that, for most phrases/regions, there are no more than two corresponding regions/phrases. For the pre-trained model without using negative mining, our deep embedding has comparable results with CCA on Recall@5 and Recall@10, but lower results on Recall@1.