Learning Visually Grounded Sentence Representations
Table 1: Retrieval (higher is better) results on COCO, plus median rank (MEDR) and mean rank (MR) (lower is better). Note that while this work underwent review, better methods have been published, most notably VSE++ Faghri et al. (2017).
['[EMPTY]', 'COCO5K Caption Retrieval', 'COCO5K Caption Retrieval', 'COCO5K Caption Retrieval', 'COCO5K Caption Retrieval', 'COCO5K Caption Retrieval', 'COCO5K Image Retrieval', 'COCO5K Image Retrieval', 'COCO5K Image Retrieval', 'COCO5K Image Retrieval', 'COCO5K Image Retrieval']
Although it is not the primary aim of this work to learn a state-of-the-art image and caption retrieval system, it is important to first establish the capability of our system to do what it is trained to do. We compare our system against several well-known approaches, namely Deep Visual-Semantic Alignments (DVSA) (FV) Klein et al. Order Embeddings (OE) Vendrov et al. The fact that the system outperforms Order Embeddings on caption retrieval suggests that it has a better sentence encoder. Cap2Both does not work as well on this task as the image-only case, probably because interference from the language signal makes the problem harder to optimize. The results indicate that the system has learned to predict image features from captions, and captions from images, at a level exceeding or close to the state-of-the-art on this task.