Crisscrossed Captions: Extended Intramodal and Intermodal Semantic Similarity Judgments for MS-COCO
Table 7: Text ↔ Text and Image ↔ Image retrieval performance on MSCOCO 5k test set with CxC annotations.
['[BOLD] Model', '[BOLD] Text → Text R@1', '[BOLD] Text → Text R@5', '[BOLD] Text → Text R@10', '[BOLD] Text → Text med r', '[BOLD] Image → Image R@1', '[BOLD] Image → Image R@5', '[BOLD] Image → Image R@10', '[BOLD] Image → Image med r']
USEmling-large provides a strong baseline for Text → Text, while InceptionV3 and ResNet-152 are used as baselines for Image → Image. The dual encoder model DE T2T achieves the strongest performance on Text → Text. DE I2T provides the best performance on Image → Image. For both modalities, the best performing DE model outperforms the baseline by a sizable margin. MultitaskI2T+T2T also achieves much better retrieval performance than the baselines for both Text → Text and Image → Image. However, multitask training performs slightly worse than the more specialized DE models for each modality. The CxC annotations capture missing examples in the first two cases, and the last shows that there are still more positive pairs that remain unassociated in the data. In the case of the clock tower, all of the top retrieved items would have been counted as misses in retrieval evaluation, but are correctly counted as hits with CxC.