UniVSE: Robust Visual Semantic Embeddings viaStructured Semantic Representations1footnote 11footnote 1 This paper is to be presented in the non-archival track of NAACL 2019 Spatial Language Understanding (SpLU) & Grounded Communication for Robotics (RoboNLP) workshops only. The full version of this paper can be accessed in https://arxiv.org/abs/1904.05521v1, which has been published in the proceedings of CVPR 2019.
Table 2: The mAP performance on the unified text-to-image retrieval task. Please refer to the text for details.
['Task', 'obj', 'attr', 'rel', 'obj (det)', 'sum']
We extend the word-to-scene retrieval used by Shi et al. In this task, models receive queries of different semantic levels, including single words (e.g., “Clock.”), noun phrases (e.g., “White clock.”), relational phrases (e.g., “Clocks on wall”) and full sentences. For all baselines, the texts of different types as treated as full sentences. Ablation study: contrastive learning of components. We evaluate the effectiveness of using contrastive samples for different semantic components. The same notation applies to other models. The UniVSE trained with a certain type of contrastive examples (e.g., UniVSEobj with contrastive nouns) consistently improves the retrieval performance of the same type of queries (e.g., retrieving images from a single noun). UniVSE trained with all kinds of contrastive samples performs best in overall and shows a significant gap w.r.t. other baselines.