Evaluating Multimodal Representations on Visual Semantic Textual Similarity
Table 4: The unsupervised scenario: train, validation and test results of the unsupervised models.
['[BOLD] Model', '[BOLD] Modality', '[BOLD] train  [ITALIC] ρ', '[BOLD] dev  [ITALIC] ρ', '[BOLD] test  [ITALIC] ρ']
S4SS2SSS0Px1 The unsupervised scenario. We report results over train and dev partitions for completeness, but note that none of them was used to tune the models. As it can be seen, multimodal representations consistently outperform their text-only counterparts. This confirms that, overall, visual information is helpful in the semantic textual similarity task and that image and sentence representation are complementary. For example, the bert model improves more than 13 points when visual information provided by the resnet is concatenated. glove shows a similar or even larger improvement, with similar trends for use and