Evaluating Multimodal Representations on Visual Semantic Textual Similarity
Table 6: Contribution of images over text representations on test.
['[BOLD] Scenario', '[BOLD] Repr', '[BOLD] text', '[BOLD] mmodal', '[BOLD] Diff', '[BOLD] E.R']
The contribution is consistent through all text-based representations. We measure the absolute difference (Diff) and the error reduction (E.R) of each textual representation with the multimodal counterpart. For the comparison we chose the best text model for each representation. As expected we obtain the largest improvement (22−26% E.R) when text-based unsupervised models are combined with image representations. Note that unsupervised models are not learning anything about the specific task, so the more information in the representation, the better. In the case of use and vse++ the improvement is significant but not as large as the purely unsupervised models. The best text-only representation is the one fine-tuned on a multimodal task, VSE++, which is noteworthy, as it is better than a textual representation fine-tuned in a text-only inference task like USE.