Learning Visually Grounded Sentence Representations
Table 3: Thorough investigation of the contribution of grounding, ensuring equal number of components and identical architectures, on the variety of sentence-level semantic benchmark tasks. STb=SkipThought-like model with bidirectional LSTM+max. 2×STb-1024=ensemble of 2 different STb models with different initializations. GroundSent is STb-1024+Cap2Cap/Img/Both. We find that performance improvements are sometimes due to having more parameters, but in most cases due to grounding.
['[BOLD] Model', '[BOLD] MR', '[BOLD] CR', '[BOLD] SUBJ', '[BOLD] MPQA', '[BOLD] MRPC', '[BOLD] SST', '[BOLD] SNLI', '[BOLD] SICK']
Grounded models outperform the STb-1024 model (which uses much more data— the Toronto Books Corpus is much larger than COCO) in all cases, often already without concatenating the textual modality. The ensemble of two STb-1024 models performs better than the individual one, and so does the higher-dimensional one. In the cases of CR and MRPC (F1), it appears that improved performance is due to having more data or ensemble effects. For the other datasets, grounding clearly yields better results. These results indicate that grounding does indeed capture qualitatively different information, yielding better universal sentence representations.