Combining Language and Visionwith a Multimodal Skip-gram Model
Table 5: Subjects’ preference for nearest visual neighbour of words in Kiela et al. (2014) vs. random pictures. Figure of merit is percentage proportion of significant results in favor of nearest neighbour across words. Results are reported for the whole set, as well as for words above (concrete) and below (abstract) the concreteness rating median. The unseen column reports results when words exposed to direct visual evidence during training are discarded. The words columns report set cardinality.
['all', '[ITALIC] global 48%', '[ITALIC] |words| 198', '[ITALIC] unseen 30%', '[ITALIC] |words| 127']
We quantify performance in terms of proportion of words for which the number of votes for the nearest neighbour picture is significantly above chance according to a two-tailed binomial test. We set significance at p<0.05 after adjusting all p-values with the Holm correction for running 198 statistical tests. MMSkip-gram-A representation is meaningfully related to the word. As expected, this is more often the case for concrete than abstract words. Still, we also observe a significant preference for the model-predicted nearest picture for about one fourth of the abstract terms. Whether a word was exposed to direct visual evidence during training is of course making a big difference, and this factor interacts with concreteness, as only two abstract words were matched with images during training. When we limit evaluation to word representations that were not exposed to pictures during training, the difference between concrete and abstract terms, while still large, becomes less dramatic than if all words are considered.