Combining Language and Visionwith a Multimodal Skip-gram Model
(b)
['[BOLD] Model', '[ITALIC] ρ']
A natural corollary of this idea is that visually-grounded representations of abstract concepts should be more diverse: If you think of dogs, very similar images of specific dogs will come to mind. You can also imagine the abstract notion of freedom, but the nature of the related imagery will be much more varied. Recently, \newciteKiela:etal:2014 have proposed to measure abstractness by exploiting this very same intuition. However, they rely on manual annotation of pictures via Google Images and define an ad-hoc measure of image dispersion. We conjecture that the representations naturally induced by our models display a similar property. In particular, the entropy of our multimodal vectors, being an expression of how varied the information they encode is, should correlate with the degree of abstractness of the corresponding words. et al. set. To test the hypothesis quantitatively, we measure the correlation of entropy and concreteness on the 200 words in the \newciteKiela:etal:2014 set. (MMSkip-gram-B*) achieve very high correlation (but, interestingly, not MMSkip-gram-B). This is further evidence that multimodal learning is grounding the representations of both concrete and abstract words in meaningful ways.