Visual Word2Vec (vis-w2v): Learning Visually Grounded Word Embeddings Using Abstract Scenes
Table 5: Performance on the common sense task of [34] using 78k real images with text baseline at 72.2, initialized from w2v-coco.
['[ITALIC] K', 'vis-w2v80k', 'vis-w2v-coco']
We first train vis-w2v80k with NH=200 and use the fc7 features as is, i.e. without PCA, in the Sents configuration (see Sec. Further, to investigate the complementarity between visual semantics learnt from real and visual scenes, we initialize vis-w2v-coco with vis-w2v-coco80k, i.e., we learn the visual semantics from the real scenes and train again to learn from abstract scenes. Comparing the improvements from real images (best AP of 73.7%) to those from abstract scenes (best AP of 74.8%), we see that that abstract visual features capture visual semantics better than real images for this task. It if often difficult to capture localized semantics in the case of real images. For instance, extracting semantic features of just the primary and secondary objects given a real image, is indeed a challenging detection problem in vision. On the other hand, abstract scene offer these fine-grained semantics features therefore making them an ideal for visually grounding word embeddings.