Exploration on Grounded Word Embedding: Matching Words and Images with Image-Enhanced Skip-Gram Model
Table 1: Image to Word vector matching results (find word given image). P@x means if one word in the top x predicted words is correct, we think it is correct.
['Model', 'P@1', 'P@5', 'P@10']
The task is to find the most relevant word given an image. We assume that if the images and corresponding words can be matched based on their vectors, they are more likely to be in the same hyper-plane, which is the objective of our paper. From the results we can see that our IESG model can achieve very high results. P@10 even reaches 85.8%, which means that the 85.8% the times top 10 results contain the correct one. This shows that the image vectors can be easily connected with the correct word embeddings, indicating a strong correlation between the image vectors and the word embeddings. On the contrary, when directly minimizing the L2 distance between image vector and word embedding, the P@1 performance is much worse than our proposed model (28.8%). We assume that this is because L2 distance model only drags the distribution of image vectors to the distribution of word embeddings while fails to make use of the context of the word.