Better Text UnderstandingThrough Image-To-Text Transfer
Table 6: Comparison of training on a word-level vs sentence-level.
['[BOLD] Model', '[BOLD] images2014', '[BOLD] images2015', '[BOLD] COCO-Test', '[BOLD] Pin-Test']
Previous methods for transferring knowledge from images to text focused on improving the word-level embeddings. A sentence representation could then be created by combining them. In our work, we learn sentence embeddings as a whole, but the best performing text encoder turned out to be BOW. This raises the following question: could the model perform equally well if we train it on word-level, and then only combine word embeddings during inference? This effect should be studied further, but while separately training word embeddings forces each of them to be close to the corresponding images, training at the sentence level gives the opportunity to have the word embeddings become complementary, each of them explaining a part of the image, and capturing co-occurences.