Lessons learned in multilingual grounded language learning
Table 4: R@10 retrieval results on the comparable part of Multi30K. Bi-translation is trained on 29K translation pair data; bi-comparable is trained by downsampling the comparable data to 29K.
['[EMPTY]', 'English I→T', 'English T→I', 'German I→T', 'German T→I']
We now study whether the model can be trained on either translation pairs or independently collected bilingual captions. \newcitegella2017image only conducted experiments on independently collected captions. However, it is known that humans have equally strong preference for translated or independently collected captions of images frank_elliott_specia_2018, which has implications for the difficulty and cost of collecting training data. Our baseline is a Monolingual model trained on 29K single-captioned images in the translation portion of Multi30K. The Bi-translation model is trained on both German and English, with shared parameters. However, the additional c2c loss degrades performance here. This could be because we only have one caption per image in each language and it is easier to find a relationship between these views of the translation pairs.