Lessons learned in multilingual grounded language learning
Table 3: German Image-to-text (I→T) and text-to-image (T→I) retrieval results on the comparable part of Multi30K, measured by Recall at 1, 5 at 10. Typewriter font shows performance of two sets of symmetric and asymmetric models from gella2017image.
['[EMPTY]', '[EMPTY]', 'I→T R@1', 'I→T R@5', 'I→T R@10', 'T→I R@1', 'T→I R@5', 'T→I R@10']
Once again, our implementation outperforms \newcitegella2017image, and this is likely due to the different visual features and max-of-hinges loss. However, our Bilingual model with the additional c2c objective performs the best for German, whereas \newcitegella2017image reports the overall best results for the monolingual baseline VSE. Their models that use the asymmetric similarity function are clearly better than the Monolingual OE model. In general, the results from \newcitegella2017image indicate the benefits of bilingual joint training, however, they do not find a clear pattern between the model configurations across languages. In our implementation, we only focused on the symmetric cosine similarity function and found a systematic pattern across both languages: bilingual training improves results on all performance metrics for both languages, and the additional c2c objective always provides further improvements.