Bootstrapping Disjoint Datasets for Multilingual Multimodal Representation Learning
Table 7: Translation retrieval results (Recall @ 1) on the M30K 2016 test set compared to the state of the art.
['[EMPTY]', 'EN → DE', 'DE → EN']
To put our results into perspective we compare to the best approach to our knowledge as reported by \newciterotman2018bridging: DPCCA is a deep partial canonical correlation analysis method maximizing the canonical correlation between captions of the same image conditioned on image representations as a third view. Our models consistently improve upon the state-of-the-art. The baseline aligned model trained on the Multi30K data slightly outperforms the DPCCA for EN → DE retrieval, and more substantially outperforms DPCCA for DE → EN. If we train the same model with the additional c2c objective, R@1 improves by 8.0 and 12.1 points, respectively. We find that adding more monolingual English data from the external COCO data set slightly degrades retrieval performance, and that performing sentence retrieval using a model trained on the disjoint M30K German and English COCO data sets result in much lower retrieval performance. We conclude that the model that we used to estimate sentence similarity is the best-performing method known for this task on this data set, but there is room for improvement for models trained on disjoint data sets.