Bootstrapping Disjoint Datasets for Multilingual Multimodal Representation Learning
Table 5: We train the aligned plus disjoint model with c2c loss and add the full pseudopair set (+pseudo) or the filtered versions (+pseudo 25% and +pseudo 75%) is added as an extra data set. The model is either re-trained from scratch or fine-tuned (+fine-tune). We also report the result of training the aligned plus disjoint model with the synthetic translations (+Translation).
['[EMPTY]', 'English I → T', 'English T → I', 'English Sum', 'German I → T', 'German T → I', 'German Sum', 'Sum(Sum)']
Aligned plus disjoint We generate pseudopairs using a model trained on M30K English-German data with the c2c objective and the English COCO data set. Overall we find that pseudopairs improve performance, however, we do not achieve the best results for English and German in the same conditions. The best results for German are to filter at 25% percentile and apply fine-tuning, while for English the best results are without filtering or fine-tuning. The best overall model is trained with all the pseudopairs with fine-tuning, according to the Sum of the Sum-of-recall scores across both English and German. The performance across both data sets is increased from 723.5 to 728.2 using the pseudopair method.