Bootstrapping Disjoint Datasets for Multilingual Multimodal Representation Learning
Table 4: A disjoint model is trained on the De+COCO datasets and used to generate pseudopairs. Then the full pseudopair set (+pseudo) or the filtered versions (+pseudo 25% and +pseudo 75%) are used as an extra data set to either re-train the moddel from scratch or fine-tune the original De+COCO model (+fine-tune).
['[EMPTY]', 'German I → T', 'German T → I', 'German Sum']
We generate pseudopairs using the disjoint bilingual model trained on the German M30K and the English COCO. Line 2 shows that using the full pseudopair set and re-training the model does not lead to noticeable improvements. However, line 3 shows that performance increases when we train with all pseudopairs and fine-tuning the original disjoint bilingual model. Filtering the pseudopairs at either the 25% and 75% percentile is detrimental to the final performance.