M{}^{3}P: Learning Universal Representations via Multitask Multilingual Multimodal Pre-training
Table 2: Multilingual image-text retrieval results on Multi30K and MSCOCO. The metric is the mean Recall (mR). Each bold number indicates the best mR score in that column. As MULE and SMALR are using different dev/test splits of MSCOCO compared with all the other models, we highlight these numbers in blue color. We report the mR results of Unicoder-VL on the en datasets, as it is pre-trained based on the same image-caption corpus (i.e. Conceptual Captions), as M3P did.
['Model', 'Multi30K en', 'Multi30K de', 'Multi30K fr', 'Multi30K cs', 'MSCOCO en', 'MSCOCO ja', 'MSCOCO zh']
We study the impacts of different fine-tuning strategies, including w/o fine-tune: apply M3P to all test sets directly without fine-tuning; w/ fine-tune on en: fine-tune M3P for English and then apply the fine-tuned model to all test sets; w/ fine-tune on each: fine-tune M3P for each language li and then apply the fine-tuned model to the test sets of li; w/ fine-tune on all: fine-tune M3P for all languages using the merged labeled data and then apply the fine-tuned model to all test sets. The same sentence in different languages may capture complementary information to help improve performance. We also compare with Unicoder-VL , which is pre-trained using the same image-caption corpus (i.e. Conceptual Captions), but for English only. Although M3P performs a bit worse than Unicoder-VL on English, it can obtain comparable results on all the other languages, which verifies its strong transfer capability. A possible reason is that the employment of xMLM task and a larger vocabulary for 100 languages. Considering that applying machine translation to translate English to all other supported languages is not general and limited to a large amount of translators, we leave this as an option for future work.