On the Cross-lingual Transferability of Monolingual Representations
Table 3: XQuAD results (F1). We bold the best result in each section and underline the overall best result.
['[EMPTY]', '[EMPTY]', 'mBERT', 'en 88.9', 'es 75.5', 'de 70.6', 'el 62.6', 'ru 71.3', 'tr 55.4', 'ar 61.5', 'vi 69.5', 'th 42.7', 'zh 58.0', 'hi 59.2', 'avg 65.0']
Similar to our findings in the XNLI experiment, the vocabulary size has a large impact on JointMulti, and JointPair models with disjoint vocabularies perform the best. The gap between MonoTrans and joint models is larger, but MonoTrans still performs surprisingly well given the nature of the task. We observe that learning language-specific position embeddings is helpful in most cases, but completely fails for Turkish and Hindi. In contrast to results on previous tasks, adding adapters to allow a transferred monolingual model to learn higher level abstractions in the new language significantly improves performance, resulting in a MonoTrans model that is comparable to the best joint system.