XTREME: A Massively Multilingual Multi-task Benchmarkfor Evaluating Cross-lingual Generalization
Table 3: The cross-lingual transfer gap (lower is better) of different models on xtreme tasks. The transfer gap is the difference between performance on the English test set and the average performance on the other languages. A transfer gap of 0 indicates perfect cross-lingual transfer. For the QA datasets, we only show EM scores. The average gaps are computed over the sentence classification and QA tasks.
['Model', 'XNLI', 'PAWS-X', 'XQuAD', 'MLQA', 'TyDiQA-GoldP', 'Avg', 'POS', 'NER']
Cross-lingual transfer gap For a number of representative models While powerful models such as XLM-R reduce the gap significantly compared to mBERT for challenging tasks such as XQuAD and MLQA, they do not have the same impact on the syntactic structured prediction tasks. On the classification tasks, the transfer learning gap is lowest, indicating that there may be less headroom for progress on these tasks. The use of MT reduces the gap across all tasks. Overall, a large gap remains for all approaches, which indicates much potential for work on cross-lingual transfer.