XTREME: A Massively Multilingual Multi-task Benchmarkfor Evaluating Cross-lingual Generalization
Table 4: Accuracy of mBERT on POS tag trigrams and 4-grams in the target language dev data that appeared and did not appear in the English training data. We show the performance on English, the average across all other languages, and their difference.
['[EMPTY]', 'trigram, seen', 'trigram, unseen', '4-gram, seen', '4-gram, unseen']
Generalization to unseen tag combinations and entities We analyze possible reasons for the less successful transfer on structured prediction tasks. The Universal Dependencies dataset used for POS tagging uses a common set of 17 POS tags for all languages, so a model is not required to generalize to unseen tags at test time. However, a model may be required to generalize to unseen tag combinations at test time, for instance due to differences in word order between languages. We gauge how challenging such generalization is by computing a model’s accuracy for POS tag n-grams in the target language dev data that were not seen in the English training data. We observe the largest differences in performance for unseen trigrams and 4-grams, which highlights that existing cross-lingual models struggle to transfer to the syntactic characteristics of other languages. For NER, we estimate how well models generalize to unseen entities at test time. We compute mBERT ’s accuracy on entities in the target language dev data that were not seen in the English training data. We observe the largest difference between performance on seen and unseen entities for Indonesian and Swahili. Isolating for confounding factors such as entity length, frequency, and Latin script, we find the largest differences in performance for Swahili and Basque. Together, this indicates that the model may struggle to generalize to entities that are more characteristic of the target language. We show the detailed results for both analyses in the appendix.