Are All Languages Created Equal in Multilingual BERT?
Table 4: Monolingual BERT on four languages with different hyperparameters. Underscore denotes best within monolingual BERT and bold denotes best among all models. Monolingual BERT underperforms mBERT in most cases. “-” denotes same as base case.
['Model Size', 'Vocabulary', 'Max Length', 'lv NER', 'lv POS', 'lv Parsing (LAS/UAS)', 'af NER', 'af POS', 'af Parsing (LAS/UAS)', 'mn NER', 'yo NER']
The model with smaller sequence length performs best for monolingual BERT and outperforms the base model in 5 out of 8 tasks and languages combination. The model with smaller vocabulary has mixed performance in the low resource languages (mn, yo) but falls short for (relatively) higher resource languages (lv, af). Finally, the smaller model underperforms the base model in 5 out of 8 cases. In conclusion, the best way to pretrain BERT with a limited amount of computation for low resource languages is to use a smaller sequence length to allow a larger batch size. Future work could look into a smaller self-attention span with a restricted transformer Vaswani et al. The Afrikaans-Dutch BERT improves more compared to Latvian-Lithuanian, possibly because Dutch is much larger than Afrikaans, as compared to Latvian and Lithuanian. These experiments suggest that pairing linguistically related languages can benefit representation learning and adding extra languages can further improve the performance as demonstrated by mBERT. It echos the finding of \newcitelample2019cross where multilingual training improves uni-directional language model perplexity for low resource languages. Concurrent work shows similar findings as the performance of low resource languages (Urdu and Swahili) improves on XNLI when more languages are trained jointly then decrease with an increasing number of languages Conneau et al. However, they do not consider the effect of language similarity.