Are All Languages Created Equal in Multilingual BERT?
Table 3: Statistic of four low resource languages.
['[EMPTY]', 'lv', 'af', 'mn', 'yo']
We have established that mBERT does not perform well in low-resource languages. Is this because we are relying on a multilingual model that favors high-resource over low-resource languages? To answer this question we train mono-lingual BERT models on several low resource languages with different hyperparameters. Since pretraining a BERT model from scratch is computationally intensive, we select four low resource languages: Latvian (lv), Afrikaans (af), Mongolian (mn), and Yoruba (yo).