Extending Multilingual BERT to Low-Resource Languages
Table 2: Comparison between B-BERT and E-MBERT: We compare B-BERT vs E-MBERT training protocols. Both the models uses same target language monolingual data. We compare the performances on languages that are not in M-BERT, so that E-MBERT doesn’t make use of M-BERT’s additional Wikipedia data.
['[BOLD] Lang', 'B-BERT', 'Extend']
Moreover, B-BERT is trained for 2M steps for convergence, while E-MBERT requires only 500k steps. We believe that this advantage comes for the following reason: E-MBERT makes use of the better multilingual model M-BERT, which potentially contains languages that help transfer knowledge from English to target, while B-BERT can only leverage English data. For example, in the case of Sinhala and Uyghur, a comparatively high-resource related language like Tamil and Turkish in M-BERT can help the E-MBERT learn Sinhala and Uyghur better.