Improving Pre-Trained Multilingual Models with Vocabulary Expansion
Table 7: POS tagging accuracy (%) for OOV tokens and non-OOV tokens on the Universal Dependencies v1.2 dataset, where the OOV/non-OOV are defined at word level with the original BERT vocabulary.
['[BOLD] Lang', 'BERT non-OOV', 'BERT OOV', '\\text{BERT}_{\\text{oov}} non-OOV', '\\text{BERT}_{\\text{oov}} OOV']
First, we argue that it is essential to alleviate the OOV issue in multilingual settings. In the original BERT, the accuracy of OOV words is much lower than that of non-OOV words, and we significantly boost the accuracy of OOV words with the expanded BERT. All these results indicate that the overall improvement mostly comes from the reduction of OOV.