Improving Pre-Trained Multilingual Models with Vocabulary Expansion
Table 3: Performance of various models on the test set of Weibo NER. \text{BERT}_{\text{zh}}: Chinese BERT pre-trained over Chinese Wikipedia. We use scripts conlleval for evaluation on NER.
['[BOLD] Approach', '[BOLD] Precision', '[BOLD] Recall', '[BOLD] F1 score']
We boost the F1 score from 59.0\% to 61.4\%. Compared to the Chinese BERT (66.9\%), there still exists a noticeable performance gap. One possible reason could be the grammatical differences between Chinese and English. As BERT uses the language model loss function for pre-training, the pre-trained Chinese BERT could better capture the language-specific information comapred to the multilingual BERT.