Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT
Table 4: NER tagging experiments. ♢ denotes model selection with target language dev set.
['[EMPTY]', 'en', 'nl', 'es', 'de', 'zh', 'Average (-en,-zh)']
mBERT outperforms a strong baseline by an average of 6.9 points absolute F1 and an 11.8 point absolute improvement in German with a simple one layer 0th-order CRF as a prediction function A large gap remains when transferring to distantly related languages (e.g. Chinese) compared to a supervised baseline. Further effort should focus on transferring between distantly related languages.