Towards Lingua Franca Named Entity Recognition with BERT
Table 2: Single language, multi-language and 0-shot F1 on OntoNotes. WordPiece fertility is the average number of BERT wordpieces per token in a given language.
['System', 'English', 'Arabic', 'Chinese']
For both CoNLL and OntoNotes, the model that was trained on all the languages (BERT-ML) performs better than the model trained on one language at a time (BERT-SL), in all cases, showing that the BERT-ML model is able to benefit from information coming from all the languages to improve each individual language, validating our first two hypotheses: that we can effectively build a multi-language model, and that it performs better than individual models. Relative to other published results, the Bert-ML CoNLL model obtains SotA results for Spanish and Dutch and is ranked second for German, and the Bert-ML OntoNotes model obtains SotA for Chinese and Arabic and is ranked third for English. only gets 12.6F. Looking at the wordpiece dictionary in the BERT tokenizer , we found out that Arabic is not well represented. As we show in the last row, one Arabic token gets split into 7.42 word pieces on average, while all other languages are in a range of 1.8-2.5. This has a damaging effect due to lengthening of the tokens, resulting in significantly worse behavior in Arabic than in the other languages. Despite this, the model is still able to incorporate information from English and Chinese, when trained together, and improve 1.2F over the Arabic only model.