Towards Lingua Franca Named Entity Recognition with BERT
Table 3: Zero-shot F1 scores for models trained on CoNLL languages. BERT-MLk is the BERT-ML system trained on k languages. Numbers from [9] are the from the English row in their Table 1. For details on the CL, CL+LI,LI,PC,PC+LI systems, see Section Experiments: Multi-task Learning
['System', 'German', 'English', 'Spanish', 'Dutch']
The BERT-ML system obtains good results on Spanish, German, and Dutch (CoNLL) and Chinese (OntoNotes), without seeing any training data in that language. For the CoNLL languages, the performance is very close with the results of the top systems participating in the original competitions, on Dutch being 2F higher than the winning system. Also shown in the Table are the (previous) SotA result in 0-shot cross language NER on CoNLL. The BERT-SL model trained on English alone has good performance across the board. Considering the case where the target language is English while training the model using the Dutch, German and Spanish labeled data, we evaluate on the CoNLL English development set after every epoch. Averaging the overall F-score over five runs, we clearly see a downward trend of the F-score as training progresses. This is outside the norm for most statistical learning problems with labeled data, where we expect to see gradual increases in the F-score as learning progresses and then possibly a plateau and not the continued downward trajectory seen in this graph.