Towards Lingua Franca Named Entity Recognition with BERT
Table 5: Zero-shot BERT multilingual on CONLL TEST data. #frz corresponds to number of frozen layers during training. -1 is where nothing is frozen and is the standard setting. 0 is where only the embedding is frozen. 1 is embedding and first layer is frozen and so forth. 12 is where all bert layers are frozen and only the output layer is tuned.
['#frz', 'German', 'English', 'Spanish', 'Dutch']
For both the dev and the test, we show that the best results are obtained when some layers are frozen with German, English and Dutch. Spanish is anomalous in this sense where no amount of freezing can outperform no freezing (i.e. -1). Furthermore, the scores for Spanish test are higher than for Spanish dev, which is highly unusual in that validation development scores are generally higher than held out test scores for most models. On dev, German performs best when 5 layers are frozen, English 7, and Dutch 5. On test, German performs best when 3 layers are frozen, English 7 and Dutch 3. #frz 11 and 12 are presented in the tables to show that having too few layers to tune is detrimental in the zero-shot setting. There, the authors were able to achieve 91 f-score on the dev set without any fine tuning. This strongly supports our assumption that multilingual BERT is well-aligned and seems to generate a common representation for the 100+ language in a shared space.