Towards Lingua Franca Named Entity Recognition with BERT
Table 6: CoNLL test results using ML, LI, CL, CL+LI and different frozen layers. It is possible that better numbers can be achieved with fewer than 8 frozen layers. The best results seem to occur around 3 frozen layers given table 5 but 8 frozen layers were chosen based on earlier, undiscussed experiments using uncased multilingual BERT with English.
['model', '#frz', 'German', 'English', 'Spanish', 'Dutch']
Here, the -1 value in the #frz columns denote that no layers are frozen during training and are equivalent to the standard setting. At least for Dutch and English, we see that CL, CL+LI, LI, PC help as secondary tasks and improve over the ML(-1) baseline. However, there are no secondary tasks that help for Spanish. And only the LI task helps German. The ML settings corresponds to the standard zero-shot setting but where the labeled training data comes from just one language. Here, the best scores occur when 3 layers are frozen.