Rethinking Generalization of Neural Models: A Named Entity Recognition Case Study
Table 4: The breakdown performance on CoNLL and WNUT datasets with different pre-training strategies, which is based on the LSTM as sentence encoder and CRF as the decoder. “Rand” represents the word representations are randomly initialized. “Overall” denotes the F1 score on the whole test set and the names of the last five columns correspond to ρ definition in Tab.3.
['[BOLD] Datasets', '[BOLD] Embed-layer  [BOLD] Char', '[BOLD] Embed-layer  [BOLD] Word', '[BOLD] Entity Coverage Rate  [BOLD] Overall', '[BOLD] Entity Coverage Rate 1', '[BOLD] Entity Coverage Rate (0.5,1)', '[BOLD] Entity Coverage Rate (0,0.5]', '[BOLD] Entity Coverage Rate  [ITALIC] C≠0', '[BOLD] Entity Coverage Rate  [ITALIC] C=0']
In general, the part of test entities with high performance are usually the ones that appear in the training set. By contrast, if the test entity is unseen, it will achieve a lower performance. 2) No matter what level (character or word) pre-trained embeddings are introduced, the performances of unseen entities are largely improved. 3) Comparing two different levels of pre-trained methods, ELMo and FLAIR achieve better performances on unseen entities but have not shown significant gain on seen entities. 4) Compared to Rand, CNN shows its superior performance on the prediction of unseen entities. 5) For different parts of the test set, we find C≠0 is the most challenging part (even for the state-of-the-art model), followed by C=0 and (0,0.5]. Interestingly, on the CoNLL dataset, we find that if the test entity is labeled as a different category in the training set, it will be more difficult to learn compared with entities which have not been seen in the training set. 6) We find that the character- and the word-level pre-trained embeddings are complementary to each other. Combining these two types of pre-trained knowledge will further improve the performance by a considerable margin.