Generalisation in Named Entity Recognition: A Quantitative Analysis
Table 14: Out of domain performance: F1 of NERC with different models
['[EMPTY]', '[BOLD] Memorisation  [BOLD] P', '[BOLD] Memorisation  [BOLD] R', '[BOLD] Memorisation  [BOLD] F1', '[BOLD] CRFSuite  [BOLD] P', '[BOLD] CRFSuite  [BOLD] R', '[BOLD] CRFSuite  [BOLD] F1', '[BOLD] Stanf  [BOLD] P', '[BOLD] Stanf  [BOLD] R', '[BOLD] Stanf  [BOLD] F1', '[BOLD] SENNA  [BOLD] P', '[BOLD] SENNA  [BOLD] R', '[BOLD] SENNA  [BOLD] F1']
The highest F1 performance is on the OntoNotes BC corpus, with similar results to the in-domain task. This is unsurprising as it belongs to a similar domain as the training corpus (broadcast conversation) the data was collected in the same time period, and it was annotated using the same guidelines. In contrast, out-of-domain results are much lower than in-domain results for the CoNLL corpora, even though they belong to the same genre as OntoNotes NW. Memorisation recall performance on CoNLL TestA and TestB with OntoNotes NW test suggest that this is partly due to the relatively low overlap in NEs between the two datasets. This could be attributed to the CoNLL corpus having been collected in a different time period to the OntoNotes corpus, when other entities were popular in the news; an example of drift Masud et al. Conversely, Stanford NER does better on these corpora than it does on other news data, e.g. ACE NW. This indicates that Stanford NER is capable of some degree of generalisation and can detect novel entity surface forms; however, recall is still lower than precision here, achieving roughly the same scores across these three (from 44.11 to 44.96), showing difficulty in picking up novel entities in novel settings.