Generalisation in Named Entity Recognition: A Quantitative Analysis
Table 15: Out-of-domain performance for unseen vs. seen NEs: F1 of NERC with different models
['[EMPTY]', '[EMPTY]', '[BOLD] CRFSuite  [BOLD] P', '[BOLD] CRFSuite  [BOLD] R', '[BOLD] CRFSuite  [BOLD] F1', '[BOLD] Stanf  [BOLD] P', '[BOLD] Stanf  [BOLD] R', '[BOLD] Stanf  [BOLD] F1', '[BOLD] SENNA  [BOLD] P', '[BOLD] SENNA  [BOLD] R', '[BOLD] SENNA  [BOLD] F1', '[BOLD] All  [BOLD] P', '[BOLD] All  [BOLD] R', '[BOLD] All  [BOLD] F1']
The last section’s observation of NERC performance being lower for unseen NEs also generally holds true in this out-of-domain setting. The macro average over F1 for the in-domain setting is 76.74% for seen NEs vs. 53.76 for unseen NEs, whereas for the out-of-domain setting the F1 is 56.10% for seen NEs and 47.73% for unseen NEs.