Interpretability Analysis for Named Entity Recognition to Understand System Predictions and How They Can Improve
Table 5: Performance of GloVe word-based BiLSTM-CRF. The first two panels were trained on the Original English CoNLL 03 training data and tested on the original English CoNLL 03 test data and the WikiGold data. The last panel was trained and tested on the respective splits of MUC-6.
['[BOLD] System', '[BOLD] CoNLL P', '[BOLD] CoNLL R', '[BOLD] CoNLL F1', '[BOLD] Wikipedia P', '[BOLD] Wikipedia R', '[BOLD] Wikipedia F1', '[BOLD] MUC-6 P', '[BOLD] MUC-6 R', '[BOLD] MUC-6 F1']
In the human evaluation we saw some mixed results, with some room for improvement on 23% of the errors on one side and some errors that seem to be due to over-reliance on context on the other. This leads us to wonder if a more sophisticated approach that decides how to combine cues would lead to a better system. We perform an oracle experiments where the oracle knows which of the systems is correct. If neither is correct, it defaults to one of the system. The default system in each case is the one listed first. Row 1 in the table shows that an oracle combination of the forward context only and backward context only does much better than the system which looks at both these contexts together. The gains are about 15, 20 and 24 points F1 on CoNLL, Wiki and MUC-6 respectively. This improvement likely captures many of the examples that people got right but not the context-only system.