Interpretability Analysis for Named Entity Recognition to Understand System Predictions and How They Can Improve
Table 1: Performance of GloVe word-level BiLSTM-CRF and BERT. All rows are for the former and only the last two rows for BERT. Local context refers to high precision constraints due to sequential CRF. Non-local context refers to the entire sentence. No document level context is included. The first two panels were trained on the Original English CoNLL 03 training data and tested on the original English CoNLL 03 test data and the WikiGold data. The last panel was trained and tested on the respective splits of MUC-6.
['[BOLD] System', '[BOLD] CoNLL P', '[BOLD] CoNLL R', '[BOLD] CoNLL F1', '[BOLD] Wikipedia P', '[BOLD] Wikipedia R', '[BOLD] Wikipedia F1', '[BOLD] MUC-6 P', '[BOLD] MUC-6 R', '[BOLD] MUC-6 F1']
We evaluate these systems on the CoNLL 2003 and MUC-6 data. Our goal is to quantify how well the models can work if the identity of the word is not available, and to compare that to situations in which only the word identity is known. Additionally, we evaluate the systems trained on CoNLL data on the Wikipedia dataset Balasuriya et al. The last line in the table, BI context + word corresponds to the system presented in Huang et al. Finally, we experiment with BERT. Full BERT improves in F1 over the biLSTM as reported in the original paper. Context-only BERT does not perform as well and performs better or worse than context-only LSTM depending on the corpora. These results show BERT isn’t always better at capturing contextual clues. While it is better in certain cases, it also misses these clues in certain cases where the biLSTM is correct. Its higher performance on the full dataset could be a result of having a better pretraining data or learning the subword structure. We leave this analysis for the future. In this work, we only focus on extent of context vs word utilization by the systems. Moreover, even when one system performs better than the other, both are correct on different examples. We randomly sampled 200 examples from CoNLL 03 where the context-only LSTM was correct (Sample-C) and another 200 where it was incorrect (Sample-I). Context-only BERT is correct on only 71.5% examples in Sample-C. However, it is also able to correctly recognize the entity type in 53.22% of the cases in Sample-I.