An Empirical Study of Named Entity Recognition in StackOverflow
Table 4: Performance of the baseline BiLSTM-CRF tagger with varying input representations on StackOverflow (SO) NER dev set. Contextualized ELMo word representations show a clear benefit relative to GloVe, when trained on StackOverflow data.
['[BOLD] NER (20 Types)', 'P', 'R', 'F1']
We observe that using in-domian contextualized ELMo word embeddings leads to very significant performance gains relative to context-independent embeddings (GloVe). We observe a significantly large performance gain of +13.64 F1 when using in-domain ELMo representations over in-domain GloVe. Compared to off-the-shelf ELMo, in-domain training results in a +15.71 absolute increase in F1. While investigating the cause of such high performance gain from in-domain ELMo, we found that the improvement is largely coming from the ability to handle ambiguous and unseen tokens. We observe that 38% of tokens appearing inside gold spans are unseen and 43% are ambiguous (appear with more than one tag in the training data). Using in-domain contextualized ELMo word representations results in a significant improvement for both unseen and ambiguous entity tokens, but the improvement for unseen tokens is especially notable (41% → 70% accuracy). This suggests that ELMo’s character-based representation, in combination with contextual information from the entire input sentence is especially effective at handling unseen tokens in the social programming domain. This also indicates that our annotated corpus provides an attractive test-bed for measuring the impacts of contextual word representations.