An Empirical Study of Named Entity Recognition in StackOverflow
Table 5: Ablation study of our proposed Attentive-BiLSTM model on StackOverflow NER dev set.
['[BOLD] NER (20 Types)', 'P', 'R', 'F1']
The inclusion of attention induced knowledge vectors provides an absolute +8.26 F1 increase on the dev set, compared to the ELMo-BiLSTM-CRF baseline. To asses the impact of the attention network, we replace it with a simple concatenation, where we concatenate three embeddings (ELMo, segmenter vector, code recognizer vector) to create the input of the BiLSTM-CRF network. Removing the attention mechanism causes -1.84 F1 drop on our dev set. We found that the removal of segmenter vector causes a -4.65 drop in F1 score, whereas removal of code recognizer vector causes -2.25 drop in F1 score. This ablation study suggest that inclusion of attention-induced knowledge vector is quite crucial for software domain named entity recognition.