An Empirical Study of Named Entity Recognition in StackOverflow
Table 6: Performance of the NER systems on dev and test set of our annotated StackOverflow (SO) NER corpus. The best performance is boldfaced.
['[BOLD] NER(20 Types) Dev', '[BOLD] NER(20 Types) Fine-tuned BERT', 'P 46.42', 'R 79.57', 'F1 58.64']
We assessed the impact of BERT devlin2018bert on the software domain NER. The original BERT model is pre-trained in the general newswire domain corpus. Hence, its data distribution is different from StackOverflow domain. To adapt BERT with our task domain, we fine-tuned it with our annotated corpus. In our experiment, we found that fine-tuned BERT token classifier can outperform the out-domain BiLSTM-CRF-ELMo model. Note that, to get this performance, we needed to train the BERT Token Classifier model much longer than the recommended steps (158 epochs instead of 5 epochs). This reveals that training the in-domain contextual embedding is crucial part of the NER model.