An Empirical Study of Named Entity Recognition in StackOverflow
Table 7: Performance of our named entity tagger on annotated GitHub (GH) dataset of readme files and issue posts. The NER taggers are all trained on our annotated StackOverflow corpus.
['[BOLD] GitHub NER', 'P', 'R', 'F1']
An interesting question is how well our StackOverflow named entity tagger performs on other programming-related text. To evaluate this, we randomly sampled 143 readme files and 741 issue reports from GitHub. This dataset comprises 6,510 sentences that contain 10,963 entities. The data was annotated by two of our annotators and inter-rater agreement was 0.68. We observe a drop from 78.41 to 62.69 in F1 due to domain differences. However, we believe that our tagger still achieves sufficient performance to be useful for some applications on GitHub. We found GitHub-domain ELMo-BiLSTM model provides +2.1 increase of F1 score. We did not observe any significant gain from the code recognizer and segmenter vectors. However, we would like to emphasize that these vectors are extracted respectively from the code recognizer and segmenter models, both of which are fine tuned on the StackOverflow data. Our code token Recognizer is specifically designed based on the StackOverflow domain to capture characteristics of code-related entities which are not frequent for GitHub texts, such as, Function, Class, and Variable mentions. This suggests that prior knowledge learned by code recognizer and segmenter is only helpful when using in-domain data.