An Empirical Study of Named Entity Recognition in StackOverflow
Table 3: Performance of our segmentation model on our StackOverflow NER dev set.
['[BOLD] Segmenter', 'P', 'R', 'F1']
Word Frequency Embedding: The first of these features is inspired by the observation that code tokens tend to have lower frequency as compared to non-code tokens. Code tokens have an average frequency of 1.47 in our data, compared to 7.41 for non-code tokens. Ambiguous tokens , that can either be part of code or non-code entities, were found to have a much higher average frequency of 92.57. We thus use the Gaussian-based binning method EMNLP-2018-Maddela to represent word frequencies for input to the segmenter.