An Empirical Study of Named Entity Recognition in StackOverflow
Table 2: Evaluation and ablation study of code token recognition on 1000 random words from StackOverflow annotated corpus.
['[BOLD] Code Token Recognizer', 'P', 'R', 'F1']
For each input token, we identify its character-level patterns. Then these patterns are utilized to find whether the token is a part of code entity or not. As an example, consider two words w1= ‘list’, w2= ‘list()’. Although w2 contains w1 as its prefix, the trailing ‘()’ in w2 indicate that it is a code token, whereas w1 is an ambiguous token and will be considered as a non-code token. To capture such n-gram characteristics of the tokens, we extract unigram word and 6-gram character probabilities as features. These features are extracted from two language models (LMs). These LMs are separately trained on Gigaword corpus and all the code-snippets of 91 million tokens in StackOverflow 10-year archive. We also train FastText joulin2016fasttext word embeddings using these code-snippets. We transform each numerical value from the language model into a 10-dimensional vector by Gaussian binning method EMNLP-2018-Maddela, and then feed them into a linear layer. The output of this layer is then combined with FastText embeddings and passed through another hidden layer followed by a single output node with sigmoid activation function. We use binary cross entropy loss to train this model and predict the token as code if the output score is >0.5.