To lemmatize or not to lemmatize: how word normalisation affects ELMo performance in word sense disambiguation
Table 5: F1 scores for target words from RUSSE’18 with significant differences between lemma-based and token-based models
['[BOLD] Word', '[BOLD] Tokens', '[BOLD] Lemmas', '[BOLD] STD']
We decided to analyse more closely those words for which the difference in the scores between lemma-based and token-based models was statistically significant. By ‘significant’ we mean that the scores differ by more that one standard deviation (the largest standard deviation value in the two sets was taken). We can see that among 18 words in the dataset only 3 exhibit significant improvement in their scores when moving from tokens to lemmas in the input data. It shows that even though the overall F1 scores for the Russian data have shown the plausibility of lemmatization, this improvement is mostly driven by a few words. It should be noted that these words’ scores feature very low standard deviation values (for other words, standard deviation values were above 0.1, making F1 differences insignificant). Such a behaviour can be caused by more consistent differentiation of context for various senses of these 3 words. For example, with the word кабачок ‘squash / small restaurant’, the contexts for both senses can be similar, since they are all related to food. This makes the WSD scores unstable.