Towards a Seamless Integration of Word Senses into Downstream NLP Applications
Table 2: Classification performance at the word, sense, and supersense levels with random and pre-trained embedding initialization. We show in bold those settings that improve the word-based model.
['[BOLD] Initialization', '[BOLD] Input type', '[BOLD] Input type', '[BOLD] BBC News  [BOLD] Acc', '[BOLD] BBC News  [BOLD] F1', '[BOLD] Newsgroups  [BOLD] Acc', '[BOLD] Newsgroups  [BOLD] F1', '[BOLD] Ohsumed  [BOLD] Acc', '[BOLD] Ohsumed  [BOLD] F1']
When the embedding layer is initialized randomly, the model integrated with word senses consistently improves over the word-based model, particularly when the fine-granularity of the underlying sense inventory is reduced using supersenses (with statistically significant gains on the three datasets). This highlights the fact that a simple disambiguation of the input can bring about performance gain for a state-of-the-art classification system. Also, the better performance of supersenses suggests that the sense distinctions of WordNet are too fine-grained for the topic categorization task. However, when pre-trained representations are used to initialize the embedding layer, no improvement is observed over the word-based model. This can be attributed to the quality of the representations, as the model utilizing them was unable to benefit from the advantage offered by sense distinctions. Our results suggest that research in sense representation should put special emphasis on real-world evaluations on benchmarks for downstream applications, rather than on artificial tasks such as word similarity. In fact, research has previously shown that word similarity might not constitute a reliable proxy to measure the performance of word embeddings in downstream applications Tsvetkov et al. ; Chiu et al.