Does BERT Make Any Sense? Interpretable Word Sense Disambiguation with Contextualized Embeddings
Table 1: Properties of our datasets. For the test sets (Te), we do not report k′ since they are not used as kNN training instances.
['[EMPTY]', '[BOLD] SE-2 (Tr)', '[BOLD] SE-2 (Te)', '[BOLD] SE-3 (Tr)', '[BOLD] SE-3 (Te)', '[BOLD] S7-T7 (coarse)', '[BOLD] S7-T17 (fine)', '[BOLD] SemCor', '[BOLD] WNGT']
We conduct our experiments with the help of four standard WSD evaluation sets, two lexical sample tasks and two all-words tasks. The all-words tasks of SemEval 2007 Task 7 The two sets differ in granularity: While ambiguous terms in Task 17 are annotated with one WordNet sense only, in Task 7 annotations are coarser clusters of highly similar WordNet senses. For training of the all-words tasks, we use a) the Princeton WordNet gloss corpus (WNGT) From this, we can observe that the SE-2 and SE-3 training data sets, which were published along with the respective test sets, provide many more examples per word and sense than SemCor or WNGT. However, we observe a major performance drop of our approach for the two all-words WSD tasks in which no training data is provided along with the test set. For S7-T7 and S7-T17, the content and structure of the out-of-domain SemCor and WNGT training datasets differ drastically from those in the test data, which prevents yielding near state-of-the-art results. In fact, similarity of contextualized embeddings largely relies on semantically and structurally similar sentence contexts of polysemic target words. Hence, the more example sentences can be used for a sense, the higher are the chances that a nearest neighbor expresses the same sense.