Moving Down the Long Tail of Word Sense Disambiguationwith Gloss Informed Bi-encoders
Table 2: F1-score (%) on the MFS, LFS, and zero-shot subsets of the ALL evaluation set. Zero-shot examples are the words and senses (respectively) that do not occur in the training data. The balanced BEM system, BEM-bal, is considered in Section 6.2.
['[EMPTY]', '[BOLD] MFS', '[BOLD] LFS', '[BOLD] Zero-shot  [BOLD] Words', '[BOLD] Zero-shot  [BOLD] Senses']
To better understand these overall results, we break down performance across different sense frequencies. We split examples from the aggregated ALL evaluation set into mentions with the most frequent sense (MFS) of the target word and mentions that are labeled with the other, less frequent senses (LFS) of that word. We also consider zero-shot performance for both unseen words and unseen senses by evaluating performance on examples that are not observed during training. We compare our model against the frozen BERT-base baseline and EWISE Kumar et al. This balanced BEM model achieves an F1 score of 77.6, underperforming the standard BEM on the aggregated ALL evaluation set. By breaking down the balanced model performance, the balanced BEM outperforms the standard BEM on LFS examples, but suffers from worse performance on the more common MFS examples. We also find that this balancing during training slightly improves performance on both zero-shot words and senses.