Word Usage Similarity Estimation with Sentence Representations and Automatic Substitutes
Table 2: Spearman ρ correlation of different sentence and word embeddings on the Usim dataset using different context window sizes (cw). For BERT and ELMo, top refers to the top layer, and av refers to the average of layers (3 for ELMo, and the last 4 for BERT).
['Context', 'Embeddings', 'Correlation']
We observe that target word BERT embeddings give best performance in this task. Selecting a context window around (or including) the target word does not always help, on the contrary it can harm the models. Context2vec sentence representations are the next best performing representation, after BERT, but their correlation is much lower. The simple GloVe-based SIF approach for sentence representation, which consists in applying dimensionality reduction to a weighted average of GloVe vectors of the words in a sentence, is much superior to the simple average of GloVe vectors and even better than doc2vec sentence representations, obtaining a correlation comparable to that of USE.