Problems With Evaluation of Word EmbeddingsUsing Word Similarity Tasks
Table 2: Word similarity datasets.
['Dataset', 'Word pairs', 'Reference']
Word similarity evaluation is attractive, because it is computationally inexpensive and fast, leading to faster prototyping and development of word vector models. The origin of word similarity tasks can be tracked back to \newciteRubenstein:1965 who constructed a list of 65 word pairs with annotations of human similarity judgment. according to which the meaning of words is evidenced by the context they occur in. They found a positive correlation between contextual similarity and human-annotated similarity of word pairs. Since then, the lack of a standard evaluation method for word vectors has led to the creation of several ad hoc word similarity datasets. To obtain generalizable machine learning models, it is necessary to make sure that they do not overfit to a given dataset. Existing word similarity datasets are not partitioned into training, development and test sets. Therefore, optimizing the word vectors to perform better at a word similarity task implicitly tunes on the test set and overfits the vectors to the task. On the other hand, if researchers decide to perform their own splits of the data, the results obtained across different studies can be incomparable. Furthermore, the average number of word pairs in the word similarity datasets is small (≈781, cf.