Problems With Evaluation of Word EmbeddingsUsing Word Similarity Tasks
Table 1: Sample word pairs along with their human similarity judgment from WS-353.
['Word1', 'Word2', 'Similarity score [0,10]']
Despite the ubiquity of word vector representations in NLP, there is no consensus in the community on what is the best way for evaluating word vectors. The most popular intrinsic evaluation task is the word similarity evaluation. In word similarity evaluation, a list of pairs of words along with their similarity rating (as judged by human annotators) is provided. The task is to measure how well the notion of word similarity according to humans is captured by the word vector representations.