Improving Reliability of Word Similarity Evaluationby Redesigning Annotation Task and Performance Measure
Table 2: The hyponym-hypernym dataset agreement (0.646) compares favorably with the agreement for nouns pairs reported by Hill et al hill2015simlex (0.612), and it is much higher than the correlation score of the word2vec model. Notice that useful insights can be gained from the per-comparison-type analysis, like the model’s difficulty to distinguish hyponym-hypernym pairs from other relations.
['[EMPTY]', 'Hyp.', 'Cohyp.']
We used two methods of measuring: the first is the conventional way (Spearman correlation), and the second is the scoring method we described in the previous section, which we used to measure general and per-comparison-type scores.