SimVerb-3500: A Large-Scale Evaluation Set of Verb Similarity
Table 2: An overview of word similarity evaluation benchmarks. All is the current best reported score on each data set across all models (including the models that exploit curated knowledge bases and hand-crafted lexical resources, see supplementary material). Text denotes the best reported score for a model that learns solely on the basis of distributional information. All scores are Spearman’s ρ correlations.
['[BOLD] Eval set', 'IAA-1', 'IAA-2', 'All', 'Text']
There is another fundamental limitation of all current verb similarity evaluation resources: automatic approaches have reached or surpassed the inter-annotator agreement ceiling. For instance, while the average pairwise correlation between annotators on SL-222 is Spearman’s ρ correlation of 0.717, the best performing automatic system reaches ρ=0.727 SimVerb-3500 does not inherit this anomaly (see Tab. A complementary measure would smooth individual annotator effects. For this aim, our IAA-2 (mean) measure compares the average correlation of a human rater with the average of all the other raters. SimVerb-3500 obtains ρ = 0.84 (IAA-1) and ρ = 0.86 (IAA-2), a very good agreement compared to other benchmarks (see Tab.