SimVerb-3500: A Large-Scale Evaluation Set of Verb Similarity
Table 5: Spearman’s ρ correlation between human judgments and model’s cosine similarity based on pair relation type. Relations are based on WordNet, and included in the dataset. The classes are of different size, 373 pairs with no relation (NR), 306 synonym (SYN) pairs, and 800 hyper/hyponym (HYP) pairs. Frequencies of member verbs are distributed in a similar way.
['[BOLD] Model', 'NR', 'SYN', 'HYP']
Lexical Relations SimVerb-3500 contains relation annotations (e.g., antonyms, synonyms, hyper-/hyponyms, no relation) for all pairs extracted automatically from WordNet. Evaluating per-relation subsets, we observe that some models draw their strength from good performance across different relations. Others have low performance on these pairs, but do very well on synonyms and hyper-/hyponyms. Selected results of this analysis are in Tab. ’s ρ may be problematic with certain categories, e.g., with antonyms. It evaluates pairs according to their ranking; for antonyms the ranking is arbitrary - every antonym pair should have a very low rating, hence they are not included in Tab. A similar effect occurs with highly ranked synonyms, but to a much lesser degree than with antonyms.