SimVerb-3500: A Large-Scale Evaluation Set of Verb Similarity
Table 1: Example verb pairs from SimVerb-3500.
['Pair', 'Rating']
In order to satisfy C1-C3, the new SimVerb-3500 evaluation set contains similarity ratings for 3,500 verb pairs, containing 827 verb types in total and 3 member verbs for each top-level VerbNet class. The rating scale goes from 0 (not similar at all) to 10 (synonymous). We employed the SimLex-999 annotation guidelines. In particular, we instructed annotators to give low ratings to antonyms, and to distinguish between similarity and relatedness. Pairs that are related but not similar (e.g., to snore / to snooze, to walk / to crawl) thus have a fairly low rating. Several example pairs are provided in Tab.