SimVerb-3500: A Large-Scale Evaluation Set of Verb Similarity
Table 3: Evaluation of state-of-the representation learning models on the full SimVerb-3500 set (SV-3500), the Simlex-999 verb subset containing 222 pairs (SL-222), cross-validated subsets of 222 pairs from SV-3500 (CV-222), and the SimVerb-3500 development (DEV-500) and test set (TEST-3000).
['[BOLD] Model', 'SV-3500', 'CV-222', 'SL-222', 'DEV-500', 'TEST-3000']
Tab. Since the number of evaluation pairs may influence the results, we ideally want to compare sets of equal size for a fair comparison. Picking one random subset of 222 pairs would bias the results towards the selected pairs, and even using 10-fold cross-validation we found variations up to 0.05 depending on which subsets were used. Therefore, we employ a 2-level 10-fold cross-validation where new random subsets are picked in each iteration of each model. The numbers reported as CV-222 are averages of these ten 10-fold cross-validation runs. The reported results come very close to the correlation on the full data set for all models. Our development set contains 500 pairs, selected to ensure a broad coverage in terms of similarity ranges (i.e., non-similar and highly similar pairs, as well as pairs of medium similarity are represented) and top-level VN classes (each class is represented by at least 1 member verb). The test set includes the remaining 3,000 verb pairs. The performances of representation learning architectures on the dev and test sets are reported in Tab. The ranking of models is identical on the test and the full SV-3500 set, with slight differences in ranking on the development set.