An Effectiveness Metric for Ordinal Classification: Formal Properties and Experimental Results
Table 3: Metric Coverage: Spearman Correlation between single metrics and the UIR combination of Mutual Information, Accuracy, and Kendall across system pairs in both the synthetic and real data sets.
['[EMPTY]', '[EMPTY]', 'Synthetic data all', 'Synthetic data minus', 'Synthetic data minus', 'Synthetic data minus', 'Synthetic data minus', 'Synthetic data minus', 'Real data', 'Real data Replab', 'Real data SEM-2014', 'Real data SEM-2014', 'Real data SEM-2015', 'Real data SEM-2015', 'Real data SEM-2015']
The metric coverage can vary substantially when changing the distribution of systems. For this reason, we first consider every synthetic output and then we repeat the experiment removing each of the system types. As the table shows, CEMORD improves all other metrics, including the individual metrics used as a reference via UIR (MI, Kendall, and Accuracy). Note that the flat (not logarithmic) version CEMORDflat performs systematically worse than the original metric, which supports the use of the logarithmic, information-theoretic formula to compute similarity. CEMORD is the top performer in four datasets, and the second best (with a minimal difference of 0.01 with respect to the best metric) in the other two. The non-logarithmic version of CEMORD is, again, worse than the logarithmic version in all cases (except one, SemEval 2014 task 9A, where they both give the same result).