An Effectiveness Metric for Ordinal Classification: Formal Properties and Experimental Results
Figure 3: Example computation of CEMORD values for two hypothetical systems A and B with respect to the same dataset. The first two tables represent the confusion matrices for both systems. The third table shows prox(ci,cj) for the ground truth, according to the distribution of items in the negative, positive and neutral classes (10, 60 and 30, respectively). The rest of the equations illustrate how proximity values between classes are computed, and the resulting CEMORD values for both systems.
['[BOLD] class proximity', '[BOLD] class proximity  [BOLD] neg', '[BOLD] class proximity  [BOLD] neu', '[BOLD] class proximity  [BOLD] pos']
The ground truth distribution is 10, 60 and 30 items, respectively, which is all the information needed to compute proximity between classes. Note that proximity of one class with respect to other is −log of the amount of items that lie between them (including all items in the ground truth class and half of the items in the system-predicted class) divided by the total number of items. The lowest score corresponds to the proximity between the two extreme cases (in the example, the negative and positive classes), because all items except half of the items in the system-predicted class lie between them, and therefore the −log value is minimal.