Machine Translation Evaluation with Neural Networks
Table 9: Comparing to the state of the art at the segment level on the WMT12 dataset, translating into English. Values marked as na were not reported by the authors.
['[BOLD] System', '[BOLD] Details', '[BOLD] Kendall’s  [ITALIC] τ (WMT12-style)  [BOLD] cz ', '[BOLD] Kendall’s  [ITALIC] τ (WMT12-style)  [BOLD] de', '[BOLD] Kendall’s  [ITALIC] τ (WMT12-style)  [BOLD] es', '[BOLD] Kendall’s  [ITALIC] τ (WMT12-style)  [BOLD] fr', '[BOLD] Kendall’s  [ITALIC] τ (WMT12-style)  [BOLD] AVG']
We refer to our metric as NNRK (for Neural Network ReranKing), using subindices to describe the metric variant. The comparison is on the WMT12 dataset, at the segment level. We can see that using absolute scores instead of pairwise comparisons yields better results: by 0.9-1.2 Kendall ’s τ points absolute. We believe that this is because by using an absolute score rather than a pairwise decision, we remove some possible circularities, e.g., in the pairwise framework, we could predict that translation x is better than y, and y is better than z, but z is better than x. This is not possible when working with absolute scores.