Machine Translation Evaluation with Neural Networks
Table 12: Comparing to the state of the art at the system level on the WMT14 dataset, translating into English.
['[BOLD] System', '[BOLD] Details', '[BOLD] Pearson correlation  [ITALIC] r  [BOLD] fr ', '[BOLD] Pearson correlation  [ITALIC] r  [BOLD] de', '[BOLD] Pearson correlation  [ITALIC] r  [BOLD] hi', '[BOLD] Pearson correlation  [ITALIC] r  [BOLD] cs', '[BOLD] Pearson correlation  [ITALIC] r  [BOLD] ru', '[BOLD] Pearson correlation  [ITALIC] r  [BOLD] AVG']
Our NNRK metric is the third in the ranking, after DiscoTK and ReVal. While DiscoTK is best overall, it is not the strongest on any of the five languages in the table. In contrast, NNRK has the highest scores on French-English and German-English, while ReVal is strongest on Czech-English and Russian-English, and LAYERED is best on Hindi-English. On this dataset, NNRK is slightly below ReVal; this can be expected, as our metric specializes on segment-level pairwise judgments and it was trained with only the smaller dataset from WMT11. Note that again our overall correlation is strongly penalized by a very low score on Russian-English. The reason for this phenomenon has to be further investigated.