Translationese in Machine Translation Evaluation
Table 3: Comparison of human evaluation scores of MT systems participating in WMT-17 and WMT-18 for test data created in the same/forward direction (F) and reverse (R) direction, where R>F (%) = the proportion of systems with a reverse DA score greater than its forward score for precisely the same test scenario; R−F μ = mean of the difference in reverse and forward DA scores; R−F σ = standard deviation of the difference in reverse and forward DA scores; n= number of MT systems.
['[EMPTY]', 'R>F (%)', 'F>R (%)', 'R−F  [ITALIC] μ', 'R−F  [ITALIC] σ', '[ITALIC] n']
and WMT-18 news translation shared task broken down by language pair, where positive differences for systems indicate a higher human evaluation score when systems are tested in the reverse direction relative to the corresponding forward direction DA score. This confirms the suspicion that absolute human evaluation results are in general higher when test data is created in the reverse direction to testing, ranging from the least average difference of 1.65, for Russian to English translation, up to the largest and substantial average difference of 18.04 for English-Latvian.