Translationese in Machine Translation Evaluation
Table 4: Comparison of BLEU scores of MT systems participating in WMT-15 – WMT-18 for test data created in the same/forward (F) and reverse (R) direction, where R>F (%) = the proportion of systems with a reverse BLEU score greater than its forward score for precisely the same test scenario; R−F μ = mean of the difference in reverse and forward BLEU scores; R−F σ = standard deviation of the difference in reverse and forward BLEU scores; n= number of MT systems.
['[EMPTY]', 'R>F (%)', 'F>R (%)', 'R−F  [ITALIC] μ', 'R−F  [ITALIC] σ', '[ITALIC] n']
Besides human evaluation, the performance of MT systems is often measured using automatic metrics, the most common of which remains to be the BLEU score Papineni et al. Counter expectation there is a clear mix of positive and negative BLEU score differences for several language pairs, as forward BLEU scores are higher than equivalent reverse BLEU scores (cs-en, en-cs, en -de, en-et, fi-en, lv-en, ro-en, ru-en, en-ru, zh-en and en-zh).