Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Table 9: Human side-by-side evaluation scores of WMT En→Fr models.
['Model', 'BLEU', 'Side-by-side']
Finally, to better understand the quality of our models and the effect of RL refinement, we carried out a four-way side-by-side human evaluation to compare our NMT translations against the reference translations and the best phrase-based statistical machine translations. During the side-by-side comparison, humans are asked to rate four translations given a source sentence. The four translations are: 1) the best phrase-based translations as downloaded from http://matrix.statmt.org/systems/show/2065, 2) an ensemble of 8 ML-trained models, 3) an ensemble of 8 ML-trained and then RL-refined models, and 4) reference human translations as taken directly from newstest2014 Source and target sentences (human translations) are from the public benchmark WMT En→Fr (newstest2014) data set. The right-hand column shows the human ratings on a scale of 0 (complete nonsense) to 6 (perfect translation). We disagree with some of the human ratings, e.g., the translation “Elle a été repéré trois jours plus tard par un promeneur de chien piégé dans la carrière” contains grammatical mistakes and changes semantics, and is still scored 6. We present it to illustrate the potential problems of the scoring process.