BLEU might be Guilty but References are not Innocent
Table 6: BLEU scores for WMT newstest 2019 English→German for MT systems trained on bitext, augmented with BT or using APE as text naturalizer. The correct column indicates if the model ranking agrees with human judgments.
['Reference', 'bitext', 'APE', 'BT', 'correct?']
Both the APE and the BT models, which use additional target-side monolingual data, are rated higher by humans than the system relying only on bitext. All references generated with human translations (WMT, HQ(R) and HQ(all 4)) show negative correlation with human ratings for these extreme cases and produce the wrong order. On the other hand, all references that rely purely on paraphrased references do produce the correct ranking of these three systems. This further suggests that reference translations based on human translations bias the metrics to generate higher scores for translationese outputs. By paraphrasing the reference translations, we undo this bias, and the metric can measure the true quality of the underlying systems with greater accuracy.