BLEU might be Guilty but References are not Innocent
Table 4: BLEU scores of the best submissions of WMT2019 English→German.
['[EMPTY]', 'WMT', 'HQ(R)', 'WMT.p', 'HQ(P)', 'human']
Nevertheless, the system outputs are better ranked and show the highest correlation of any references explored in this paper. Nevertheless, the paraphrased references show higher correlation with human judgment, which motivates us to investigate which n-grams of the MT output are actually matching the paraphrased references during BLEU calculation.