BLEU might be Guilty but References are not Innocent
Table 5: WMT 2019 English→German: Correlations (Kendall’s τ) of alternative metrics: BLEU, 1.0 - TER, chrF, METEOR, BERTScore, and Yisi-1.
['metric', 'WMT', 'HQ(R)', 'WMT.p', 'HQ(P)', 'HQ(all)']
Any reference-based metric can be used with our new reference translations. In addition to BLEU, we consider TER Snover et al. (winning system of WMT 2019 English→German metric task). Comparing the two paraphrased references, we see that HQ(P) shows higher correlation for chrF and Yisi when compared to WMT.p. In particular Yisi (which is based on word embeddings) seems to benefit from the higher accuracy of the reference translation.