BLEU might be Guilty but References are not Innocent
Table 3: Spearman’s ρ and Kendall’s τ for the WMT2019 English→German official submissions with human ratings conducted by the WMT organizers.
['Full Set (22)', 'Reference', '[ITALIC] ρ', '[ITALIC] τ']
On the full set of 22 submissions, all 3 new references (AR, WMT.p, AR.p) show higher correlation with human judgment than the original WMT reference, with the paraphrased references WMT.p coming out on top. Furthermore, each paraphrased reference set shows higher correlation when compared to the “standard” reference set that it was paraphrased from. Note that the correlation numbers already appear relatively high for the full set of systems. This is because both Kendall’s τ and Spearman’s ρ rank correlation operate over all possible pairs of systems. Since the submissions to WMT2019 covered a wide range of translation qualities, any metric able to distinguish the highest-scoring and lowest-scoring systems will already have a high correlation.