On conducting better validation studies of automatic metrics in natural language generation evaluation
Table 2: Sample Pearson’s r and Kendall’s τ correlations between BLEU and DA at the system level.
['[EMPTY]', '[ITALIC] n', '| [ITALIC] r|', '| [ITALIC] τ|']
Intuitively, xi and yi represent the BLEU and DA scores. Kendall’s τ then tells us the percentage difference that BLEU would agree over disagree with DA pairwise judgment on system quality. For instance in zh-en, BLEU agrees more than disagrees 86 percentage points with DA for pairwise judgments. We leave the interpretation of correlation metrics more relevant in practice as future work.