Communication-based Evaluation for Natural Language Generation
Table 1: Pearson’s ρ, Spearman’s r, and Kendall’s τ correlation values between assigned scores and quality categories. BLEU-1 is reported because it is the best of the BLEU scores. All correlations are significant at p<0.05 and all Pearson’s correlations are different at p<0.05 according to a Williams’ test.
['Metric', '[ITALIC] ρ', '[ITALIC] r', '[ITALIC] τ']
We want to see the extent to which these scores correlate with the quality categories of the given utterances. In this situation, we have large numbers of points around certain scores (e.g. 1 for the Literal Listener), and these scores have meaning themselves, so we report Pearson’s ρ. We are also interested in the overall monotonicity of the metric scores across categories—we want to avoid good candidates receiving bad scores and vice versa. As such, we report the Spearman’s r and Kendall ’s τ as well.