Evaluating the Evaluation of Diversity in Natural Language Generation
Table 3: decTest ranking results: Spearman’s (ρ) correlation between temperature differences and each metric score. Accuracy (acc) of classifying which set has the higher temperature. Standard deviation is up to 0.02 for all automatic metrics for both Spearman’s correlation and accuracy.
['[EMPTY]', '[BOLD] storyGen  [ITALIC] ρ', '[BOLD] storyGen acc', '[BOLD] respGen  [ITALIC] ρ', '[BOLD] respGen acc', '[BOLD] promptGen  [ITALIC] ρ', '[BOLD] promptGen acc']
We observe that humans are better at ranking compared to giving absolute scores, and are doing as well as automatic metrics. However, the scores of all automatic metrics also improve, making it difficult to separate between the different metrics.