Evaluating the Evaluation of Diversity in Natural Language Generation
Table 5: conTest results: Spearman’s (ρ) correlation between set’s class (1 – high content diversity, 0 – low content diversity) and each metric score. The optimal classifier accuracy (OCA) between the two classes over the metrics’ score.
['[EMPTY]', '[BOLD] storyGen  [ITALIC] ρ', '[BOLD] storyGen OCA', '[BOLD] respGen  [ITALIC] ρ', '[BOLD] respGen OCA', '[BOLD] promptGen  [ITALIC] ρ', '[BOLD] promptGen OCA']
This time, n-gram-based metrics perform poorly, indicating they do not measure well content diversity. Neural models perform better than n-gram-based metrics (especially sent-BERT), but there is still a clear gap between automatic metrics and humans. Clearly, HDS separates high and low content diversity much better than neural metrics. In addition, n-gram-based metrics saturate both classes to near maximal values, similarly to decTest.