BLEU Neighbors: A Reference-less Approach to Automatic Evaluation
Table 1: The mean squared error (MSE) and Spearman’s ρ of language quality predictions ˆq(⋅) with respect to the ground truth q(⋅). The lowest MSE and highest ρ across all models is in bold and * signifies p<0.01. For all tasks, BLEU Neighbors achieves a higher Spearman’s ρ than its ROUGE, METEOR, and BERTScore counterparts. For dialogue and open-ended generation, it even has a lower MSE and higher ρ than human annotators on average!
['[EMPTY]', '[BOLD] Dialogue MSE', '[BOLD] Dialogue  [ITALIC] ρ', '[BOLD] Dialogue Coverage', '[BOLD] Open-ended Generation MSE', '[BOLD] Open-ended Generation  [ITALIC] ρ', '[BOLD] Open-ended Generation Coverage', '[BOLD] Summarization MSE', '[BOLD] Summarization  [ITALIC] ρ', '[BOLD] Summarization Coverage']
Because the ground-truth quality is the mean over 20 annotator judgments, we provide the performance of the best human annotator (across all examples) and the average performance across all individual annotators. Note that not all annotators scored all the examples: We find that there is a significant gap between the best- and average-case, both in terms of MSE and Spearman’s ρ. For example, on the summarization task, the MSE and Spearman’s ρ of the best human annnotator is 4x and 2x better than those of annotators on average. For open-ended generation and dialogue, it even outperforms the average-case human annotator! Only on evaluating summaries does the average-case annotator beat all nearest neighbors models with respect to Spearman’s ρ; this is unsurprising, given that summary quality is strongly conditioned on the source text, which these models ignore. However, even for summarization, the difference between the average-case annotator and BLEU Neighbors is not statistically significant at p<0.01 (per a Williams test), owing to the small dataset size; the same holds for dialogue and open-ended generation. Despite the impressive performance of BLEU Neighbors, it should be noted that it is still well behind the best human annotator for each task.