BLEU Neighbors: A Reference-less Approach to Automatic Evaluation
Table 3: Spearman’s ρ between predicted essay quality and the ground truth, where the test essays are from Topic #3 and * denotes p<0.01. When using essays from Topic #8 as the training data, BLEU Neighbors is state-of-the-art, even beating out models with access to a gold-standard reference essay for Topic #3.
['Model', 'Train Topic', '[ITALIC] ρ', 'Coverage']
Unlike the NLG tasks, every test example here is a multi-sentence paragraph, which makes scoring more difficult: ten random sentences may be high-quality on their own while making little sense when put together. The difficulty of this task is compounded by the fact that the ground-truth quality of each essay is based on a gold-standard reference for Topic #3. Despite not being given the gold-standard reference essay, when BLEU Neighbors is trained with sample essays from Topic #8, it achieves a new state-of-the-art: a Spearman’s ρ of 0.500 between its predicted scores and the ground-truth quality judgments. Due to the small amount of test data, however, this improvement over the state-of-the-art is not statistically significant at p<0.01 when using a Williams test. This is unsurprising, given that some topics are more related to #3 than others. Some topics (e.g., #4) are so different from the test topic that its training examples are not useful at all.