BLEU Neighbors: A Reference-less Approach to Automatic Evaluation
Table 2: BLEU Neighbors performance when the training and test examples are sourced from different tasks. For example, the intersection of G → and → D means that training examples from open-ended generation are used to score dialogue data. In this setup, a moderate Spearman’s ρ can still be achieved on the dialogue data; for summarization, the correlation drops to near zero.
['Source Task', 'Target Task → D', 'Target Task → G', 'Target Task → S']
Of all the tasks, performance on dialogue is the most robust: regardless of which task is used to source the training data, it is possible to achieve a moderately strong correlation (ρ>0.27), albeit with lower coverage. Performance on summarization drops to near zero in this setup – this is unsurprising, given that summary quality is strongly conditioned on the source text, which is ignored here. For open-ended generation, it is still possible to achieve a weak correlation (ρ>0.09) with this setup. Curiously, we find that the coverage for open-ended generation actually improves when the training data is sourced from a different task, suggesting that it may be possible to adjust parameters a,b to trade off some coverage for a higher correlation.