BERTScore: Evaluating Text Generation with BERT
Table 5: Pearson correlation on the 2015 COCO Captioning Challenge. The M1 and M2 measures are described in Section 4. Leic uses images as additional inputs. Numbers with ∗ are cited from Cui et al. (2018). We bold the highest correlations of task-specific and task-agnostic metrics.
['Metric', 'M1', 'M2']
BERTScore outperforms all task-agnostic baselines by large margins. Image captioning presents a challenging evaluation scenario, and metrics based on strict n-gram matching, including Bleu and Rouge, show weak correlations with human judgments. idf importance weighting shows significant benefit for this task, suggesting people attribute higher importance to content words. We observe that supervised classifiers trained on QQP perform worse than random guess on PAWSQQP, which shows these models predict the adversarial examples are more likely to be paraphrases. Most metrics have decent performance on QQP, but show a significant performance drop on PAWSQQP, almost down to chance performance. This suggests these metrics fail to to distinguish the harder adversarial examples. In contrast, the performance of BERTScore drops only slightly, showing more robustness than the other metrics.