BERTScore: Evaluating Text Generation with BERT
Table 9: Ablation Study of MoverScore and BERTScore using Pearson correlations on the WMT17 to-English segment-level data. Correlations that are not outperformed by others for that language pair under Williams Test are bolded. We observe that using WMD does not consistently improve BERTScore.
['Ablation', 'Metric', 'cs-en', 'de-en', 'fi-en', 'lv-en', 'ru-en', 'tr-en', 'zh-en']
We follow the setup of Zhao et al. We report correlations for the two variants of WMD Zhao et al. Our F\textscBERT corresponds to the vanilla setting and the importance weighted variant corresponds to the [IDF-S] setting. The complete MoverScore metric corresponds to [IDF-S]+[SEP]+[PMEANS]+[MNLI]+[RM]. We make several observations. First, for all language pairs except fi-en and lv-en, we can replicate the reported performance. For these two language pairs, Zhao et al. Second, we confirm the effectiveness of [PMEANS] and [MNLI]. However, the contribution of other techniques, including [RM] and [SEP], seems less stable. Third, replacing greedy matching with WMD does not lead to consistent improvement. In fact, oftentimes BERTScore is the better metric when given the same setup. In general, for any given language pair, BERTScore is always among the best performing ones. Given the current results, it is not clear tht WMD is better than greedy matching for text generation evaluation.