BERTScore: Evaluating Text Generation with BERT
Table 7: Examples sentences where similarity ranks assigned by Human, F\textscBERT, and Bleu differ significantly on WMT16 German-to-English evaluation task. x: gold reference, ^x: candidate outputs of MT systems. Rankings assigned by Human, F\textscBERT, and Bleu are shown in the right three columns. The sentences are ranked by the similarity, i.e. rank 1 is the most similar pair assigned by a score. An ideal metric should rank similar to humans.
['Case', 'No.', 'Reference and Candidate Pairs', 'Human', '[ITALIC] F \\textsc  [ITALIC] BERT', 'Bleu']
We observe that BERTScore is effectively able to capture synonyms and changes in word order. For example, the reference and candidate sentences in pair 3 are almost identical except that the candidate replaces opinion in Hungary with Hungarian view and switches the order of the quotation mark (‘‘) and a. While BERTScore ranks the pair relatively high, SentBleu judges the pair as dissimilar, because it cannot match synonyms and is sensitive to the small word order changes. Pair 5 shows a set of changes that preserve the semantic meaning: replacing to cut with will reduce and swapping the order of 30,000 and jobs. BERTScore ranks the candidate translation similar to the human judgment, whereas SentBleu ranks it much lower. We also see that SentBleu potentially over-rewards n-gram overlap, even when phrases are used very differently. In pair 6, both the candidate and the reference contain the human dignity of the man. Yet the two sentences convey very different meaning. BERTScore agrees with the human judgment and ranks the pair low. In contrast, SentBleu considers the pair as relatively similar because of the significant word overlap. We observe that BERTScore finds it difficult to detect factual errors. For example, BERTScore assigns high similarity to pair 11 when the translation replaces German language with English and pair 12 where the translation incorrectly outputs Tuesday when it is supposed to generate Monday. BERTScore also fails to identify that 5ft 11in is equivalent with 1.80 metres in pair 18. SentBleu also suffers from these limitations.