BLEU might be Guilty but References are not Innocent
Table 2: Human adequacy assessments for different kinds of references, over the full set of 1997 sentences. HQ(P) has been generated by picking sentence-by-sentence the more accurate rated translation from WMT.p and AR.p. HQ(R) and HQ(all 4) have been generated with the same method by either combining WMT and AR or all four reference translations.
['[EMPTY]', 'adequacy rating']
We explore collecting diverse references using paraphrasing to steer away from translationese, with the ultimate goal of generating a natural-to-natural test set, where neither the source sentences nor the reference sentences contain translationese artifacts. In an initial experiment on a sample of 100 sentences, we asked linguists to paraphrase (translated) sentences. The paraphrased references had only minor changes and consequently only minor impact on the automatic metrics. Therefore, we changed the instructions and asked linguists to paraphrase the sentence as much as possible while also suggesting using synonyms and different sentence structures. These instructions satisfy not only our goal to generate an unbiased sentence, but also have the side effect that two paraphrases of the same sentence are quite different. All our paraphrase experiments in this paper are done with these instructions. As a side note, one might be concerned that paraphrasing “as much as possible” might yield excessive reformulation at the expense of adequacy in some cases. It may indeed be true that more investigation into the manner of paraphrasing would yield better instructions. To compensate for this in the present paper, we collect adequacy ratings for all produced paraphrases. For the purpose of MT evaluation, a reference translation is considered good if its use as reference for an automated metric yields scores that can replace human judgments. In other words, measuring overlap between an MT output and a good reference correlates well with human ratings of that MT output. It is often believed that reference translations with high human ratings should also be good for automated evaluation.