The BLEU Screen of Death: Why We Need New Metrics for NLG
Table 5: Example pairs of MRs and system outputs from our data, contrasting the average of word-overlap metrics (normalised in the 1-6 range) and semantic similarity (sim) with human ratings (median of all measures).
['[BOLD] No.', '[BOLD] system', '[BOLD] MR', '[BOLD] system output', '[BOLD] human reference', '[BOLD] WOMs', 'sim', '[BOLD] humans']
As shown in previous sections, word-based metrics moderately agree with humans on bad quality output, but cannot distinguish output of good or medium quality. Again, we observe different behaviour between WOMs and sim scores. In Example 1, lols generates a grammatically correct English sentence, which represents the meaning of the MR well, and, as a result, this utterance received high human ratings (median = 6) for informativeness, naturalness and quality. However, WOMs rate this utterance low, i.e. scores of bleu1-4, nist, lepor, cider, rouge and meteor normalised into the 1-6 range all stay below 1.5. This is because the system-generated utterance has low overlap with the human/corpus references. Note that the sim score is high (5), as it ignores human references and computes distributional semantic similarity between the MR and the system output. Examples 2 and 3 show outputs which receive low scores from both automatic metrics and humans. WOMs score these system outputs low due to little or no overlap with human references, whereas humans are sensitive to ungrammatical output and missing information (the former is partially captured by GBMs). Examples 2 and 3 also illustrate inconsistencies in human ratings since system output 2 is clearly worse than output 3 and both are rated by human with a median score of 1. Example 4 shows an output of the rnnlg system which is semantically very similar to the reference (sim=4) and rated high by humans, but WOMs fail to capture this similarity. GBMs show more accurate results for this utterance, with mean of readability scores 4 and parsing score 3.5. Please note that WBMs tend to match against the reference that is closest to the generated output.