Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challenge
Table 8: Word-overlap metrics scores (see Section 7.1) for all primary systems, plus the average of all metrics’ values normalised into the 0-1 range. The list is sorted by the normalised average; any values higher than the corresponding baseline are marked in bold. System architectures are coded with colours and symbols: ♡seq2seq, ♢other data-driven, ♣rule-based, ♠template-based.
['[BOLD] System', '[BOLD] BLEU', '[BOLD] NIST', '[BOLD] METEOR', '[BOLD] ROUGE-L', '[BOLD] CIDEr', '[BOLD] norm.\xa0avg.']
It is apparent that the TGen baseline is very strong in terms of word-overlap metrics: No primary system is able to beat it in terms of all metrics, or in terms of the normalised metrics’ mean – only Slug comes very close. Several other systems manage to beat TGen in one of the metrics but not in others. All five metrics showed a very similar pattern. For comparison, we also measure similarity of system outputs against the reference texts in the test set, as well as a subset of the test set with a single, randomly sampled reference text per MR.