Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challenge
Figure 5: Similarity of the systems’ outputs as measured by automatic metrics (mean of normalised BLEU, NIST, METEOR, ROUGE-L and CIDEr where one system output is used as reference). Systems are sorted by their architecture. For comparison, we also include metrics values against the full test set with multiple references (test set all) and against a single-reference randomly sampled subset of the test set (test set rand). The table on the right shows mean values of similarity of each system against all other systems (average over columns on the left, excluding the 1st line). System architectures are coded with colours and symbols: ♡seq2seq, ♢other data-driven, ♣rule-based, ♠template-based.
['[BOLD] System', '[BOLD] Mean']
All five metrics showed a very similar pattern. For comparison, we also measure similarity of system outputs against the reference texts in the test set, as well as a subset of the test set with a single, randomly sampled reference text per MR. The exception to this rule in case of the Chen and Sheff2 systems can be explained by the brevity of their outputs (cf. Systems that aim at output diversity (ZHAW1, ZHAW2, Slug-alt and mainly Adapt) also exhibit lowered similarity of their outputs to those of other systems, which might indicate that their outputs are indeed more original. The outputs of rule-based and template-based systems are markedly less similar to other outputs than that of the data-driven systems.