Retrieval-Enhanced Adversarial Training for Neural Response Generation
Table 3: Human evaluation results of mean score, proportions of three levels (+2, +1, and 0), and the agreements measured by Fleiss’s Kappa in appropriateness, informativeness, and grammaticality.
['[EMPTY]', 'Appropriateness Mean', 'Appropriateness +2', 'Appropriateness +1', 'Appropriateness 0', 'Appropriateness  [ITALIC] κ', 'Informativeness Mean', 'Informativeness +2', 'Informativeness +1', 'Informativeness 0', 'Informativeness  [ITALIC] κ', 'Grammaticality Mean', 'Grammaticality +2', 'Grammaticality +1', 'Grammaticality 0', 'Grammaticality  [ITALIC] κ']
Our approach has the highest mean score and the largest proportions of +2 and +1 in appropriateness. Meanwhile, it outperforms all generation-based and retrieval-enhanced approaches in informativeness. This suggests that our approach is able to respond more appropriately and incorporate informative content into responses at the same time. Note that Rtr has the highest informativeness mean score due to its diverse human-written content. However, it may also contain some irrelevant information, leading to a bad performance in appropriateness. Besides, most responses in Rtr are annotated as +2 or 0 in informativeness. This is also because Rtr responses are extremely diverse which always include new content, making a response tend to get +2 if the content are relevant, otherwise 0. In terms of grammaticality, the mean score of our approach is higher than that of S2S and MS2S, and is comparable with that of AL, indicating that our approach is competitive in generating fluent responses. Edit has a high mean score mainly due to its relatively simple sentence structure. is that they have some repetitive responses, like “windows, windows, windows”.