Posterior-GAN: Towards Informative and Coherent Response Generation with Posterior Generative Adversarial Network
Table 2: The human evaluation results. We calculate each score by averaging the rank of each model in corresponding metrics. Lower is better.
['DailyDialog Models', 'DailyDialog Coherence', 'DailyDialog Informativeness']
We randomly select 200 samples from the test sets in the two dialogue datasets separately. Each sample consists of query, future turn and responses generated by different models. We invite three annotators to rank the generated responses with respect to two aspects: coherence and informativeness. Ties are allowed. Coherence indicates how likely the generated response is relevant to both query and future turn. Informativeness specifies how much the information related to the context is contained in the generated response. The annotators are all well educated and are ignorant of the models by which the responses are generated. To ensure a stable comparison, the evaluated models consist of 2 Seq2Sea-att models with different initialization, Adver-REGS, DP-GAN, and 3 Posterior-GAN outputs. We also report the inter-annotator agreement to demonstrate the consistency of three annotators. The spearman’s rank correlation coefficient for coherence and informativeness is 0.3948 and 0.3406, with p<0.0001. Whereas, our model obtains substantial and consistent improvements in terms of coherence and informativeness on two public datasets, DailyDialog and Opensubtitles (OSDb).