ARAML: A Stable Adversarial Training Framework for Text Generation
Table 5: Human evaluation on WeiboDial. The scores represent the percentages of Win, Lose or Tie when our model is compared with a baseline. κ denotes Fleiss’ kappa (all are moderate agreement). The scores marked with * mean p-value<0.05 and ** indicates p-value<0.01 in sign test.
['Model', 'Grammaticality Win (%)', 'Grammaticality Lose (%)', 'Grammaticality Tie (%)', '[ITALIC] κ', 'Relevance Win (%)', 'Relevance Lose (%)', 'Relevance Tie (%)', '[ITALIC] κ']
We also conducted sign test to check the significance of the differences. This result indicates that the samples surrounding true responses provide stable rewards for the generator, and stable RAML training paradigm significantly enhances the performance in both metrics.