Stochastic Language Generation in Dialogue using Recurrent Neural Networks with Convolutional Sentence Reranking
Table 3: Pairwise comparison between four systems. Two quality evaluations (rating out of 5) and one preference test were performed in each case. Statistical significance was computed using a two-tailed Wilcoxon rank sum test and a two-tailed binomial test (*=p
['Metrics', 'handcrafted 148 dialogs, 829 utt.', 'RNN1 148 dialogs, 829 utt.', 'handcrafted 148 dialogs, 814 utt.', 'RNN5 148 dialogs, 814 utt.', 'RNN1 144 dialogs, 799 utt.', 'RNN5 144 dialogs, 799 utt.', 'O&R5 145 dialogs, 841 utt.', 'RNN5 145 dialogs, 841 utt.']
As can be seen, the human judges preferred both RNN1 and RNN5 compared to the rule-based generator and the preference is statistically significant. Furthermore, the RNN systems scored higher in both informativeness and naturalness metrics, though the difference for informativeness is not statistically significant. When comparing RNN1 with RNN5, RNN1 was judged to produce higher quality utterances but overall the diversity of output offered by RNN5 made it the preferred system. that showed that language variability by paraphrasing in dialogue systems is generally beneficial. Lastly, RNN5 was thought to be significantly better than O&R in terms of informativeness.