Hierarchical Recurrent Attention Network for Response Generation
Table 2: Human annotation results (in %)
['Models', 'Win', 'Loss', 'Tie', 'Kappa']
The ratios were calculated by combining the annotations from the three judges together. We can see that HRAN outperforms all baseline models and all comparisons have relatively high kappa scores which indicates that the annotators reached relatively high agreements in judgment. Compared with S2SA, HRED, and VHRED, HRAN achieves preference gains (win-loss) 6.7%, 6%, 4.8% respectively. Sign test results show that the improvement is statistically significant (p-value <0.01 for HRAN v.s. S2SA and HRAN v.s. HRED, and p-value <0.05 for HRAN v.s. VHRED). Among the three baseline models, S2SA is the worst one, because it loses relationships among utterances in the context. We checked the cases on which VHRED loses to HRAN and found that on 56% cases, VHRED generated irrelevant responses while responses given by HRAN are relevant, logically consistent, and fluent.