Multiresolution Recurrent Neural Networks: An Application to Dialogue Response Generation
Table 1: Ubuntu evaluation using precision (P), recall (R), F1 and accuracy metrics w.r.t. activity, entity, tense and command (Cmd) on ground truth utterances, and human fluency and relevancy scores given on a scale 0-4 (∗ indicates scores significantly different from baseline models at 90% confidence)
['[BOLD] Model', '[BOLD] Activity  [BOLD] P', '[BOLD] Activity  [BOLD] R', '[BOLD] Activity  [BOLD] F1', '[BOLD] Entity  [BOLD] P', '[BOLD] Entity  [BOLD] R', '[BOLD] Entity  [BOLD] F1', '[BOLD] Tense  [BOLD] Acc.', '[BOLD] Cmd  [BOLD] Acc.', '[BOLD] Human Eval.  [BOLD] Fluency', '[BOLD] Human Eval.  [BOLD] Relevancy']
The MrRNNs clearly perform substantially better than the baseline models both w.r.t. human evaluation and automatic evaluation metrics. The MrRNN with noun representations achieves 2x−3x higher scores w.r.t. entities compared to other models, and the human evaluators also rate its fluency and relevancy substantially higher than other models. The MrRNN with activity representations achieves 2x−3x higher scores w.r.t. activities compared to other models and nearly 2x higher scores w.r.t. entities compared to all baselines. Human evaluators also rate its fluency substantially higher than the baseline models. However,its relevancy is rated only slightly higher compared to the HRED model, which we believe is caused by human evaluators being more likely to noticing software entities than actions in the dialogue responses (even though actions are critical to solving the actual technical problem). Overall, the results demonstrate that the MrRNNs have learned to model high-level goal-oriented sequential structure on Ubuntu.