On Incorporating Structural Information to improve Dialogue Response Generation
Table 1: Results of automatic evaluation. Our proposed architecture SSS(BERT) outperforms the baseline methods.
['Model', 'BLEU', 'ROUGE 1', 'ROUGE 2', 'ROUGE L']
SSS(BERT) is our proposed architecture in terms of the SSS framework. We report best results within SSS chosen across 108 configurations comprising of four different graph combinations, three different contextual and structural infusion methods, three M-GCN layers, and, three embeddings. The best model was chosen based on performance of the validation set. We presented 100 randomly sampled outputs to three different annotators. The annotators were asked to pick from four options: A, B, both, and none. The annotators were told these were conversations between friends. Tallying the majority vote, we obtain win/loss/both/none for SSS(BERT) as 29/25/29/17, SSS(GloVe) as 24/17/47/12 and SSS(ELMo) as 22/23/41/14. This suggests qualitative improvement using SSS framework. We also provide some generated examples in the Appendix B1. We found that the SSS framework had less confusion in generating the opening responses than the GTTP baseline. These “conversation starters” have a unique template for every opening scenario and thus have different syntactic structures respectively. We hypothesize that the presence of dependency graphs over these respective sentences helps to alleviate the confusion as seen in Example 1. The second example illustrates why incorporating structural information is important for this task. We also observed that SSS encoder framework does not improve on the aspects of human creativity such as diversity, initiating a context-switch, and common sense reasoning as seen in Example 3.