When to Talk: Chatbot Controls the Timing of Talking during Multi-turn Open-domain Dialogue Generation
Table 4: Automatic evaluation results of all the models on Ubuntu and modified dailydialog test dataset (The table above is the results of ubuntu dataset, and the below is dailydialog). Baseline HRED cannot control the timing of talking and the accuracy and macro-F1 of it is empty. All the best results are shown in bold.
['[BOLD] Models', '[BOLD] PPL', '[BOLD] BLEU1', '[BOLD] BLEU2', '[BOLD] BLEU3', '[BOLD] BLEU4', '[BOLD] BERTScore', '[BOLD] Dist-1', '[BOLD] Dist-2', '[BOLD] Acc', '[BOLD] Macro-F1']
For all the models, we adopt the early stopping to avoid the overfitting. Generation Quality: The strong baseline HRED-CF which explicity models the timing of talking is much better than HRED on almost all the metrics such as PPL and BLEU. Especially in terms of BLEU, HRED-CF exceeds HRED by up to 1.5% on Embedding Average points. The experimental results demonstrate that controlling the timing of talking can significantly improve the quality the dialogue generation. The other two modified datasets shows the same conclusion which can be found in appendix. Compared with the simple graph neural networks such as GCN and GAT (third and forth rows), our proposed gated neural network is much better. It indicates that the simply appling the graph neural networks is not enough and the gated mechanism is very important in modeling timing of talking. Compared with the strong baseline HRED-CF, the response quality of our proposed graph neural networks is weak. We attribute this phenomenon to the negative transfer problem Liu et al. We leave this problem to the future research. But the quality of responses is still much better than HRED. It can be shown that although the BLEU scores, PPL and embedding average scores of GNN-based models are lower than HRED-CF, the diversity of the generation is better than strong baseline HRED-CF (distinct-1 and distinc-2). In order to test the effectiveness of the gated mechanism, we also replace the first gated mechanism with the self-attention attention mechanism and propose another strong baseline W2T-GGAT (fifth row). Except for PPL and accuracy, W2T-GGAT ’s performance is far worse than our proposed double-gated graph neural network. Decision Performance: It can be shown that our proposed graph neural network models significantly outperforms the strong baseline HRED-CF by up to 5.77% Accuracy and 2.07% macro-F1. Furthermore, compared with the graph neural networks without gated mechanism, it can be found that out proposed model can achieve better performance on accuracy and macro-F1. It demonstrates that the gated mechanism is very important in modeling timing of talking.