Learning a Simple and Effective Model for Multi-turn Response Generation with Auxiliary Tasks
Table 2: Evaluation results on automatic metrics. Numbers in bold indicate the best performing model on the corresponding metrics.
['Dataset', 'Model', 'PPL', 'BLEU', 'Distinct-1', 'Distinct-2', 'Average', 'Greedy', 'Extrema', 'Parameter size', 'Decoding speed']
Our model outperforms all baseline methods on most of the metrics on all the three datasets. The last two columns of the tables compare different models in terms of parameter size and decoding speed. Note that in training, the auxiliary tasks contain parameters outside the generation model. Therefore, in the column of parameter size, we report two numbers for our model with the one before “/” parameter size in training and the one after “/” parameter size of the generation model. It is remarkable that the parameter size of our model, even in training, is smaller than HRED. In spite of this, the model still outperforms ReCoSa with only 19.5%/18.2%/14.0% parameters on the DailyDialog/PERSONA-CHAT/Ubuntu data. This is because (1) the auxiliary tasks can effectively aid the learning of the generation model in our method; and (2) ReCoSa, although in a deep structure, is still inadequate in terms of context modeling due to the RNN-based encoder and the only utterance-level attention. Besides the superior performance on response quality, our model also enjoys a fast decoding process, thanks to the small model size. In terms of decoding speed, our model is comparable with HRED, and 2x faster than ReCoSa. The generation model of SNN is just a simple RNN sequence-to-sequence with one layer encoder and one layer decoder. Therefore, our model is comparable with SSN in terms of complexity and speed. However, SSN is worse than our model on response quality due to (1) the RNN-based seq2seq model in SSN is worse than a transformer-based structure on the benchmarks used in this work, which has been indicated by Sankar et al. ; (2) SSN only considers utterance order, while we also leverage word order, word content, and utterance content in learning. In fact, we find that the proposed auxiliary tasks can improve a 2-layer (one for encoder and one for decoder) RNN-based seq2seq model as well, as reported in Supplementary Material. On most metrics, RNN with full auxiliary tasks is better than SSN but worse than the proposed model.