A Pre-training Based Personalized Dialogue Generation Model with Persona-sparse Data
Table 3: Automatic evaluation on the biased test set.
['Model', 'Acc.', 'BLEU', 'F1', 'Dist.', 'ppl.']
It can be seen that our model outperforms all the baselines on all the metrics except for the perplexity. This indicates that our proposed model can produce diversified dialogue responses carrying rich persona-related features. We can further observe that: 1) Generating personalized dialogue responses hurts perplexity scores since persona-related words are relative rare and more biased generation of such words will lead to higher perplexity. Though the baseline models fit the test data well (with lower perplexity), they fail to produce more persona-related responses (with lower persona accuracy scores) compared to our model. The weight α in Eq. Higher α values lead to more persona-consistent responses; 4) Larger performance gaps between our model and the baselines are obtained on the biased test set. This shows that our model can generate more persona-consistent responses in biased contexts. Our model outperforms all the baselines in all the measures. Particularly for persona consistency, our full persona model (i.e., α=1) significantly outperforms all the baselines with a large margin. This indicates that our model can generate more persona-consistent responses that are fluent and context-coherent. Further observations also show that: 1) Exhibiting too many persona-related features (i.e., obtaining higher persona consistency) hurts response fluency and context coherency. This is in line with the trade-off between the persona accuracy and perplexity as observed in the automatic evaluation results. Moreover, our dynamic weight predictor provides a better balance between the persona-consistency and context coherency, especially on the biased test set; 2) The persona consistency of our full persona model (i.e., α=1) even surpasses the human-generated response on the random test set. This further proves that our model can incorporate richer persona-related features in the generated responses. 3) Although directly fine-tuning on the noisy persona dense data (i.e., TTransfo+P and LConv+P) helps to produce more persona-consistent responses, our model still surpasses these baselines significantly. This verifies the effects of the proposed dynamic weighting scheme.