Generate, Delete and Rewrite: A Three-Stage Framework for Improving Persona Consistency of Dialogue Generation
Table 3: Results of human evaluations (on the left) and automatic metrics (on the right). The Dist-1.& 2. are scaled by 10−2. Significant tests (t-test) are performed, and our method is significantly better than all methods on most metrics (p-value<0.05), with the exceptions marked by †. We also present two model-based ratios, the Entdiin and the Entbert, as an additional reference for persona consistency assessments. Note that the automatic metrics are calculated on the whole test set. * The sampling process in CVAE leads to very unstable PPL.
['[BOLD] Model', '[BOLD] Const.', '[BOLD] Fluc.', '[BOLD] Relv.', '[BOLD] Info.', '[BOLD] PPL', '[BOLD] Dist-1.', '[BOLD] Dist-2.', 'Entdiin', 'Entbert']
Compared with baseline methods, our GDR model obtains the highest consistency score of 49.2% in human evaluation, which is significantly better than other methods. The target responses in the sampled data are also annotated, and 65.4% of them expressed persona information. Moreover, the two model-based entailment ratios, which are calculated on the whole test set, also prove the effectiveness of our GDR model. Although the two NLI models differ in results, our GDR model ranks first under the evaluation of both DIIN and BERT.