Alternating Roles Dialog Model with Large-Scale Pre-Trained Language Models
Table 1: Results on CamRest676 dataset.
['Model', 'Entity Match rate', 'Ground Truth Belief State BLEU-4', 'Ground Truth Belief State Success. F1', 'Generated Belief State BLEU-4', 'Generated Belief State Success. F1']
We first use ground truth belief state in all methods to evaluate their response generation quality. ARDM achieves the best BLEU and Success F1 score. We observe that after fine-tuning GPT-2 on the CamRest676, it achieves similar results compared to the previous state-of-the-art method, Sequicity with reinforcement fine-tuning. This suggests pre-trained large-scale language model, such as GPT-2, transfers the meaningful representations to help fine-tuning. However, without the alternating mechanism, GPT-2 alone does not perform as well as ARDM in terms of both BLEU-4 and Success F1, especially in BLEU-4 ( improved 19%). Without modeling the speaker role, the model blends two speakers language distribution and ignores the inherent speaker role difference. Moreover, to test if our model preserves its performance with even less training data, we reduce the training data to 50%, and the performance only drops slightly. With half of the training data, our method still performs significantly better than Sequicity. This result suggests ARDM is robust on low-resource settings due to the advantage of the large-scale pre-training language model.