\model: Few-shot Task-Oriented Dialog with A Single Pre-trained Auto-regressive Model
Table 11: Ablation study on different negative samples in the contrastive objective on MultiWOZ in the end-to-end evaluation setup; The 2nd and 3rd row indicate removing individual belief only and individual belief & response, respectively.
['Model', 'Inform↑', 'Success↑', 'BLEU↑', 'Combined↑']
To study the effect of different schemes to construct negative samples in the pre-training objectives, we conduct an ablation analysis on MultiWOZ. We see that removing belief state only from negative samples substantially degrades the inform and success score. We further remove the mismatched responses, and only use the mismatched belief state and response pairs to constructive negative samples. This decreases the success rate and BLEU score. The results show the importance of constructing various types of negative samples in designing an effective contrastive objective.