\model: Few-shot Task-Oriented Dialog with A Single Pre-trained Auto-regressive Model
Table 9: Machine teaching results.
['Model', 'Inform↑', 'Success↑', 'BLEU↑', 'Combined↑']
We firstly use 20 dialog sessions from restaurant domain to perform the few-shot fine-tuning step of \model. We then deploy it to interact with users. The row of \model + Teach shows the result of machine teaching where a human teacher is involved to manually correct 5 failed dialog sessions, which are then utilized to continually fine-tune the deployed system. We observe that \model + Teach improves the combined score by 10% compared to that without human teaching. The results demonstrate the effectiveness of our two-step fine-tuning scheme to deploy \model for a new task. \model + Extra is used as an ablation baseline, where 5 randomly selected dialog sessions are added as extra examples to fine-tune the model. It shows lower performance than machine teaching. Assume that one slot-value pair of belief state correction counts one edit and a response correction counts ten edits. The total numbers of edits for \model + Teach and \model + Extra are 61 and 396, respectively, suggesting that machine teaching reduces the labeling cost by 6x.