\model: Few-shot Task-Oriented Dialog with A Single Pre-trained Auto-regressive Model
Table 10: Human evaluation results. The remaining results except are quoted from (Li et al., 2020b). Under. is short for language understanding score and Appr. stands for appropriateness score.
['Model', 'Success↑', 'Under.↑', 'Appr.↑', 'Turns↓']
\model achieves substantially better performance than the DSTC8 Track 1 Winner over all the metrics. These results are consistent with the automatic evaluation. However, it is interesting to see that the gap of success rate is more than 20 percent on human evaluation, much larger than the 10 percent gap on automatic evaluation. We attribute this to the fact that the language usage from different Turks varies considerably and is noisy, containing redundant or deficient words and typos etc. By virtue of large-scale pre-training on homogeneous dialog corpora, \model is able to handle language variations and tolerate noisy inputs to accomplish a task. It is worth mentioning that in quite a few cases Turks comment that they feel like talking to a real person.