The Eighth Dialog System Technology Challenge
Table 1: Task 1 Evaluation Results
['Team', 'Human Evaluation Succ. %', 'Human Evaluation Under.', 'Human Evaluation Appr.', 'Human Evaluation Turns', 'Simulator-based Evaluation Succ. %', 'Simulator-based Evaluation Reward', 'Simulator-based Evaluation Turns', 'Simulator-based Evaluation Prec.', 'Simulator-based Evaluation Rec.', 'Simulator-based Evaluation F1', 'Simulator-based Evaluation Book %']
Twelve teams participated in this task. A component-wise system with BERT-based NLU model Devlin et al. However, there are discrepancies between human evaluation and simulator-based evaluation. The best system in the human evaluation is based on fine-tuning GPT-2 Radford et al. It predicts dialog states, system actions, and responses in an end-to-end fashion, and achieves a success rate of 68.32%.