Schema-Guided Dialogue State Tracking Task at DSTC8
Table 1: Comparison of our SGD dataset to existing related datasets for task-oriented dialogue. Note that the numbers reported are for the training portions for all datasets except FRAMES, where the numbers for the complete dataset are reported.
['[BOLD] Metric ↓ Dataset →', '[BOLD] DSTC2', '[BOLD] WOZ2.0', '[BOLD] FRAMES', '[BOLD] M2M', '[BOLD] MultiWOZ', '[BOLD] SGD']
The especially larger number of domains, slots, and slot values, and the presence of multiple services per domain, are representative of these scale-related challenges. Furthermore, our evaluation sets contain many services, and consequently slots, which are not present in the training set, to help evaluate model performance on unseen services. The entire dataset consists of over 16K dialogues spanning multiple domains. Overall statistics of the dataset and comparison with other datasets can be seen in Table The single-domain dialogues in our dataset contain an average of 15.3 turns, whereas the multi-domain ones contain 23 turns on average. The dataset also contains a significant number of unseen domains/APIs in the dev and test sets. 77% of the dialogue turns in the test set and 45% of the turns in dev set contain at least one service not present in the training set. This facilitates the development of models which can generalize to new domains with very few labelled examples.