Towards Scalable Multi-Domain Conversational Agents: The Schema-Guided Dialogue Dataset
Table 1: Comparison of our SGD dataset to existing related datasets for task-oriented dialogue. Note that the numbers reported are for the training portions for all datasets except FRAMES, where the numbers for the complete dataset are reported.
['[BOLD] Metric ↓ Dataset →', '[BOLD] DSTC2', '[BOLD] WOZ2.0', '[BOLD] FRAMES', '[BOLD] M2M', '[BOLD] MultiWOZ', '[BOLD] SGD']
To highlight these challenges, we introduce the Schema-Guided Dialogue (SGD) Further, to adequately test the models’ ability to generalize in zero-shot settings, the evaluation sets contain unseen services and domains. The dataset is designed to serve as an effective testbed for intent prediction, slot filling, state tracking and language generation, among other tasks in large-scale virtual assistants. An important goal of this work is to create a benchmark dataset highlighting the challenges associated with building large-scale virtual assistants. Our Schema-Guided Dialogue (SGD) dataset exceeds other datasets in most of the metrics at scale. The especially larger number of domains, slots, and slot values, and the presence of multiple services per domain, are representative of these scale-related challenges. Furthermore, our evaluation sets contain many services, and consequently slots, which are not present in the training set, to help evaluate model performance on unseen services.