Few-Shot Natural Language Generation by Rewriting Templates
Table 1: Comparison between MultiWOZ and SGD-NLG datasets.
['[BOLD] Statistics', '[BOLD] MultiWOZ', '[BOLD] SGD-NLG']
We conduct experiments on 2 datasets - SGD-NLG Rastogi et al. Budzianowski et al. SGD-NLG features a larger number of domains and slots as compared to MultiWOZ and the presence of multiple services per domain makes it representative of practical scale-related challenges faced by today’s virtual assistants. Furthermore, the evaluation sets contain many domains, and consequently slots, which are not present in the training set, to help evaluate model performance on unseen domains. Prior work Mi et al. in a simulated setting mainly by holding out domains for adaptation one at a time and creating small subsets. Datasets so far are also very limited in the number of domains. The largest dataset so far, MultiWOZ, has just 5 domains in the test set. Moreover, lack of knowledge of the exact data splits makes it difficult to make comparisons to other methods. On the other hand, the large size and variety of the SGD dataset makes it a great testbed to study zero-shot learning, few-shot adaptation etc. Having a canonical split will make it easier for future work to compare results across methods. To encourage reproducible research and a single benchmark that can support different paradigms like joint modeling, domain adaptation etc, we make a new version of the SGD dataset as follows: To study few-shot learning from scratch, we make k-shot subsets for varying values of k [5,10,20,40,80]. In this setting each domain has k dialogues. For all the few shot splits we make sure that they contain examples for every dialogue act and slot type. Multi-domain dialogues are removed from the training data. Though many dialogues are discarded, we found that this led to minimal loss in quality. The dev and test sets are left untouched. We call this dataset SGD-NLG. The dataset and code will be made publicly available in the future.