Zero-Shot Transfer Learning with Synthesized Data for Multi-Domain Dialogue State Tracking
Table 4: Accuracy on the zero-shot MultiWOZ experiment (test set), with and without data augmentation. TRADE refers to Wu2019May, SUMBT to Lee2019Jul. “Zero-shot” results are trained by withholding in-domain data. “Zero-shot (Wu)” results are obtained with the unmodified TRADE zero-shot methodology, trained on MultiWOZ 2.1. “Zero-shot (DM)” refers to zero-shot learning using our Dialogue-Model based data synthesis. The last line of each model compares DM with full training, by calculating the % of the accuracy of the former to the latter.
['[BOLD] Model', '[BOLD] Training', '[BOLD] Attraction  [BOLD] Joint', '[BOLD] Attraction  [BOLD] Slot', '[BOLD] Hotel  [BOLD] Joint', '[BOLD] Hotel  [BOLD] Slot', '[BOLD] Restaurant  [BOLD] Joint', '[BOLD] Restaurant  [BOLD] Slot', '[BOLD] Taxi  [BOLD] Joint', '[BOLD] Taxi  [BOLD] Slot', '[BOLD] Train  [BOLD] Joint', '[BOLD] Train  [BOLD] Slot']
Before we evaluate zero-shot learning on new domains, we first measure the accuracy obtained for each domain when trained on the full dataset. For each domain, we consider only the subset of dialogues that include that particular domain and only consider the slots for that domain when calculating the accuracy. In other words, suppose we have a dialogue involving an attraction and a restaurant: a prediction that gets the attraction correct but not the restaurant will count as joint-accurate for the attraction domain. This is why the joint accuracy of individual domains is uniformly higher than the joint accuracy of all the domains. The domain accuracy with the SUMBT model is better than that of TRADE by between 1% and 4% for all domains, except for “Taxi” where it drops by about 4.5%. For TRADE, the joint accuracy improves between 6% on “Taxi” and 19% on “Restaurant”, whereas for SUMBT, joint accuracy improves between 3% on “Taxi” and 30% on “Attraction”. With synthesis, SUMBT outperforms TRADE by a large margin. Except for “Taxi” which has uncharacteristically high joint accuracy of 65%, SUMBT outperforms TRADE from 8% to 18%. This suggests SUMBT can make better use of synthesized data. To compare synthesized with real training data, we calculate how close the accuracy obtained with the synthetic data gets to full training.