From Machine Reading Comprehension to Dialogue State Tracking: Bridging the Gap
Table 8: Joint goal accuracy for few-shot experiments. Best numbers reported by TRADE and DSTQA are also shown.
['[EMPTY]', '[BOLD] Hotel  [ITALIC] 1%', '[BOLD] Hotel  [ITALIC] 5%', '[BOLD] Hotel  [ITALIC] 10%', '[BOLD] Restaurant  [ITALIC] 1%', '[BOLD] Restaurant  [ITALIC] 5%', '[BOLD] Restaurant  [ITALIC] 10%', '[BOLD] Attraction  [ITALIC] 1%', '[BOLD] Attraction  [ITALIC] 5%', '[BOLD] Attraction  [ITALIC] 10%', '[BOLD] Train  [ITALIC] 1%', '[BOLD] Train  [ITALIC] 5%', '[BOLD] Train  [ITALIC] 10%', '[BOLD] Taxi  [ITALIC] 1%', '[BOLD] Taxi  [ITALIC] 5%', '[BOLD] Taxi  [ITALIC] 10%']
In few-shot setting, our model (both extractive and categorical) is pre-trained on reading comprehension datasets and we randomly select limited amount of target domain data for fine-tuning. We do not use out-of-domain MultiWOZ data for training for few-shot experiments unlike previous works. We evaluate our model with 1%, 5% and 10% of training data in the target domain. We also report the few-shot results for other two models: TRADE We can see that under all three different data settings, our model outperforms the TRADE and DSTQA models (expect the attraction domain for DSTQA) by a large margin. Especially in 1% data setting for hotel domain, which contains the most number of slots (10) among all the five domains, the joint goal accuracy dropped to 19.73% for TRADE while our model can still achieve relatively high joint goal accuracy of 45.91%. This significant performance difference can be attributed to pre-training our models on reading comprehension datasets, which gives our model ability to comprehend passages or dialogues (which we have empirically verified in next section). The formulation of dialogue state tracking as a reading comprehension task helps the model to transfer comprehension capability.