Transferable Multi-Domain State Generator for Task-Oriented Dialogue Systems
Table 3: We run domain expansion experiments by excluding one domain and fine-tuning on that domain. The first row is the base model trained on the four domains. The second row is the results on the four domains after fine-tuning on 1% new domain data using three different strategies. One can find out that GEM outperforms Naive and EWC fine-tuning in terms of catastrophic forgetting on the four domains. Then, we evaluate the results on new domain for two cases: training from scratch and fine-tuning from the base model. Results show that fine-tuning from the base model usually achieves better results on the new domain compared to training from scratch.
['[BOLD] Evaluation on 4 Domains', '[BOLD] Evaluation on 4 Domains', '[BOLD] Joint  [ITALIC] Except Hotel', '[BOLD] Slot  [ITALIC] Except Hotel', '[BOLD] Joint  [ITALIC] Except Train', '[BOLD] Slot  [ITALIC] Except Train', '[BOLD] Joint  [ITALIC] Except Attraction', '[BOLD] Slot  [ITALIC] Except Attraction', '[BOLD] Joint  [ITALIC] Except Restaurant', '[BOLD] Slot  [ITALIC] Except Restaurant', '[BOLD] Joint  [ITALIC] Except Taxi', '[BOLD] Slot  [ITALIC] Except Taxi']
Domain Expanding In this setting, the TRADE model is pre-trained on four domains and a held-out domain is reserved for domain expansion to perform fine-tuning. After fine-tuning on the new domain, we evaluate the performance of TRADE on 1) the four pre-trained domains and 2) the new domain. We experiment with different fine-tuning strategies. In general, GEM outperforms naive and EWC fine-tuning in terms of overcoming catastrophic forgetting. We also find that pre-training followed by fine-tuning outperforms training from scratch on the single domain.