Multi-domain Dialogue State Tracking as Dynamic Knowledge Graph Enhanced Question Answering
Table 3: Joint accuracy on domain expansion experiments. Models are either trained from scratch on the target domain, or trained from the 4 source domains and then fine-tuned on the target domain.
['Domain Expansion', 'Train on 5% Data from Scratch TRADE', 'Train on 5% Data from Scratch DSTQA w/o graph', 'Train on 5% Data by Fine Tuning TRADE', 'Train on 5% Data by Fine Tuning DSTQA w/o graph', 'Train on 5% Data by Fine Tuning DSTQA w/ graph', 'Train on 10% Data From Scratch TRADE', 'Train on 10% Data From Scratch DSTQA w/o graph', 'Train on 10% Data by Fine Tuning TRADE', 'Train on 10% Data by Fine Tuning DSTQA w/o graph', 'Train on 10% Data by Fine Tuning DSTQA w/ graph']
We take one domain in MultiWOZ 2.0 as the target domain, and the remaining 4 domains as source domains. Models are trained either from scratch using only 5% or 10% sampled data from the target domain, or first trained on the 4 source domains and then fine-tuned on the target domain with sampled data. In general, a model that achieves higher accuracy by fine-tuning is more desirable, as it indicates that the model can quickly adapt to new domains given limited data from the new domain. In this experiment, we compare DSTQA w/span with TRADE. With 5% new domain data, DSTQA fine-tuning has an average of 43.32% relative improvement over DSTQA training from scratch, while TRADE fine-tuning only has an average of 19.99% relative improvement over TRADE training from scratch. DSTQA w/ graph also demonstrates its benefit over DSTQA w/o graph, especially on the taxi domain. This is because the ‘taxi’ domain is usually mentioned at the latter part of the dialogue, and the destination and departure of the taxi are usually the restaurant, hotel, or attraction mentioned in the previous turns and are embedded in the graph.