Memory-augmented Dialogue Management for Task-oriented Dialogue Systems
Table 7. Fine-grained and overall accuracy on the DM-DSTC dataset. The number in brackets are the accuracy at the session level, and number without brackets are at the turn level.
['Metrics MEM', 'DA type 62.5', 'DA type (9.9)', 'slot-value 14.2', 'slot-value (0.0)', 'mask 71.0', 'mask (0.1)', 'All 0', 'All (0.0)']
To assess the performance of our proposed model on real dialogue data, we conducted another experiment on DM-DSTC. Different from DMBD, there is only one task in the DM-DSTC dataset. We only reported the results of the methods which predict dialogue act as output. It should be pointed out that in this new dataset, many values in dialogue act annotation didn’t appear exactly in user utterances (such as asian oriental), thus for those values we can not provide precise attention supervision, which will affect the performance of slot-level attention. Moreover, the Res_name slot in this dataset degrades the accuracy because its value does not appear in the dialogue context, and is queried from a knowledge base conditioned on previous search constraints, which is not consistent with our model setting. Compared to MEM and RNN, our proposed method obtains higher accuracy on turn-level overall prediction, as well as the dialogue act type and mask prediction. Although MEM’s accuracy on DA type , slot-value and mask prediction is slightly lower than ours, its overall accuracy on turn-level is far less than our proposed model. This can be attributed to the framework of MEM, where its DA type, mask and slot-value prediction is trained separately, while in our model these three tasks are trained. For the variants of MAD, the experiment results are consistent with what we observed in DMBD. MAD-SM obtains lower accuracy on slot-value prediction compared to MAD, while maintains similar accuracy on DA type and mask. For MAD-Attn, the result is similar to MAD-SM when compared to MAD, but its accuracy on slot-value prediction is obviously higher than that of MAD-SM since it maintains the slot-value memory network. MAD-EM, which removes the external memory, obtains significantly lower accuracy on the prediction of dialogue act type and mask, and its accuracy on slot-value prediction is also reduced.