Memory-augmented Dialogue Management for Task-oriented Dialogue Systems
Table 5. The accuracy across all tasks and methods. The numbers in brackets are the accuracy at the session level, and numbers without brackets are at the turn level. A session is correct only if all the sentences in the session are predicted correctly.
['Metrics', '1 Issuing API calls', '1 Issuing API calls', '2 Updating API calls', '2 Updating API calls', '3 Displaying options', '3 Displaying options', '4 Providing options', '4 Providing options', '5 Full dialogs', '5 Full dialogs']
Overall Performance Analysis Both turn-level and session-level results on all the five tasks are reported. We have the following observations: MAD obtains the best performance on most of the tasks. The model obtains an accuracy of about 100% at both turn and session-level evaluation, which shows the effectiveness of our proposed model. While in Task 1, MAD is at the second place, where Supervised Ebd and MEMN2N (+match) methods obtains 100% accuracy at both turn and session-level evaluation, which is 1% higher than ours. MAD’s defect on Task 1 can be attributed to a potential rule in Task 1, where if the user doesn’t provide enough values to form a query, the agent will request for the value of slots in a fixed order. For example in task 1, the agent requests for slots in an order of (Cuisine → Location → Size → Price). However, this order rule is not essential for a practica application, where the agent can request for values in an arbitrary order as long as it can obtain all necessary values.