Towards End-to-End Learning for Dialog State Tracking and Management using Deep Reinforcement Learning
Table 3: Performance of the three systems
['[EMPTY]', 'Win Rate (%)', 'Avg Turn']
The horizontal axis is the total number of interaction between the agent and either the user or the database. The baseline model has the fastest learning speed but its performance saturated quickly because the dialog policy was not trained together with the state tracker. So the dialog policy is not aware of the uncertainty in slot-filling and the slot-filler does not distinguish between the consequences of different wrong labels (e.g classify yes to no versus to unknown). On the other hand, although RL reaches high performance at the end of the training, it struggles in the early stages and suffers from slow convergence. This is due to that fact that correct slot-filling is a prerequisite for winning 20Q, while the reward signal has a long delayed horizon in the RL approach. Finally, the hybrid-RL approach is able to converge to the optimal solution much faster than RL due to the fact that it efficiently exploits the information in the state tracking label.