A Corpus-free State2Seq User Simulator for Task-oriented Dialogue
Table 6: Evaluation of user simulator model.
['Model', 'Action Accuracy Movie', 'Action Accuracy Restaurant', 'Generalization Ability Test Movie', 'Generalization Ability Test Movie', 'Generalization Ability Test Restaurant', 'Generalization Ability Test Restaurant']
Action accuracy. The results also show a correlation between the evaluation of action prediction and the agent policy’s performance, which demonstrates that user simulator quality affects agents performance. Analysis of generalization ability The generalization ability is also important for a user simulator in agent training. We compare different user simulators’ generalization by making them chat with a same rule-based agent. Our best model outperforms the Agenda for 5.0 and 7.4 points success rate on the 2 domains. As the user simulators are trained to mimic user rules in the template, the improvement reflects that our framework can generate new diverse dialogues data to avoid user simulator overfitting to user response strategy in the template. Other user simulators perform worse on this test, we address this to the fact that these methods are less accurate in generating user actions and rule-based agent has a low allowance for response error.