Towards Personalized Dialog Policies for Conversational Skill Discovery
Table 3: Success rate and average dialog length for the rule-based and RL dialog policies.
['[BOLD] Policy', '[BOLD] Number', '[BOLD] Success', '[BOLD] Avg. Dialog']
To evaluate the performance of the skill discovery agent, we deployed the dialog policies and evaluated them with real users (see Appendix A for examples of dialogs). We first conducted a test with a baseline policy of recommending up to five skills based on popularity and allowing the user to either accept or reject the recommendation. The success rate of this simple policy was 46.42%, illustrating the importance of providing flexible search to the user. We then conducted an A/B test on the rule-based and RL policies to compare their effects on skill launches in a production environment. Both policies were tested on randomly sampled users, with the additional constraint of using the same policy for returning users. Both policies significantly outperform the baseline policy, indicating the importance of providing flexible search and navigation to users. Additionally, the difference between the success rate of the rule-based (73.41%) and RL (76.99%) policies is statistically significant (p-value < 0.0001) and the RL policy has significantly shorter dialogs (p-value < 0.0001), showing the importance of optimizing for the entire interaction with the user.