A Deep Reinforcement Learning Chatbot
Table 6: First A/B testing experiment with six different policies (± 95% confidence intervals). Star ∗ indicates policy is significantly better than other policies at 95% statistical significance level.
['[BOLD] Policy', '[BOLD] User score', '[BOLD] Dialogue length', '[BOLD] Pos. utterances', '[BOLD] Neg. utterances']
Policy Evaluation The results are given in The table shows the average Alexa user scores, average dialogue length, average percentage of positive user utterances and average percentage of negative user utterances. In total, over a thousand user ratings were collected after discarding returning users. Ratings were collected after the end of the semi-finals competition, where all ratings had been transcribed by human annotators. Each policy was evaluated by about two hundred unique Alexa users.