A Deep Reinforcement Learning Chatbot
Table 8: First A/B testing experiment user scores with six different policies w.r.t. varying dialogue length (± one standard deviation).
['[BOLD] Policy', '[BOLD] Dialogue length  [BOLD] < 10', '[BOLD] Dialogue length  [BOLD] 10 - 19', '[BOLD] Dialogue length  [BOLD] 20 - 39', '[BOLD] Dialogue length ≥ [BOLD]  40']
The estimates are based on between 30-70 Alexa user ratings for each policy and interval combination. First, we observe that Q-learning AMT performs better than all other policies for all intervals except the medium-short interval (10−19, or 5−10 back-and-forth turns). Further, its high performance for the long intervals (20−39 and ≥40) would suggest that Q-learning AMT performs excellent in long dialogues. The other learned policies Supervised AMT, Off-policy REINFORCE and Off-policy REINFORCE Learned Reward also appear to perform excellent in long dialogues. On the other hand, the heuristic Evibot + Alicebot policy and the Supervised Learned Reward policy appear to perform poorly in long dialogues, but that is not surprising given their low overall performance. In particular, Supervised Learned Reward seems to be performing well only for very short dialogues. This potentially indicates that the policy fails to either maintain user engagement or memorize longer-term context. However, further investigation is required.