A Deep Reinforcement Learning Chatbot
Table 3: Policy evaluation on AMT w.r.t. score mean and score standard deviation (std). 90% confidence intervals are given for means (after ±) and standard deviations (in square brackets).
['[BOLD] Policy', '[BOLD] Full test set  [BOLD] Score mean', '[BOLD] Full test set  [BOLD] Score std', '[BOLD] Difficult test set  [BOLD] Score mean', '[BOLD] Difficult test set  [BOLD] Score std']
The results show that the three learned policies are all significantly better w.r.t. mean score compared to both Alicebot and Evibot + Alicebot. Not surprisingly, this difference is amplified on the difficult test set. Q-learning AMT, Supervised AMT and Off-policy REINFORCE appear to perform overall equally well. This shows that machine learning has helped learn effective policies, able to select other model responses when neither the Alicebot and Evibot responses are appropriate. Next, the results show that Q-learning AMT has higher standard deviations than the other policies on both the full test set and the difficult test set. Furthermore, since these standard deviations are evaluated at the level of a single response, we might expect this variability to compound throughout an entire conversation. This strongly indicates that Q-learning AMT is more risk tolerant than the other policies.