A Deep Reinforcement Learning Chatbot
Table 4: Off-policy evaluation w.r.t. expected (average) Alexa user score and number of time steps (excluding priority responses) on test set.
['[BOLD] Policy', '[BOLD] Alexa user score', '[BOLD] Time steps']
The expected (average) Here we observe that the Off-policy REINFORCE performs best followed by Q-learning AMT and Supervised AMT w.r.t. expected Alexa user score. Off-policy REINFORCE reaches 2.45, which is a major 17.8% improvement over the second best performing model Q-learning AMT. However, this advantage should be taken with a grain of salt. As discussed earlier, the off-policy evaluation in eq. Moreover, Off-policy REINFORCE has been trained specifically to maximize this biased estimator, while all other policies have been trained to maximize other objective functions. Similarly, w.r.t. expected number of time steps, Off-policy REINFORCE reaches the highest number of time steps followed by Q-learning AMT and Supervised AMT. As before, we should take this result with a grain of salt, since this evaluation is also biased and does not take into account priority responses. Further, it’s not clear that increasing the number of time steps will increase user scores. Nevertheless, Off-policy REINFORCE, Q-learning AMT and Supervised AMT appear to be our prime candidates for further experiments.