A Deep Reinforcement Learning Chatbot
Table 2: Scoring model evaluation on Amazon Mechanical Turk test set w.r.t. Pearson correlation coefficient, Spearman’s rank correlation coefficient and mean squared error.
['[BOLD] Model', '[BOLD] Pearson', '[BOLD] Spearman', '[BOLD] Mean squared error']
The metrics are computed after linearly transforming the AMT class categories to the scalar output score (i.e. by taking the dot-product between the one-hot class vector and the vector [1.0,2.0,3.0,4.0,5.0]). The Average Predictor is a baseline model, which always predicts with the average output score. As shown, Supervised AMT achieves a Pearson correlation coefficient of 0.40, a Spearman’s rank correlation coefficient of 0.38 and a significant reduction in mean squared error. This indicates Supervised AMT performs significantly better than the baseline.