On Evaluating and Comparing Open Domain Dialog Systems
Table 5: Correlation of the regression model with user ratings
['Algorithm', 'RMSE', 'Spearman', 'Pearson']
We did a preliminary analysis on 60,000 conversations and ratings and we trained a model to predict user ratings. Although the results for GBDT are significantly better than Random selection for 5 classes and the model trained using Hierarchical LSTM, there is a need to extend this study the millions of Alexa Prize conversations. Furthermore, some of the evaluation metrics (coherence, topical depth, topical breadth, domain coverage, etc.) obtained at conversation level can also be used as features. Given subjectivity in ratings, we appropriately found inter-user agreement to be quite low for ratings analysis. A user might give a conversation 5 stars because he/ she thought the socialbot was humorous, while another user might find it unknowledgeable. Users may have their own criteria to evaluate the socialbots. Therefore, as a part of the future work, we would like to train the model with user level features as well. This experiment was done to obtain the potential of automating the ratings.