On Evaluating and Comparing Open Domain Dialog Systems
Table 3: Correlation of Evaluation Metrics with User Ratings
['Metric', 'Users', 'Frequent-Users', 'Engagement Evaluators']
While aggregating evaluation metrics to come up with a unified metric, it is important to find the relevance of each of these metrics. For the Alexa Prize Competition, we evaluated the relevance by identifying the correlation of each of these metrics with user ratings and Frequent-User ratings. In this section we share our findings. Conversational User Experience: To capture subjectivity involved with CUX, as mentioned earlier we used Frequent-User ratings as a measure of evaluation. As a part of future work, we plan to train a model to predict CUX based on Frequent-User ratings seperately. As described earlier, ratings from Frequent-Users are capable of measuring subjectivity, expectation and other CUX metrics. It can be observed from the fact though these users come from the same pool of all Alexa users, still their average ratings are lower than average of all Alexa user ratings. Coherence: For coherence, we evaluated response error rate (RER).