Predictive Engagement: An Efficient Metric for Automatic Evaluation of Open-Domain Dialogue Systems
Table 5: Pearson and Spearman correlations between human judgements and several automatic dialogue evaluation metrics on generated responses, human-written responses, and their mixture. We adopt the mean aggregation of the relevance score of contextualized RUBER mentioned as Ctx_RUBER and the predictive utterance-level engagement scores. The first two rows in each group show correlations between human judgements and baseline models with only relevance scores, the middle two rows are for only engagement scores and the last two rows add engagement scores into relevance scores. Boldface indicates the improvements are significant compared to the baseline in the corresponding group (p 
['[BOLD] Dataset', '[BOLD] Metric', '[BOLD] Pearson', '[BOLD] Spearman']
We inferred the engagement scores from fine-tuned utterance-level engagement models for the 300 utterances with generated replies and aggregated them with the relevance scores obtained from the Contextualized RUBER model. We only included the mean aggregation of relevance and engagement metrics that resulted in the highest correlation in comparison with the other two aggregations (minimum and maximum) that we tried. Many off-topic replies generated by the attention-based sequence-to-sequence dialogue system could be the reason for this observation. Incorporating engagement scores leads to higher correlations with human judgements. Indeed, the baseline models score the majority of human-written responses very high, while users consider other aspects such as engagement for giving the utterance an overall quality score. This is probably because the annotators do not pay attention to other aspects like engagement for evaluating a response that is not relevant to a given query. The higher correlations between human annotations with relevance and engagement scores illustrate the success of applying engagement as an extra score to baseline metrics in order to have a better automatic evaluation system.