Predictive Engagement: An Efficient Metric for Automatic Evaluation of Open-Domain Dialogue Systems
Table 2: The results for the Amazon Mechanical Turk (AMT) experiments on utterance-level engagement. 49 annotators annotated 297 utterances and demonstrated quite high inter-annotator Kappa agreement and Pearson correlation between annotations.
['[BOLD] Utterances', '[BOLD] Annotators', '[BOLD] Kappa Agreement', '[BOLD] Pearson']
We propose to first study whether humans are capable of scoring engagement of a response for a given query without knowing any context or previous utterances. To achieve this, we executed experiments to check users’ agreement level about engagement of each utterance. We conducted Amazon Mechanical Turk (AMT) experiments on randomly selected 50 conversations from ConvAI, 25 human-to-human and 25 human-to-bot dialogues. Overall 297 utterance pairs have been extracted and rated by annotators in the same range (1-5) of engagement score in ConvAI. 49 workers participated in about 215 surveys, where each utterance pair has been annotated by 5 individual workers. We rejected users that did not pass attention-check tests in the surveys and reassigned their pairs to other workers. This provides evidence that engagement can be measured not only at the conversation level but also at the utterance level.