Towards Coherent and Engaging Spoken Dialog Response Generation Using Automatic Conversation Evaluators
Table 7: Mean ratings for Qualitative and Human Evaluation of Response Generators
['[BOLD] Metric', 'Coherence', 'Engagement']
As noted earlier, automatic evaluation metrics may not be the best way to measure chatbot response generation performance. Therefore, we performed human evaluation of our models. We asked annotators to provide ratings on the system responses from the models we evaluated, i.e., baseline model, S2S_RR, S2S_FT, and S2S_RR_FT. A rating was obtained on two metrics: coherence and engagement. Coherence measures how much the response is comprehensible and relevant to a user’s request and engagement shows interestingness of the response (venkatesh2018evaluating). We asked the annotators to provide the rating based on a scale of 1–5, with 5 being the best. We had four annotators rate 250 interactions. Our inter-annotator agreement is 0.42 on Cohen’s Kappa Coefficient, which implies moderate agreement. We believe this is because the task is relatively subjective and the conversations were performed in the challenging open-domain setting. The S2S_RR_FT model provides the best performance across all the metrics, followed by S2S_RR, followed by S2S_FT.