Studying the Effects of Cognitive Biases in Evaluation of Conversational Agents
Table 5: ICC scores on the metrics of readability and coherence for each experiment condition. All values are statistically significant p-value<0.001
['Setup 1', 'No Anchor (n=18)', 'Readability 0.74', 'Coherence 0.76']
We find that there is a significant (p<0.001) increase in the consistency of the ratings in the anchor condition in Setup 1. The consistency values obtained in Setup 2 for readability and coherence show mixed results. We find that the no anchor condition of Setup 2 produces more consistency in ratings for the readability metrics whilst on the metric of coherence, we find that there is extremely low consistency between the raters when they are presented with no anchors. However, we see a significant increase in consistency for Setup 2, when participants are in anchoring condition. We also notice the significant impact anchoring has towards increasing consistency of ratings. We see that it seems harder to agree upon the more subjective metric of coherence, without any textual or numerical anchor. We also suspect the impact of instructions might have towards consistency. In the instructions screen in our study, Readability was defined as: Is the response easy to understand, fluent and grammatical and does not have any consecutive repeating words (following [novikova-etal-2017-need, novikova-etal-2018-rankme]), which provides clear indicators regarding evaluating a response on the metric of readability. Coherence was defined as: Is the response relevant to the topic and context of the conversation. (following [dziri2018augmenting, venkatesh2018evaluating] making it more subjective.