Beyond User Self-Reported Likert Scale Ratings: A Comparison Model for Automatic Dialog Evaluation
Table 2: The correlation between the self-reported Likert scale ratings and our pair-wise comparison annotation. For a pair of dialogs, if the delta of self-reported Likert scale ratings is large, then they are more likely to align with the comparison results from experts.
['Delta of Self-Reported Ratings (e.g., 5-1=4)', 'Δ [BOLD] =1', 'Δ [BOLD] =2', 'Δ [BOLD] =3', 'Δ [BOLD] =4']
To further analyze the self-reported Likert scale ratings, we also compare the annotated labels of the 403 dialog pairs with the self-reported Likert scale ratings of these dialogs. For each pair of dialogs, we compare the pairwise comparison label and the delta between the self-reported Likert scale ratings of the two dialogs. Ideally, the dialog with a higher self-reported Likert scale rating should be the one that is annotated as having a better user experience in the pairwise comparison. We count the number and fraction of “disagreement” between the two types of ratings. Overall, roughly 1/3 of the dialog pairs disagree. This suggests that if the difference between the two dialogs’ Likert score is huge, they are more likely to be consistent with the comparison ratings.