Beyond User Self-Reported Likert Scale Ratings: A Comparison Model for Automatic Dialog Evaluation
Table 4: Test accuracy and kappa agreement comparison among variants of CMADE.
['[BOLD] No.', '[BOLD] Model', '[BOLD] Test Acc.', '[BOLD] Kappa  [ITALIC] κ', '[BOLD] Kappa  [ITALIC] SE']
We also report the kappa agreement cohen1968weightedkappa (kappa κ and Standard Error SE) between the predicted output and the expert annotations. (1) BERT-Classification and (2) BERT-Regression fine tune the pre-trained BERT to perform a 5-class classification and regression respectively directly using the noisy self-reported ratings. To test BERT-Classification on dialog pairs, we apply the DEX trick ageEstimationDEX to get a floating-point number of predicted rating and thus get rid of the cases when the model predicts the dialog pairs as tie. (3) BERT-Pairwise shares the same model architecture with CMADE. It constructs dialog pairs for training by randomly sample dialog pairs xi and xj and derive zi,j by comparing the corresponding self-reported user rating yi and yj. We discard the pairs with equal yi and yj. (4) BERT-Pairwise+Dev augments (3) by adding the 200 expert annotated dialog pairs in the development into the training data. We also compare the variants of CMADE which skips one or two of the three stages.