Towards a Metric for Automated Conversational Dialogue System Evaluation and Improvement
Table 1: Average correlations between the judgements predicted by AutoJudge and the human judgement. Convo Split denotes the cross-validation split according to the contexts and System Split denotes the cross-validation split according to the dialogue system.
['[EMPTY]', 'Pearson Corr', 'Spearman’s Rho', 'MAE']
We train AutoJuge using k-fold cross validation. There are two ways of splitting the data into folds, in order to ensure that all turns of the same dialogue are in the same fold. First, we group the 100 contexts into 10 folds, thus, each fold consists of 50 dialogues ( i.e. 10 contexts times the number of dialogue systems), this is denoted as Convo Split. The second option is to split the data according to the system which created the conversation, which evaluates the performance of AutoJuge in rating dialogues of unseen dialogue systems. We denote this as System Split. With moderate correlations of 0.573 on the dialogue level, we get results which are comparable to Lowe et al. Note that we cannot directly compare our results to BLEU score and ADEM, since these base their predictions on gold standards, which we do not have in our setting. An interesting result is the System Split, i.e. that our approach is able to maintain a high correlation (0.544) with the ratings of a dialogue system when removing the data of that system from the training, which is not the case in Lowe et al.