Re-evaluating ADEM: A Deeper Look at Scoring Dialogue Responses
Table 5: Correlation of ADEM scores on different variants of the response with the ADEM scores on original (reference) response. p-values in all these cases are <0.001. The last column indicates the percentage of times the concerned variant received a better score than original
['[BOLD] Variant', '[BOLD] Pearson', '[BOLD] Spearman', '[BOLD] Better score']
This corpus contains a curated list of 3 turn twitter conversations, all of which received an average score of 4 or higher for good response quality by 3 crowd-sourced annotators apiece. Human scoring of the dialogues was performed on a 5-point scale similar to ADEM’s automated scoring on a scale [1-5] which makes it easier for comparison. This analysis on ADEM shows: multiple adversarial scenarios where the evaluator is easily fooled or does the opposite of what is expected the ADEM score in most cases is a conservative average value of around 2.75 Although these experiments are reported on ADEM model, they can be used to analyze any other dialog evaluator model. Evaluation of the Ground Truth Response to estimate the reliability of the evaluator. We evaluate the scores on the responses in the training set of MSR Social Media Conversation corpus, which are already rated high by human annotators. This first experiment revealed that the ADEM scores are concentrated around the average value of 2.75 which seems like a conservative middle value on a scale of [1-5]. The standard deviation of the scores is 0.34 with 71.65% of the scores within the first standard deviation. Slight Modifications such as removing punctuation and stopwords should probably lower the score only a little for a given response.