Re-evaluating ADEM: A Deeper Look at Scoring Dialogue Responses
Table 3: ADEM scores on simple test cases (∗Machine generated responses are obtained by training a GAN based neural dialogue generation model [Li et al.2017])
['[BOLD] Response to be evaluated', '[BOLD] ADEM mean', '[BOLD] ideal score']
Borrowing the idea of the ERE metric proposed by \citeauthorli2017adversarial (\citeyearli2017adversarial), we evaluate ADEM on certain similar straightforward cases with easy-to-guess desired outcomes. While the ERE metric deals with a binary classification of dialogues as human versus machine-generated, we adopt the idea for a score based evaluators like ADEM. We first check ADEM ’s evaluations of the reference responses. One would expect the scores on good reference response to be high. Note that the reference responses for dialogues in this dataset have high human ratings. Similarly, when the context itself is repeated back as the response, it forms an unnatural reply in most cases and one can expect scores to be very low. Next we check ADEM ’s scores in the case of the machine generated responses. We train a GAN based neural dialogue generation model based on \citeauthorli2017adversarial (\citeyearli2017adversarial) for a very limited number of 5 epochs before using the generated responses for evaluation. We deem this training to be insufficient and hence expect an ideal evaluator’s scores to be low. Finally we think of the case where the reference response and the responses-to-be-evaluated (in this case, the machine generated responses) are swapped. The idea is to find out how much the reference response influences the evaluation of a response. In the numerous dialogue datasets which often involve some automation in their creation, it would not be uncommon to find certain mediocre reference responses. A human evaluator merely uses the reference response as an unabiding guideline and can recognize a better-suited response without being misled/distracted by a bad reference response. We hence suggest an ideal score of 5 for this scenario. In all of these cases, we find ADEM’s score to be quite similar. We find that ADEM has a mean score between 2.6−3 in all of these cases with a very low standard deviation ranging between 0.32−0.36. We also find 60−71% of the scores within 1 standard deviation of the mean.