Towards a Metric for Automated Conversational Dialogue System Evaluation and Improvement
Table 2: Human judgements on the dialogue level for each dialogue system. For this, a each dialogue system (the five base-systems and the re-ranking system) generate 100 dialogues using self-talk, which human judges rated on the dialogue level. Here we see the average ratings for each system.
['Systems', 'Dialogue Level Rating']
In order to evaluate the improvements achieved by the re-ranking method, we sample a disjoint set of 100 new contexts and apply self-talk to generate conversations. Then, we use AMT to let humans judge the automatically generated conversations on the dialogue level (i.e. a rating for the entire dialogue as opposed to turn-based ratings). We compare the performance of the five base dialogue systems to the performance of the re-ranking strategy. Our results show that the re-ranking approach works very well. It raises the score to 3.47, which is 0.16 points higher than the best base-system (i.e. Seq2Seq).