Survey on Evaluation Methods for Dialogue Systems
Table 6: Predictive power of PARADISE. Where ALL denotes that the collection of all the annotated data from the three different systems. The distinction between NOVICES and EXPERTS denotes the level to which the test subjects were instructed to use the dialogue system.
['Training Set', '[ITALIC] R2 Training (SE)', 'Test Set', '[ITALIC] R2 Test (SE)']
Thus, PARADISE models the (subjective) performance of the system with a linear combination of objective measures (task-success and dialogue costs). Applying multiple linear regressions showed that only the task-success measure and the number of repetitions are significant. For this, they applied PARADISE on three different dialogue systems: ELVIS (a dialogue system for accessing emails), ANNIE (a dialogue system for voice dialing and messaging), and TOOT (a dialogue system for accessing train schedules). In a large-scale user study, they collected 544 dialogues over 42 hours of speech. For these experiments, the authors worked with an extended number of quality measures: e.g. number of barge-ins (i.e. sudden interruption by the user), number of cancel operations, number of help requests. A survey at the end of the dialogue was used to measure the user satisfaction. The survey asked about various aspects: e.g. speech recognition performance, ease of the task, if the user would use the system again. Based on the survey, the user satisfaction score is computed and used as the target variable to train the PARADISE framework as described above.