Investigation of Error Simulation Techniques for Learning Dialog Policies for Conversational Error Recovery
Table 6: Cell and marginal means of the dialog ratings
['Volume Low', '[BOLD] Audio-level 3.06', '[BOLD] Text-level 3.38', '3.22']
We also conducted a user study to compare the real-world performances of the dialog policies learned with audio- and text-level techniques. To increase the statistical power given the limited number of subjects, we adopted the within-subjects experimental design and analysis Lane et al. More specifically, we recruited 16 internal subjects and considered two treatment factors: noise volume (high, low) and error simulation technique (audio-level, text-level), for a total of four unique treatment combinations. The sample size and number of treatment combinations are comparable to previously published studies under the same experimental design Pilla et al. We asked each subject to engage in four dialogs with MovieBot, one for each treatment combination. For example, if the treatment combination was "high noise volume + text-level technique", the subject would talk to the SLU application that adopted a dialog policy learned with the text-level error simulation technique, while a sound track of people talking on high volume was played in the background. We asked subjects to rate the quality of their most recent dialog on a 5-point scale at the end of each dialog. We followed the standard practice for conducting within-subjects experiments, including counter-balancing the order of treatment combinations. Note that by design the observations from the same subject are correlated. We performed the two-way repeated measures ANOVA Lane et al. The audio- and text-level techniques have the same marginal mean ratings (3.09), hence there is 0 main effect for "technique" in the ANOVA. The audio-level technique has lower mean ratings than text-level for low noise volume, and vice versa for high noise volume. However, the differences are not statistically significant based on the p-value of the interaction effect of "noise × technique" in the ANOVA. Overall, the quantitative results from the user study do not reveal significant differences between the two techniques.