Investigation of Error Simulation Techniques for Learning Dialog Policies for Conversational Error Recovery
Table 2: Evaluation of ASR confidence score prediction models
['[EMPTY]', '[BOLD] Linear Correlation', '[BOLD] Mean Abs. Error']
Both regression and classification approaches substantially outperformed the baseline and yielded predictions that are close to the real scores in terms of correlation and MAE. The predictions from the classification approach match the ground-truth scores much better in distribution but have slightly smaller correlation and larger MAE compared to the predictions from the regression approach. It is worth pointing out that regardless of the score model, there is an inherent gap between scores predicted from the hypothesis-reference pairs and real scores, because for the same hypothesis-reference pair, the real score could still vary based on the audio quality.