Investigation of Error Simulation Techniques for Learning Dialog Policies for Conversational Error Recovery
Table 3: Discriminator performance (poorer performance means more realistically simulated output)
['[EMPTY]', '[BOLD] Accuracy', '[BOLD] Precision', '[BOLD] Recall', '[BOLD] F-Score']
Poorer discriminator performance means more realistically simulated ASR output. Disregarding ASR scores, the discriminator has low accuracy on both text- and audio-level methods at 60% and 63%, respectively, indicating that the simulated hypotheses from both methods are realistic. In comparison, the accuracy for both text-level methods have much larger increases. For text-level method with regression, it is easy to see why the scores can be a strong predictor, given the large score distribution differences between the real and predicted scores by this method For text-level method with classification, the reason is more subtle and a result of the score model only predicting the most frequent bin for hypothesis-reference pairs that have multiple duplicates with different real ASR scores. The training and test set sizes reduced to 20,721 and 7,639, respectively. We see that the discriminator performance for the text-level method with classification is now very close to when scores are excluded.