Performance Monitoring for End-to-End Speech Recognition
Table 2: Mean Square Error (×10−2) of linear regression trained on performance monitoring techniques. All results are reported on test sets.
['Dataset/Domain', 'Entropy Dec', 'Entropy Att', 'MCD Dec', 'MCD Att', 'AE', 'RNN']
It is worth noting that decoder features work more effectively than attention probabilities in all cases for predicting CERs. Entropy gives the lowest MSEs for WSJ and Aurora4, domains which have been seen in ASR training. The RNN predictor achieves best performance in CHiME4-Sim (not surprising, since this domain was seen in RNN predictor training) as well as real recordings from CHiME4-Real and Dirha-Real, domains not seen in any training stage at all. MCD measure performs the best at the two unseen simulated domains with reverberant conditions. Overall, entropy, MCD, and RNN prediction all provide reasonably good CER predictions, with average prediction errors (square root of MSE) of 10.1%, 8.8% and 10.1%, respectively, where MCD outperforms the rest of PM measures across all test set.