Neural Zero-Inflated Quality Estimation ModelFor Automatic Speech Recognition System
Table 3: The comparison of different quality estimation models.
['Method', 'MAE', 'Pearson', 'F1-OK/BAD']
The second experiment we conduct on our in-domain test dataset is to compare our ASR-QE model as an integrated pipeline with the state-of-the-art quality estimation model Notice that for fair comparison, we have to modify the text encoder of QEBrain to the exactly same speech encoder of ours and the last layer to a zero-inflated regression one. In addition to previous metrics, we also introduce F1-OK/BAD to evaluate the recognition result is acceptable or not, which is prevalent in word-level quality estimation of machine translation. It shows that speech-BERT outperform QEBrain in all aspects. Furthermore, we have a detailed analysis on the Pearson with respect to different sentence lengths in Fig. We simply use two linear regressions fit the trend of performance decreasing when the sentence length grows. QEBrain demonstrates more excellent performance when the sentence is shorter, but speech-BERT has a stable performance across all length ranges. This finding makes sense because the longer sentence is likely to have lower even zero WER, which can be better dealt with by zero-inflated Beta regression layer.