Automatic Quality Estimation for ASR System Combination
Table 8: WER[%] (↓) of different ranking methods on IWSLT2012 using 4-fold CV.
['[BOLD] Ranking', 'L3', 'L4', 'L5', 'L6', 'Avg. Impr.']
, we first run QE-informed ROVER on the training set (IWSLT2012) using 4-fold cross validation. Data partitioning is done with regard to the speakers in order to guarantee that there is no speaker overlap between training and test folds. Moreover, the partitioning is done in such a way that each instance occurs only once in each test part. Although the various ASR systems provide different hypotheses for each decoded segment, the corresponding signal feature vectors are the same and, therefore, the trained regressor could be misled by the co-occurrence of similar feature values with different training labels. This approach, using both basic and word-based features (RR2+BW), yields the best performance among the RR methods (on average, the WER is reduced by 2.2%). Interestingly, both RR1 and RR2 reach the best performance at combination level L6. This behavior, differently from the oracle conditions, indicates that the majority of the words in each confusion set of the ROVER word network is not concentrated on the top “automatically” ranked positions but is more uniformly distributed over all combination levels. Machine-learned ranking with the basic features (MLR+B), word-based features (MLR+W) and their combination (MLR+BW) always exhibits the largest WER reductions over random ROVER (-2.4% on average). Its results are not only consistently better than RR, but also close to system-level oracle (SysO). In summary, this preliminary analysis suggests that: i) regardless of the QE method and the features used, QE-informed ROVER outperforms random ROVER with a large margin, ii) among all the QE strategies, MLR+BW shows the best performance at most of the levels, and iii) though still far from the strongest term of comparison (SegO), its results are competitive with those of the system-level oracle. -1.1%).