Automatic Quality Estimation for ASR System Combination
Table 9: WER[%] (↓) of different ranking methods on IWSLT2013. “†” = the result is not statistically different from random ROVER; “∙” = the result is not statistically different from SysO; “⋆” the result is not statistically different from SegO.
['[BOLD] Ranking', 'L3', 'L4', 'L5', 'L6', 'L7', 'L8', 'Avg. Impr']
S7SS1SSS0Px2 Test Using the QE models trained on IWSLT2012 , we carry out the same set of experiments on the test data. Since in IWSLT2013 there are 8 teams who submitted their transcriptions, in this case the combination level varies in [L3,L8] interval. Although the overall trend is similar to the one observed on the training data, the improvement margins are smaller. The reason for this unsurprising difference is the mismatch between training and test data. In fact, in our preliminary experiments, the same set of ASR systems (6 systems) was used both in training and test. Here, instead, the ASR systems are different in number and nature (6 systems from IWSLT2012 for training, 8 systems from IWSLT2013 for test). the WER reduction is large and significant. The best result (MLR+BW at L6), in particular, is not only better than random ROVER, but also not statistically different from the strongest oracle (SegO). Also on the test set, MLR seems to work better than RR in ranking the input hypotheses. One possible reason is the higher reliability of pairwise comparisons compared to considering numeric scores that reflect independent quality predictions. So far, we showed that: i) we can improve random ROVER using ASR QE for ranking the inputs, and ii) with an appropriate QE approach and an efficient set of features, we can approach the strong system-level and segment-level oracles.