Automatic Quality Estimation for ASR System Combination
Table 1: WER[%] (↓) results of different hypothesis ranking strategies on IWSLT and CHiME-3 test data.
['[BOLD] Ranking', 'IWSLT (best WER=13.5%) L1', 'IWSLT (best WER=13.5%) L3', 'IWSLT (best WER=13.5%) L5', 'IWSLT (best WER=13.5%) L8', 'CHiME-3 (best WER=32.6%) L1', 'CHiME-3 (best WER=32.6%) L3', 'CHiME-3 (best WER=32.6%) L5', 'CHiME-3 (best WER=32.6%) L10']
(SegO) when the best k hypotheses (Lk) are used for the combination (for IWSLT k=1,3,5,8 while, for CHiME-3, k=1,3,5,10). At both levels of granularity, oracle rankings are obtained by measuring the true WER of the candidate transcriptions. As it can be seen in the table, hypothesis combination allows for considerable WER reductions compared to the best individual system, both at system and at segment-level. In particular, on IWSLT, system-level combination results in a WER reduction from 13.5% (best individual system) to 11.8% when combining the best 5 systems. Note that, in both tasks, system-level combination achieves the best results with only 5 different hypotheses, which is less than all the available ones. This can be explained by observing that all systems/channels contribute to the final voting with the same weight. Therefore, the insertion of the worst hypotheses in the ranked list contributes to worsen rather than to improve global performance. The poor results achieved, especially at lower combination levels, demonstrate that correctly ranking the input hypotheses is essential to improve ROVER combination results. To address this problem, we investigate two different strategies: ranking by regression (RR) and machine-learned ranking (MLR).