Two-Pass End-to-End Speech Recognition
Table 2: WER results, LAS Rescoring.
['Exp-ID', 'Decoding', 'SU', 'LU']
We noticed that the RNN-T-only model (B0) has much lower oracle WERs than its decoding WERs. This motivates us to explore rescoring RNN-T hypothesis with the LAS decoder. The table shows that rescoring takes a small hit in WER on SU compared to beam search, probably because the first-pass RNN-T decoder, with a much higher SU WER of 6.9 (B0), generates a set of hypotheses with slightly lower quality than those generated by the LAS decoder during beam search. However, rescoring’s quality on LU is much better than that of beam search, likely because RNN-T (B0) performs much better on longer utterances compared to LAS. Overall, LAS rescoring not only improves SU WER significantly upon the first pass RNN-T, but also improves WER for LU, demonstrating that rescoring is able to combine the strengths of RNN-T and LAS. Since rescoring gives us the best tradeoff between quality on SU and LU, we will focus only on rescoring and present further improvements in the next section.