Deliberation Model Based Two-Pass End-to-End Speech Recognition
Table 4: Comparison of RNN-T, LAS two-pass and deliberation models in WERs (%) and GFLOPS. LAS two-pass and deliberation models are augmented with AE layers. All models are trained with MWER loss except the RNN-T model.
['ID', 'Model', 'Decoding', 'WER (%) VS', 'WER (%) SxS', 'WER (%) Songs', 'WER (%) Contacts-Real', 'WER (%) Contacts-TTS', 'WER (%) Apps', 'Estimated GFLOPS']
We include two deliberation models: An MWER trained 8-hypothesis deliberation model with AE layers (E9), and a jointly trained version (E10). We note that all models are MWER trained except the RNN-T model, which we find little improvement. First, we note that two-pass models perform substantially better than RNN-T (B0) in both VS task (15%–25% WERR) and rare word test sets (e.g. up to 30% in E10 for the SxS set). This confirms that second-pass decoding brings additional benefits. Second, the MWER trained 8-hypothesis deliberation model with AE layers (E9) performs significantly better than LAS rescoring (B4) or beam search (B5). When beam search is used for both of the deliberation and LAS models, the WERR is 7% for VS, and 8% for the SxS set. We observe significant improvements for voice command test sets too. Third, joint training (E10) brings an additional 2% relative improvement for VS, 9% for the SxS set, and uniform improvements for voice command test sets. Since the deliberation decoder has a larger size than LAS decoder (67MB vs. 33MB), it requires around 1.8 times GFLOPS as LAS rescoring. The increase mainly comes from the bidirectional encoder for 8 first-pass hypotheses. Latency estimation is complicated, and we will quantify that in future works.