Deliberation Model Based Two-Pass End-to-End Speech Recognition
Table 1: WERs (%) of deliberation with different number of RNN-T hypotheses.
['ID', 'E1', 'E2', 'E3', 'E4']
The deliberation decoder may attend to multiple first-pass hypotheses. We encode the hypotheses separately, and then concatenate them as the input to the attention layer. We use a beam size of 8 for RNN-T decoding. Unless stated otherwise, the WER we report is for VS test set. However, after applying MWER training, the WER improves continuously: 5.4% to 5.1%. We suspect that MWER training specifically helps deliberation attend to relevant parts of first-pass hypotheses. Since 8-hypothesis model gives the best performance, we use that for subsequent experiments. MWER training is not used for simplicity.