Exploring Pre-training with Alignments for RNN Transducer based End-to-End Speech Recognition
Table 2: WER comparison of different encoder pre-training methods on 3400 hours Cortana data. Greedy search is used. ’CTC’ means pre-training of CTC loss with target sequence, ’CE’ means pre-training of CE loss with target alignment. ’no’ means training from the random initialization.
['Model', 'Enc. Pre-train', 'WER (%)']
Using a pre-trained CTC model to initialize the encoder does not improve the accuracy. This is because the output of CTC is a sequence of spikes, in which there are lots of blank tokens without any meaning. Hence, if we use the pre-trained CTC as the seed for the encoder of RNN-T, most encoder output henct will generate blank, which does not help the RNN-T training. When we use CE loss pre-trained encoder to initialize the encoder of RNN-T, it achieves significant improvement compared with training from the random initialization. It obtains 28% relative WER reduction from the RNN-T baseline and CTC based encoder pre-training. The only difference is the parameters seed of encoder. When comparing CTC loss and CE loss based encoder pre-training methods, there is a huge WER gap between these two approaches. Initializing the encoder as a token aligning model rather than a sequence mapping model results in the much better accuracy. This is because the RNN-T encoder performs the frame-to-token aligning, which extracts the high-level features of each input frame.