Exploring Pre-training with Alignments for RNN Transducer based End-to-End Speech Recognition
Table 1: WER comparison of different whole-network pre-training methods on 3400 hours Cortana data. Pre-train (all align) uses y1, Pre-train (correct decoding) uses y2, and Pre-train (align path - sp blank) uses y3 as the target label sensors, respectively.
['Model', 'WER(%)']
The RNN-T baseline is trained from random initialization. For pre-trained models, the whole network is pre-trained with CE loss, then it is trained with RNN-T loss. Using the pre-trained network as a seed model, the final word error rate (WER) can be significantly reduced. All designed label tensors can improve the RNN-T training, achieving 10% to 12% relative WER reduction.