Deep LSTM for Large Vocabulary Continuous Speech Recognition
Table 3: The CER and RTF of 9-layers, 2-layers regular-trained and 2-laryers distilled LSTM.
['[BOLD] Models', '[BOLD] CER (%)', '[BOLD] RTF']
9-layers unidirectional LSTM model achieves outstanding performance, but meanwhile it is too computationally expensive to allow deployment in real time recognition system. In order to ensure real-time of the system, the number of layers needs to be reduced. The shallower network can learn the knowledge of deeper network with distillation. It is found that RTF of 2-layers network is acceptable, so the knowledge is distilled from 9-layers well-trained model to 2-layers model. The knowledge of deep network is almost transfered with distillation, Distillation brings promising RTF reduction, but only little knowledge of deep network is lost. Moreover, CER of 2-layers distilled LSTM decreases relative 14%, compared with 2-layers regular-trained LSTM.