Constructing Long Short-Term Memory based Deep Recurrent Neural Networks for Large Vocabulary Speech Recognition
Table 2: Speech recognition results of different strategies of constructing deep LSTM networks.
['Model Descriptions', 'CER(%)']
In order to construct a LSTM network with deep input-to-hidden function, we constructed a LSTM network by putting a LSTM network on three feed-forward intermediate layers, and each feed-forward layer had 2000 ReLU units. Similarly, we trained a model indicated as “2-layer Conv + 2-layer ReLU + LSTM”. The stacked LSTMs network was also evaluated, in which, three conventional LSTMs were stacked, and each layer had 750 LSTM cells. Concretely, in the training procedure of “3-layer ReLU + LSTM”, three ReLU hidden layers were firstly pre-trained, and then the original output softmax layer was replaced by a new random initialized LSTM layer along with a new output softmax layer. Finally, the whole network was jointly optimized. Through making deep hidden-to-hidden transitions, obvious performance improvements can be obtained, especially the LSTM-OP. Besides, the performance can also been improved by making deep input-to-hidden and hidden-to-output functions.