A Comprehensive Study of Deep Bidirectional LSTM RNNs for Acoustic Modeling in Speech Recognition
Table 5: Comparison of pretraining and no pretraining, compared for different final number of layers, cf. Table 1.
['#layers', 'WER[%] no pretrain', 'WER[%] pretrain']
For deeper networks, this scheme seems to help more. This indicates that our initialization might have room for improvement. We got our overall best result with such a pretraining scheme applied for a 6 layer bidirectional LSTM and we note that esp. for the deeper networks, the improvements by pretraining increases. We were not able to train a 9 layer BLSTM without pretraining, it diverged and broke after two epochs. Also, the training calculation time of the first few epochs is shorter.