A Comprehensive Study of Deep Bidirectional LSTM RNNs for Acoustic Modeling in Speech Recognition
Table 1: Comparison of number of layers, layer size fixed to 500 for each forward and backward direction. Dropout 0.1 + L2, Adam, n\scriptsize chunks=40, WER on eval10, reported on the best epoch. Note that the CE values are not necessarily from the same epoch as the WER but they are the minimum from all epochs. Also, the train CE is accumulated while training, i.e. with dropout applied.
['#layers', '#params[M]', 'WER[%]', 'epoch', 'train CE', 'CV CE']
For this experiment, the optimum is somewhere between 4 to 6 layers. In earlier experiments, the optimum was at about 3 to 4 layers. It seems the more we improve other hyperparameters, the deeper the optimal network becomes. This gives a hint about the amount of overfitting. but they do not which is probably due to a harder optimization problem. It also seems as if the CV CE optimum is slightly deeper than the WER optimum. That indicates that sequence discriminative training will further improve the results.