A Comprehensive Study of Deep Bidirectional LSTM RNNs for Acoustic Modeling in Speech Recognition
Table 2: Comparison of hidden layer size. 5 layers, dropout 0.1, L2 0.01, Adam, n\scriptsize chunks=40. WER reported on eval10, reported for the best epoch.
['layer size', '#params[M]', 'WER[%]', 'epoch']
In most experiments, we use a hidden layer size of 500 (i. e. 500 nodes / memory cells for each the forward and the backward direction). Note that the number of parameters increases quadratically. We see that the optimum for this experiment is at about 600-700 (for 3 layers with Adadelta at about 700), however a model with size 500 is much smaller and not so much worse, so we used that size for most other experiments.