A Comprehensive Study of Deep Bidirectional LSTM RNNs for Acoustic Modeling in Speech Recognition
Table 3: Comparison of different optimization methods. 3 layers, hidden layer size 500, dropout 0.1, n\scriptsize chunks=20. WER5, WER10, bWER and ep is the eval10 WER[%] of epoch 5, 10, best WER[%] and the epoch of the best WER, respectively.
['method', 'lr', 'details', 'WER5', 'WER10', 'bWER', 'ep']
We compare many optimization methods and variations between hyperparameters and esp. We compare stochastic gradient descent (SGD), SGD with momentum where one variant only depends on the last minibatch (mom) and another variant depends on the full history (mom2), SGD with Nesterov momentum (Adam with incorporated Nesterov momentum) but it did not converge in any of our experiments for this ASR task. We also test the effect of Newbob. Note that we only use 3 layers, no L2 and a smaller n\scriptsize chunks, which leads to worse results here compared to some other sections.