Deep Recurrent Neural Networks for Acoustic Modelling
Table 1: WERs for Wall Street Journal. The ReLU DNN requires no pretraining and matches the WER of the Kaldi s5 recipe which uses DBN pretraining.
['[BOLD] Model', 'dev93 WER', 'eval92 WER']
We also built a ReLU DNN which requires no pretraining. The ReLU DNN consisted of 4 layers of 2048 ReLU neurons followed by softmax and trained with geometrically decayed SGD. We also experimented with deeper and wider networks, however we found this 5 layer architecture to be the best. Our ReLU DNN is much easier to train (e.g., no expensive pretraining) and achieves a WER of 3.79 matching the WER of the pretrained Sigmoid DNN. The ReLU DNN results suggest that pretraining may not be necessary given sufficient supervised data and is competitive for the acoustic modelling task.