Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
Table 1: Comparison of WER on a training and development set for various depths of RNN, with and without BatchNorm. The number of parameters is kept constant as the depth increases, thus the number of hidden units per layer decreases. All networks have 38 million parameters. The architecture “M RNN, N total” implies 1 layer of 1D convolution at the input, M consecutive bidirectional RNN layers, and the rest as fully-connected layers with N total layers in the network.
['Architecture', 'Hidden Units', 'Train Baseline', 'Train Baseline', 'Train BatchNorm', 'Dev Baseline', 'Dev Baseline', 'Dev BatchNorm']
For each hidden unit, we compute the mean and variance statistics over all items in the minibatch over the length of the sequence. When comparing depth, in order to control for model size we hold constant the total number of parameters and still see strong performance gains. We would expect to see even larger improvements from depth if we held constant the number of activations per layer and added layers. We also find that BatchNorm harms generalization error for the shallowest network just as it converges slower for shallower networks.