Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
Table 2: Comparison of WER on a training and development set with and without SortaGrad, and with and without batch normalization.
['[EMPTY]', 'Train Baseline', 'Train Baseline', 'Train BatchNorm', 'Dev Baseline', 'Dev Baseline', 'Dev BatchNorm']
In the first training epoch, we iterate through the training set in increasing order of the length of the longest utterance in the minibatch. After the first epoch, training reverts back to a random order over minibatches. This effect is particularly pronounced for networks without BatchNorm, since they are numerically less stable. In some sense the two techniques substitute for one another, though we still find gains when applying SortaGrad and BatchNorm together. Even with BatchNorm we find that this curriculum improves numerical stability and sensitivity to small changes in training. Numerical instability can arise from different transcendental function implementations in the CPU and the GPU, especially when computing the CTC cost. This curriculum gives comparable results for both implementations.