Building DNN Acoustic Models for Large Vocabulary Speech Recognition
TABLE II: Results for DNN systems trained with dropout regularization (DO) and early realignment (ER) to improve generalization performance. We build models with early realignment by starting realignment after each epoch starting after epoch two (ER2) and epoch five (ER5). Word error rates are reported on the combined Hub5 test set (EV) which contains Switchboard (SWBD) and CallHome (CH) evaluation subsets. DNN model sizes are shown in terms of hidden layer size and millions of total parameters (e.g. 100M)
['Model', 'SWBD', 'CH', 'EV']
DNNs trained with dropout improve over the baseline model for all acoustic model sizes we evaluate. The improvement is a consistent 0.2% to 0.4% reduction in absolute WER on the test set. While beneficial, dropout seems insufficient to fully harness the representational capacity of our largest models. Additionally, we note that hyper-parameter selection was critical to finding any gain when using dropout. With a poor setting of the dropout probability p preliminary experiments found no gain and often worse results from training with dropout. Realignment after five epochs is beneficial compared to the baseline DNN system, but slightly worse than a system which realigns after two epochs of training. Early realignment leads to better WER performance than all models we evaluated trained with dropout and early stopping. This makes early realignment the overall best regularization technique we evaluated on the Switchboard corpus. We note that only early realignment outperforms dropout regularization – a DNN trained with realignment after five epochs performs comparably to a DNN of the same size trained with dropout.