Sequence Training of DNN Acoustic Models With Natural Gradient
Table 1: Sequence training and dev.sub MPE phone accuracy/WER for the 50hr training set. The WERs shown at the last column were computed using the weak pruned biased LM used in MPE training.
['method', '#epochs', '#updates', 'phone acc. train', 'phone acc. dev.sub', 'WER dev.sub']
On the 50hr setup, we investigated the efficacy of the proposed optimisation framework and compared to both SGD and HF. The NG approach replaces the ‘Hessian’ component of the HF optimiser with the empirical FI (FI) matrix and this leads to both faster and improved convergence which makes the proposed optimisation framework more effective where only a relatively few updates are performed with large batch sizes. In comparison to SGD baseline, both HF and NG training were observed to be highly stable and did not require any form of additional update clipping. This is attributed to the fact that at each iteration, the direction explored is a modified gradient estimate that has been appropriately rescaled using the local KL curvature information.