Sequence Training of DNN Acoustic Models With Natural Gradient
Table 3: Performance achieved with different optimisers on the 200 hr setup.
['method', '#epochs', '#updates', 'phone acc. train', 'phone acc. dev.sub', 'WER dev.sub']
The experiments on the 200 hr training set were performed to ensure that the sequence training techniques generalise to a somewhat larger training set (and output layer size) and also to present more detailed comparisons of how training proceeds. In this case a total of 8 epochs were performed with SGD and 15 epochs (120 updates) with HF, DSAG-HF and NG. In DSAG-HF, the HF optimiser is equipped with a ‘momentum’ component which essentially contributes to initialising CG with a weighted sum of the current and previous gradient directions. In Fig. In contrast, replacing the GN with the empirical FI matrix leads to progressively better updates at each epoch. In Fig. WER are very similar over the first 8 epochs between NG and SGD, but NG only requires eight updates per epoch. Furthermore, through following a path where the MPE training criterion remains a good approximation to the WER, the NG method allows training to converge to a better solution than either SGD or any of the HF variants.