Accelerating Hessian-Free Optimization for Deep Neural Networks by Implicit Preconditioning and Sampling
Table 3: Overall Training Time Improvements, Switchboard
['Method', 'WER', 'Total Accessed Data Points']
Since using 32 L-BFGS stats performed well for the smaller 50-hour BN task, we used the same on the Switchboard task for preconditioning. In addition, because of the increased amount of training data associated with the larger task, we found that using a smaller sample size (i.e., α) for the gradient and CG iteration calculations still allowed for an appropriate estimate of these statistics. Since we use more parallel machines (i.e. 64) for SWB compared to BN, it was not possible to exclusively reserve machines for timing calculations. Therefore, training time is estimated by calculating total number of accessed data points for training, which is correlated to timing. Notice that with a larger dataset, because we are able to decrease the fraction of data used for gradient and conjugate gradient calculations, was can achieve an even larger speedup of 2.3x over the baseline, with no loss in accuracy. This suggests that even further speedups are possible as the data size grows.