Empirical Evaluation of Parallel Training Algorithms on Acoustic Modeling
Table 4: WER and training speedup of DNNs trained by ASGD and BMUF on 4 GPUs with various synchronization periods.
['Parallel Algorithm', 'Sync Period', 'Training Speedup', 'WER(%) test-clean', 'WER(%) test-other']
The choice of synchronization period τ affects the behavior of BMUF and ASGD. This is because ASGD suffers from the problem of delayed gradient update and larger synchronization period results in greater latency. In contrast, we observed no performance degradation on BMUF when varying the synchronization period. As for training speedup, when synchronization period τ is small, the parameter exchange among multi-GPUs is quite frequent and the communication overhead is the main bottleneck of training speed. When the value of τ continues to increase, the communication overhead is reduced so that the computation speed becomes the main bottleneck.