Empirical Evaluation of Parallel Training Algorithms on Acoustic Modeling
Table 5: WER and training speedup of DNNs trained by BMUF on 4 GPUs with various minibatch sizes.
['Minibatch Size', 'Sync Period', 'Training Speedup', 'WER(%) test-clean', 'WER(%) test-other']
With single-GPU training, the training speed is lower when smaller minibatch size is used. This is because with small minibatch size the GPU power is not fully utilized and the model is updated more frequently.