Deep Recurrent Neural Networks for Acoustic Modelling
Table 4: Effects of distributed optimization for our TC-DNN-BLSTM-DNN model. The ASGD experiments uses 3 independent SGD shards.
['[BOLD] Model', 'Epochs', 'Time (hrs)', 'dev93 WER', 'eval92 WER']
Our baseline TC-DNN-BLSTM-DNN SGD system took 17 epochs or 51 wall clock hours to converge to a dev and test WER of 6.58 and 3.47. Our distributed implementation converges in 14 epochs and 16.8 wall clock hours, achieves a dev and test WER of 6.57 and 3.72. The distributed optimization is able to match the dev WER, however the test WER is significantly worse. It is unclear whether this WER differential is due to the asynchronicity characteristic of the optimizer or due to the small datasets, we suspect with larger datasets the gap between the ASGD and SGD will shrink. The conclusion we draw is that ASGD can converge much quicker and faster, however there may be a impact to final WER performance.