Self-Teaching Networks
Table 3: Experimental results of the larger LSTM model with roughly 63 million parameters.
['System', 'WER Cortana', 'WER DMA-O', 'WER DMA-I']
We then increased the size of the model with the size of cell being 1800 and the dimension of the hidden vector being 600. The total number of model parameters becomes roughly 63 million, which is about twice as many as the model has in the previous experiments. This model takes around 2 weeks to converge when trained with 16 GPUs in parallel. The large model improved the baseline system, with WER reductions vary from 3.8% for the Cortana task to 4% - 8% for the DMA task. For self-teaching networks, compared to the results of the smaller model, we observed relatively larger improvements after increasing the model size with the relative WER reduction up to 3% – 4%, which indicates that the regularization term plays a larger role in the model training. Notably, the improvement for the Cortana task is comparable to that achieved by doubling the model size. In fact, measured by the amount of training data in our experiments, the model size is not considerably large. We may achieve an even larger gain by increasing the model size further. Finally, we also evaluated the impact of the entropy term in these experiments, and the observation is consistent with that in the previous experiments.