Self-Teaching Networks
Table 2: Experimental results of the smaller LSTM model with around 31 million parameters. LS refers to label smoothing, and CP is short for confidence penalization. ST refers to the self-teaching network. H means the entropy term H(Pθ(y|hL)) in Eq (1).
['System', 'WER Cortana', 'WER DMA-O', 'WER DMA-I']
The number of the total model parameters is roughly 31 million. We searched the hyper-parameter λ∈{0.001,0.005,0.01,0.02}, and for clarity, we only show results with the optimal λ. In fact, we observed a consistent trend by tuning the hyper-parameter λ. Confidence penalization shows the similar trend. For self-teaching networks, however, we observed small but consistent improvements across the 4 evaluation sets. The new model is also fairly robust to the hyper-parameter in our experiments, as the results only fluctuated slightly when λ is within the range from 0.001 to 0.02. Since the difference between confidence penalty and self-teaching network is only the cross entropy term H(Pθ(y|hL),Pθ(y|hl)), we evaluated its impact by removing the entropy term H(Pθ(y|hL)) in the experiments of self-teaching networks. This may help explain the results of confidence penalization, which show that the entropy term alone cannot achieve consistent WER reductions.