Small-footprint Deep Neural Networks with Highway Connections for Speech Recognition
Table 2: Results of highway networks with and without the transform and carry gate.
['System', '#Layer', 'Dim', 'Transform', 'Carry', 'WER']
We then evaluated the specific role of the transform and carry gate in the highway architectures. We observed that using only one of the two gates, the HDNN can still achieved lower WER compared to the plain DNN, but the best results were obtained when both of the gates were active, which indicates that the two gating functions are complementary to each other. We observed that it converged faster when both of the gates were turned on. With only the transform gate, the convergence rate was much slower. As discussed before, the carry gate can be viewed as a particular type of skip connection, and it was more important to speed up the convergence compared to the transform gate in our experiments.