Small-footprint Deep Neural Networks with Highway Connections for Speech Recognition
Table 3: Results of using constrained carry gate, where C(⋅)=1−T(⋅).
['System', '#Layer', 'Dim', 'Constrained', 'WER']
In this approach, the computational cost is reduced since the matrix-vector multiplication for the carry gate is not required. Contrary to our expectations, with the constrained carry gate only we obtained worse results when the networks were relatively wide, while the accuracy gap was reduced when the number of hidden units was smaller. The reason may be that in the constrained setting, the transform gate T(⋅) learns the scaling function for both the input and output at the same time. As regularisation is expected to be more important for training wide and deep networks, this may not be achieved by using a single gating function. For instance, both the input and output of one hidden layer may require larger or smaller scaling weights at the same time, which is impossible in the constrained setting. In the future, we shall look into the regularisation and generalisation properties of the two gating functions more closely.