Small-footprint Deep Neural Networks with Highway Connections for Speech Recognition
Table 4: Comparison to residual networks (ResNets).
['System', '#Layer', 'Dim', 'Activation', 'WER']
Our experiments showed that without the two gating functions, training the residual networks was comparably more challenging. For instance, with 10 hidden layers and using sigmoid activations, residual networks achieved higher WER compared to highway networks. However, the differences in terms of the accuracy were smaller when using ReLU (rectified linear unit) activations for residual networks, because training ReLU networks are relatively less difficult. Furthermore, we experienced difficulty to train residual networks with 15 hidden layers using sigmoid activations instead of ReLU (The CE cost did not come down after over 20 epochs), although with ReLU activations, residual networks slightly outperformed highway networks in this case. Note that, residual networks still performed better compared to the plain networks with RBM pre-training, e.g., when the depth was 10. From our experiments, we may draw the conclusion that residual networks are more powerful to train deeper networks compared to plain DNNs, particular with ReLU activation functions which reduce the optimisation difficulty. However, highway networks are more flexible with the activation functions due to the two gating functions that control the follow of information.