Small-footprint Deep Neural Networks with Highway Connections for Speech Recognition
Table 1: Comparison of depth and width between plain DNNs and HDNNs. ∗indicates that the models were trained using the Kaldi toolkit, where the networks were initialised with restricted Boltzmann machine (RBM) based pre-training because random initialisation did not yield convergence.
['System', '#Layer', 'Dim', '#Parm (M)', 'WER']
As the number of hidden units decreases, the accuracy of plain DNNs degrade rapidly, and the accuracy loss cannot be compensated by increasing the depth of the network. We faced the difficulty to train thin and deep networks directly without RBM pre-training (the CE loss did not decrease at all after many epochs). However, with highway connections we did not have this difficulty. With highway connections, we can cut down the number of model parameters by around 80% with marginal accuracy loss, and with less than 1 million model parameters, the CE trained HDNN can achieve comparable or slight higher accuracy compared to a strong GMM baseline with speaker adaptive training (SAT) and bMMI-based discriminative training. The accuracy of smaller-size HDNN models may be further improved by the teacher-student style training, which will be investigated in the future.