Highway Long Short-Term Memory RNNs for Distant Speech Recognition
Table 3: Comparison of shallow and deep networks
['System', '#layers', 'with overlap', 'no overlap']
When a network goes deeper, the training usually becomes difficult. From the table we can see that for a normal LSTMP network, when it goes from 3 layers to 8 layers, the recognition performance degrades dramatically. For the highway network, however, the WER only increase a little bit. The table suggests that the highway connection between LSTM layers allows the network to go much deeper than the normal LSTM networks.