Highway Long Short-Term Memory RNNs for Distant Speech Recognition
Table 2: Performance of highway (B)LSTMP RNNs with dropout
['System', '#Layers', 'with overlap', 'no overlap']
Dropout is applied to the highway connection to control its flow: a high dropout rate essentially turns off the highway connection, and a small dropout rate, on the other hand, keeps the connection alive. In our experiments, for early training stages, we use a small dropout rate of 0.1, and increase it to 0.8 after 5 epochs of training.