Improving LSTM-CTC based ASR performance in domains with limited training data
Table 2: Forget gate bias initialization.
['System', '%WER ( [ITALIC] dev-clean)']
Typically, LSTM parameters are initialized to small random values in an interval, e.g., in our experiments we initialize weights and biases to random values in the interval [−0.1,0.1]. However, for the forget gate, this is a suboptimal choice, where small weights effectively close the gate, preventing cell memory and its gradients from flowing in time. One approach to address this issue is to initialize the forget gate bias, bf, to a large value, say 1, that will force the gate to be initialized in an open position and allow memory cell gradients in time to flow more readily. In our work, however, we found that simply increasing the forget gate bias did not provide the anticipated gains, and during training, accuracy in early epochs plateaued very quickly and triggered our stopping criterion, with worse results than if we had left the forget gate bias set to a random small initial value. However, if we disable the stopping criterion and force a minimum number of epochs, in this case 6 epochs, we did observe consistent gains with bf=1.