Sequence Prediction with Neural Segmental Models
Table 7.2: Phoneme error rates (%) for end-to-end training from random initialization comparing CTC and segmental models trained with marginal log loss.
['[EMPTY]', 'regular LSMT dev', 'regular LSMT test', 'pyramid LSTM dev', 'pyramid LSTM test']
We use 3-layer 256-unit biderectional LSTMs paired with CTC, one-state HMM, and two-state HMMs. We do not use transition probabilities for two-state HMMs. We optimize marginal log loss with the corresponding weight functions and the corresponding constraint FSTs for CTC, one-state HMM, and two-state HMMs. We run vanilla SGD with step size 0.1 and gradient clipping of norm 5 for 20 epochs. Starting from the best performing model of the first 20 epochs, we run vanilla SGD for another 20 epochs with step size 0.75 decayed by 0.75 after each epoch. The dropout rate is 0.2, and no other explicit regularizer is used except early stopping on the development set. The same experiments are repeated with pyramid LSTMs.