Sequence Prediction with Neural Segmental Models
Table 5.2: Phoneme error rates (%) comparing different losses with two-stage training (2s) followed by end-to-end fine-tuning (ft).
['weight function', 'loss', '2s dev', '2s test', '+fine-tuning dev', '+fine-tuning test']
Next, we compare segmental models trained with different losses, including hinge loss, log loss, and marginal log loss. We use the FC weight function and a maximum segment duration of 30 frames. All losses are minimized with RMSProp for 20 epochs with step size 10−4 and decay 0.9. No explicit regularizers are used except early stopping on the development set. All losses achieve similar results with log loss slightly behind. Note that although marginal log loss does not require manual alignments, the frame-wise cross entropy used to train the frame classifier still does. After two-stage training, we use the trained segmental models and frame classifiers as an initialization for end-to-end training. Each loss is optimized with vanilla SGD for 20 epochs with step size 0.1 decayed by 0.75 after each epoch. Gradients are clipped with norm 5. Dropout is added to the LSTMs with rate 0.2, and no other regularizers are used except early stopping on the development set. End-to-end fine-tuning is able to improve the error rates for all losses with marginal log loss being the best performer.