END-TO-END NEURAL SEGMENTAL MODELS FOR SPEECH RECOGNITION
TABLE IV: PERs (%) for segmental models trained end to end with multiple tasks, i.e., the marginal log loss plus either the frame-wise cross entropy or the CTC loss.
['[ITALIC] λ', 'frame loss dev', 'frame loss test', 'CTC dev', 'CTC test']
Instead of optimizing different losses in different stages as in the previous section, we next optimize multiple losses jointly from random initialization. Here we only focus on marginal log loss paired with either the frame-wise cross entropy or the CTC loss, because the marginal log loss is the best performer in the previous experiments. We use early stopping based on the PERs of the segmental model on the development set. We see that end-to-end training with multiple tasks further improves over end-to-end training with a single task. The best test-set result (and the best dev-set result) is obtained by multitask training with marginal log loss + CTC loss, and improves over the CTC error rate by 1% absolute (19.5% ⟶ 18.5% on the test set). We further investigate the loss values for the case of jointly optimizing the marginal log loss and the CTC loss. In the multitask case, both the marginal log loss and the CTC loss achieve lower values on the training set compared to the single-task case, suggesting that multitask learning might help optimization. However, both loss values on the development set end up higher when multiple losses are used. The fact that models with higher losses on the development set end up having lower PERs is unsatisfying and needs further investigation.