Multitask Learning with Low-Level Auxiliary Tasks for Encoder-Decoder Based Speech Recognition
Table 2: WER (%) on Eval2000 for different end-to-end models. PhoneDec-n refers to a phoneme decoder applied at layer n of the encoder. Similarly, PhoneCTC-3 means phoneme CTC loss applied at layer 3 and State-2 means state-label supervision applied at layer 2 of the encoder.
['[BOLD] Model', '[BOLD] SWB', '[BOLD] CHE', '[BOLD] Full']
In order to determine whether the improved performance is a basic multitask training effect or is specific to the low-level application of the loss, we compare these results to those of adding the phoneme decoder at the topmost layer (Enc-Dec + PhoneDec-4). The top-layer application of the phoneme loss produces worse performance than having the supervision at the lower (third) layer. Finally, we obtain the best results by adding both phoneme decoder supervision at the third layer and frame-level state supervision at the second layer (Enc-Dec + PhoneDec-3 + State-2). The results support the hypothesis that lower-level supervision is best provided at lower layers. For comparison, we also include a variety of other recent results with neural end-to-end approaches on this task. With the addition of the low-level auxiliary task training, our models are competitive with all of the previous end-to-end systems that do not use a language model.