Deja-vu: Double Feature Presentation and Iterated Loss in Deep Transformer Networks
Table 2: Librispeech experimental results. The baseline consists of VGG + 24 layers of Transformers trained with SpecAugment [18]. Trf is transformer. 4-gr LM is the official 4-gram word LM. S2S denotes sequence-to-sequence architecture.
['[BOLD] Model', '[BOLD] LM', '[BOLD] test-clean', '[BOLD] test-other']
We test both CTC and CE/hybrid systems. There are consistent gains first from iterated loss, and then from multiple feature presentation. We also run additional CTC experiments with 36 layers Transformer (total parameters 120 millions). The baseline with 36 layers has the same performance with 24 layers, but by adding the proposed methods, the 36 layer performance improved to give the best results. This shows that our proposed methods can improve even very deep models.