Deja-vu: Double Feature Presentation and Iterated Loss in Deep Transformer Networks
Table 1: Librispeech CTC experimental results without any data augmentation technique and decoded with FST based on 4-gram LM. (Notes: Trf is Transfromers, “+ Iter. Loss 12-24” means adding iterative losses in the 12-th and 24-th layer, “+ Feat. Cat. 12-24” means adding feature concatenation in the 12-th layer.)
['[BOLD] Model', '[BOLD] Config', '[BOLD] dev  [BOLD] clean', '[BOLD] dev  [BOLD] other', '[BOLD] test  [BOLD] clean', '[BOLD] test  [BOLD] other']
Our baseline is a 24 layer Transformer network trained with CTC. For the proposed method, we varied the number and placement of iterated loss and the feature re-presentation. The next three results show the effect of using CTC multiple times. We see 12 and 8% relative improvements for test-clean and test-other. Adding feature re-presentation gives a further boost, with net 20 and 18% relative improvements over the baseline.