Single headed attention based sequence-to-sequence model for state-of-the-art results on Switchboard-300
Table 2: Ablation study on the final training recipe.
['[EMPTY]', '[EMPTY]', 'WER [%] swb', 'WER [%] chm', 'WER [%] total', '#err. [word]']
In a second set of experiments, we characterize the importance of each of the regularization methods described in Sec. In these experiments, decoding is performed without an external language model. Curriculum learning is evaluated by either switching to randomized batches after 35 epochs or leaving the sorting on throughout training. SpecAugment is by far the most important method, while using Δ and ΔΔ features or switching off the curriculum learning in the later stage of training have marginal but positive effects. Other direct input level perturbation steps (speed/tempo perturbation and sequence noise injection) are also key techniques that can be found in the upper half of the table. If we compare the worst and baseline models, we find that the relative performance difference between them is nearly unchanged by including the external LM in decoding. Without the LM, the gap is 18% relative, while with the LM the gap is 17% relative. This clearly underlines the importance of the regularization techniques.