SpecAugment: A Simple Data Augmentation Methodfor Automatic Speech Recognition
Table 2: LibriSpeech test WER (%) evaluated for varying networks, schedules and policies. First row from [25].
['[BOLD] Network', '[BOLD] Sch', '[BOLD] Pol', '[BOLD] No LM  [BOLD] clean', '[BOLD] No LM  [BOLD] other', '[BOLD] With LM  [BOLD] clean', '[BOLD] With LM  [BOLD] other']
The learning rate schedule turns out to be an important factor in determining the performance of ASR networks, especially so when augmentation is present. Here, we introduce training schedules that serve two purposes. Second, based on this, we introduce very long schedules that are used to maximize the performance of the networks. The three networks LAS-4-1024, LAS-6-1024 and LAS-6-1280 are trained on LibriSpeech 960h with a combination of augmentation policies (None, LB, LD) and training schedules (B/D). Label smoothing was not applied in these experiments. The experiments were run with peak learning rate of 0.001 and batch size of 512, on 32 Google Cloud TPU chips for 7 days. Other than the augmentation policies and learning rate schedules, all other hyperparameters were fixed, and no additional tuning was applied. We see that augmentation consistently improves the performance of the trained network, and that the benefit of a larger network and a longer learning rate schedule is more apparent with harsher augmentation.