Very Deep Self-Attention Networks for End-to-End Speech Recognition
Table 3: The Transformer results on the TED-LIUM test set using TED-LIUM 3 training set.
['[BOLD] Models', 'Test WER']
With a similar configuration to the SWB models, we outperformed a strong baseline which uses both an external language model trained on larger data than the available transcription and speed perturbation, using our model with 36 encoder layers and 12 decoder layers. This result continues the trend that these models benefit from a deeper encoder, and together with the stochastic residual connections we further improved WER by 21.8% relatively, from 14.2 to 11.1%.