Very Deep Self-Attention Networks for End-to-End Speech Recognition
Table 2: Comparing our best model to other hybrid and end-to-end systems reporting on Hub5’00 test set with 300h SWB training set.
['[BOLD] Hybrid/End-to-End Models', '[BOLD] Tgt Unit', '[BOLD] SWB', '[BOLD] CH']
Third, it was revealed that the combination of our regularization techniques (dropout, label-smoothing and stochastic networks) are additive with data augmentation, which further improved our result to 18.1% with the 36−12 setup. Comparing to the best hybrid models with similar data constraints, our models outperformed on the CH test set while remaining competitive on the SWB test set without any additional language model training data. This result suggests the strong generalizability of the Stochastic Transformer.