Very Deep Self-Attention Networks for End-to-End Speech Recognition
Table 1: The performance of deep self-attention networks with and without stochastic layers on Hub5’00 test set with 300h SWB training set.
['[BOLD] Layers', '[BOLD] #Param', '[BOLD] SWB', '[BOLD] CH']
A shallow configuration (i.e 4 layers) is not sufficient for the task, and the WER reduces from 20.8% to 12.1% on the SWB test as we increase the depth from 4 to 24. The improvement is less significant between 12 and 24 (only 5% relative WER), which seems to be a symptom of overfitting.