Weak-Attention Suppression For Transformer Based Speech Recognition
Table 1: WER comparison on the LibriSpeech benchmark. For models with citations, we directly use the results from the referred paper. ‘-WAS’ means the weak-attention suppression with γ=0.5 is applied. Note results in this table are obtained without the NNLM rescoring.
['Model Arch', '#Params (M)', 'test-clean', 'test-other']
Overall, the transformer-based models show improvement over recurrent neural network counterparts. For small transformer based model ‘vggTrf-12‘, applying WAS brings a relative WER reduction of 5.8% on test-clean and 5.7% on test-other. On a large model with 24 transformer layers, WAS generates a relative WER reduction of 6.0% on test-clean and 5.7% on test-other. ‘AmTrf-24 ’ is a streaming transformer-based model using augmented memory that limited the attention within an audio segment with 320ms lookahead. The WAS achieves relative WER reduction 10.0% on test-clean and 5.2% on test-other over the streaming baseline. In this section, we analyze the effect of WAS in one individual utterance and the whole dev-clean split in LibriSpeech using the model ‘vggTrf-12-WAS‘ in Table