Transformers with convolutional context for ASR
Table 3: WER for different transformer depth in encoder and decoder
['Setup', 'Enc/Dec depth', 'dev clean', 'dev other', 'test clean', 'test other']
Using a deep transformer encoder capture long range structure of the data as an acoustic LM built on top of learned concepts from the convolutional layers. Also deeper encoder help marginalize out global utterance specific speaker and environment characteristics while focusing on the content. A deeper deocder, although not as critical, showed better overall performance. We wanted to understand the effect of fixing encoder depth while changing decoder and vice versa, fixing the total sum of encoder and decoder depths, as well as using same depth on both sides all the way up to 14 transformer layers.