Transformers with convolutional context for ASR
Table 1: The impact of different modeling decision on WER
['Model', 'dev clean', 'dev other', 'test clean', 'test other']
We first studied performance of our approach to alternative architectures and positional encoding schemes. By combining sinusoidal and convolutional position embedding (rows 1+2), we don’t observe any gains. This supports our intuition that the relative convolutional positional information provides sufficient signal for the transformer layers to recreate more global word order. We also found that having multiple encoder-side attention layers is critical for achieving the best WER. Increasing the intermediate ReLU layer in each encoder and decoder layer was found to greatly improve the overall WER across different sets, however increasing the number of attention heads, while keeping the attention dimension the same, deteriorates the performance.