Exploring Transformers for Large-Scale Speech Recognition
Table 2: Results of the 12-layer Transformer model with different number of attention heads. VGG net was used as the encoding layer for all the Transformers. N denotes the number of attention heads, and dk is the model dimension as in Eq(1).
['Model', 'IC', 'Size (M)', '[ITALIC] N', '[ITALIC] dk', 'dev']
With the interleaved convolution, the Transformer with 16 attention heads achieved the lowest WER, while for the vanilla Transformer, 8 attention heads are sufficient. We did not further increase the number of the attention heads in our experiments due to the memory constraint. Compared to the BLSTM with around 55 million model parameters, the Transformer can achieve around 6% relative WER reduction with a similar model size.