Exploring Transformers for Large-Scale Speech Recognition
Table 1: Results of the Transformers with convolution layers in the offline mode. The number of heads is 4, and the number of layers is 12. IC stands for interleaved 1D convolution. All models have around 50 million (M) model parameters.
['Model', 'IC', 'Size(M)', 'Encode layer', '[ITALIC] dk', 'dev']
The self-attention operation cannot maintain the monotonicity of input sequence, which is particularly harmful for time-synchronous acoustic model such as the hybrid model studies in this paper. The kernel size for the 1D convolution is 3, while the VGG net has 4 layers of 2D convolutions with 3x3 filters. When the VGG encoder was applied, we used features of frame rate as 10ms, and employed a max-pooling layer to down-sample the features by a factor of 2. Both interleaved convolution and VGG net can significant improve the accuracy of Transformer models. In addition, when applying a VGG as the encoder, it is more beneficial to remove the interleaved convolutions but increase the model dimension of self-attention layers if the model size is constrained to be the same.