A Transformer with Interleaved Self-attention and Convolution for Hybrid Acoustic Models
Table 3: Results of the transformer model with or without 1D convolution. Norm refers to the layer normalization before output linear layer.
['Model', 'Conv', 'Norm', 'dev clean', 'dev other', 'test clean', 'test other']
We then evaluated the impact of the 1D convolution layers in our transformer model by removing all the convolution layers. We still added the positional encoding feature to the inputs since the sequential information from the convolution layers is no longer available. We also have to insert another layer normalization to the output of the transformer before the output linear layer to stabilize the training. Otherwise, the training diverges quickly after one or two epochs in our experiments. Besides, without the convolution layers, the convergence of the model during training also became much slower, which demonstrates that the convolution layers are helpful for both convergence in model training and the recognition accuracy. For a fair comparison, we also added another layer normalization to the model with convolutions. While it can further speed up the convergence, it does not improve the recognition accuracy further. In the future, we shall compare the interleaved combination of self-attention and convolution to the sequential combination of self-attention and TDNN