A Transformer with Interleaved Self-attention and Convolution for Hybrid Acoustic Models
Table 2: Results of the transformer model with or without positional encoding denoted as POS, and dropout regularization denoted as DP.
['Model', 'POS', 'DP', 'dev clean', 'dev other', 'test clean', 'test other']
Unlike the observations in the area of machine translation, positional encoding did not make a big difference in terms of recognition accuracies for our transformer models. One possible reason is that we have used 1D convolution, which has encoded some sequential information in to the model. During the model training, the information in the positional encoding may be ignored. We performed a sanity check by removing the positional encoding when evaluating a transformer model trained with positional encoding, and obtained results which are only around 0.1% worse absolute. In the future, we shall investigate if it would make a difference after scaling the positional encoding features to the same dynamic range as the acoustic feature after the linear projection layer.