Simplified Self-Attention for Transformer-based End-to-End Speech Recognition
Table 1: Results of different model architectures on AISHELL-1 test sets. SAN: self-attention network; SSAN: simplified self-attention network; #L: the number of layers; M: Million.
['Encoder (#L)', 'Decoder (#L)', 'Param. (M)', 'CER (%)']
We first validate our approach on the publicly available AISHELL-1 dataset. In order to verify whether the key, query, and value in self-attention can be formed by a simple FSMN memory block, we run a series of experiments with different model architectures. This conclusion is consistent for different architectures with different number of layers. Specifically, the SSAN-based transformer with encoder of 10 layers and decoder of 3 layers obtains 6.7% relative improvement in CER and 21.7% relative reduction in model parameters compared to SAN-based transformer with the same layer number setup. In Fig. As the monotonic characteristics of speech, the energies are mainly concentrated along the diagonal. Clear diagonal means better attention alignment. The figures clearly demonstrate that SSAN can learn better attention alignment compared to SAN, especially for the decoder-attention and the cross-attention. We visualized 50 random-selected utterances and SSAN can achieve consistently better attention alignment.