improving generalization of transformer for speech recognition with parallel schedule sampling and relative positional embedding
Table 2: Comparison in CER [%] between RPE and APE methods for transformer. k is the RPE range and 10-4 represents k=10 and k=2 for encoder and decoder respectively.
['ExpID', 'APE', 'RPE', '[ITALIC] k', 'SU', 'LU']
Without PE, although LU improves, SU drops a not. In E6, we replace the fixed APE with a learnable token ID APE, just like the token embedding in the decoder. Token ID embedding is worse on LU than sin/cos APE but similar on SU. Then we try to introduce RPE in encoder (E7) and decoder (E8). When RPE is added to MHA layer in encoder, LU improves from 42.41 to 33.56. It further improves to 29.87 without APE and set k=10, which represents to attend to about 2 Chinese character on both sides. The CER continues to decrease to 12.73 as we utilize a 2-character-range RPE to the decoder MHA. With RPE, even long sentence presents, the local attention relationship has already been learned. Besides, the possibility of similar segments appearing in the near context becomes smaller which makes the model has little burden to distinguish among them. Thus RPE helps to decrease TD and ID. This also indicates local and relative position is more suitable for speech recognition. Overall, with relative position embedding, we lower CER of LU set from 42.41 to 12.73, an absolute 30% gains.