improving generalization of transformer for speech recognition with parallel schedule sampling and relative positional embedding
Table 1: Performance in CER [%] of proposed PSS methods for transformer. Pmin and K are teacher forcing rate and number of decoding pass respectively.
['ExpID', 'Model', '[ITALIC] Pmin', 'K', 'SU', 'LU']
The base transformer model contains 5 MHA-FFN blocks and 3 MHA-MHA-FFN blocks. The model dimension dm is 768 and the inner-layer dimension of FFN is 2,048. We employ h=16 attention heads. For each of these heads we use dk=dv=dm/h=48. A two layer feed forward network with 2048 and 768 nodes maps the raw 284-dimensional feature to 768 dimension. Compare with B0 and B1, it is interesting to find that transformer performs better than LAS on SU but degrades a lot on LU.