A Transformer with Interleaved Self-attention and Convolution for Hybrid Acoustic Models
Table 4: Results of the transformer model trained with or without the time restriction.
['Model', 'window', 'dev clean', 'dev other', 'test clean', 'test other']
Follow the convention, we denote [−l,r] as the attention time window, where l and r are the sizes of left and right context. When l is ∞, it means that we perform self-attention up to the start of the acoustic sequence, while r=∞ means the attention operation spans to the end of the sequence. Note that, each 1D convolution layer looks ahead 1 frame since the kernel size is 3 without stride (e.g., [−1,1]). If the time window for each self-attention layer is [−∞,2], then the total accumulated time window for 6 self-attention layers would be [−∞,12].