Improving the Performance of Online Neural Transducer Models
Table 4: WER for NT with MHA
['System', 'Chunk', 'Single Attention - WER', 'MHA - WER']
The MHA model uses a 5x1400 encoder with 4 attention heads, and a 2x1024 decoder. One hypothesis is that multi-head attention computes attention from multiple points in the encoder space that come after the current prediction, which are ignored by streaming models such at NT.