Enhancing Monotonic Multihead Attention for Streaming ASR
Table 1: Results of offline MMA models on the dev sets with standard beam search decoding. HD: HeadDrop.
['ID', '[ITALIC] Dlm', '[ITALIC] Hma', '[ITALIC] Htotalma', 'HD', 'dev-clean / dev-other %WER', 'dev-clean / dev-other  [ITALIC] Rcov', 'dev-clean / dev-other  [ITALIC] Rstr']
HeadDrop and pruning MA heads in lower layers improve WER and streamability: Boundary coverage and stremability were averaged over two sets. A naïve implementation A1 showed a very poor performance. By pruning MA heads in lower layers with increasing Dlm, WER was significantly reduced, but the boundary coverage was not so high (A5). The proposed HeadDrop also significantly improved WER, and with the increase of Dlm, the boundary coverage was drastically improved to almost 100%. We can conclude that MA heads in lower layers are not necessarily important. This is probably because (1) the modalities between input and output sequences are different in the ASR task and (2) hidden states in lower decoder layers tend to represent the previous tokens and play a role of LM, thus are not suitable for alignment.