Hand-crafted Attention is All You Need? A Study of Attention on Self-supervised Audio Transformer
Table 2: Performance of all attentions
['[BOLD] Attention', '[BOLD] Speaker  [BOLD] Utterance', '[BOLD] Speaker  [BOLD] Frame', '[BOLD] Phoneme  [BOLD] 1-hidden', '[BOLD] Phoneme  [BOLD] 2-hidden']
Baseline (QK) and Baseline (Q) (shared-QK attention) remarkably outperform Baseline (Mel), which shows the importance of pre-training. LSH /ALSH algorithms have negative influences on most downstream tasks, showing that restricting the attention by LSH/ALSH algorithm is not effective enough. For utterance-level speaker classification, the average pooling layer in the downstream model acts like a global attention mechanism, which compensates the effects of LSH/ALSH.