MultiSpeech: Multi-Speaker Text to Speech with Transformer
Table 3: The comparison of similarity and diagonal attention rate r between LN, LW and Baseline settings.
['[BOLD] Setting', '[ITALIC] LN', '[ITALIC] LW', '[ITALIC] Baseline']
Comparison between layer normalization and learnable weight. We calculate the similarity between p and three settings: 1) LN(x)+p, our proposed layer normalization (LN); 2) x+αp, the learnable weight (LW) used in [li2019neural]; 3) x+p, the naive Transformer baseline (Baseline). Our proposed LN achieves the highest r while LW the lowest, which demonstrates that too strong position dominates phoneme embedding and harms the attention alignment.