Highway Transformer: Self-Gating Enhanced Self-Attentive Networks
Table 6: Final results of 12-layer Transformer-XL (XL-L12) and augmented SDUs with different settings.
['[BOLD] model', '[BOLD] eval loss', '[BOLD] eval bpc', '[BOLD] test loss', '[BOLD] test bpc']
The experimental results illustrate that SDU components have contributed to expediting the convergence during training (see Fig. But supplementing gated units on each Transformer block could encounter the premature convergence phenomenon. It is observed from Fig. two layers promote the convergence process and further improve the bpc performance on the dev and test set. Interestingly, the performance does not follow a positive correlation with the increase of gated layer numbers. (see Fig. Therefore, the lower layers benefit more from our proposed gated units than higher layers, again illustrating that SDUs could enhance feature-wise information on shallow layers of deep-stacked Transformer components.