Improving Deep Transformerwith Depth-Scaled Initialization and Merged Attention
Table 1: Empirical measure of output variance Var(r) of RC and error signal change ratio β\textscLn, β\textscRc and β (Eq. 14) averaged over 12 layers. These values are estimated with ∼3k target tokens at the beginning of training using 12-layer Transformer. “Base”: the baseline Transformer. “Ours”: the Transformer with DS-Init. Enc and Dec stand for encoder and decoder respectively. Self, Cross and FFN indicate the self-attention, encoder-decoder attention and the feed-forward sublayer respectively.
['Method', 'Module', '[EMPTY]', 'Self', 'Cross', 'FFN']
We resort to empirical evidence to analyze these ratios. RC strengthens it (β\textscRc>1). One explanation about LN’s decay effect is the large output variance of RC (Var(r)>1) which negatively affects δr as shown in Eq. By contrast, the short-cut in RC ensures that the error signal at higher layer δr can always be safely carried on to lower layer no matter how complex ∂f∂z would be as in Eq. The gradient norm is preserved better through the self-attention layer than the encoder-decoder attention, which offers insights on the successful training of the deep Transformer in BERT Devlin et al. That is why BERT and GPT adopt a much smaller standard deviation (0.02) for initialization, in a similar spirit to our solution. According to the property of uniform distribution, the variance of model parameters decreases from γ23 to γ2α23l after applying DS-Init. By doing so, a higher layer would have smaller output variance of RC so that more gradients can flow back. This is because DS-Init endows lower layers with parameters of larger variance and activations of larger norm. When error signals at different layers are of similar scale, the gradient norm at lower layers would be larger. Nevertheless, this increase does not hurt model training based on our empirical observation. Optimizers with adaptive step rules, such as Adam, could have an adverse effect that enables gradient scale correction through the accumulated first and second moments. DS-Init makes the training more stable and robust.