When Can Self-Attention Be Replaced by Feed Forward Layers?
Table 1: Character error rate (CER) on WSJ for the Transformer models with different encoders.
['Number of Layers Self-attention', 'Number of Layers Feed forward', 'CER eval92', 'CER dev93']
To demonstrate that the range of learned context is stacked up and that multi-head attention is redundant for the upper layers of the encoder, we train models whose encoders are built by different numbers of self-attention layers and feed forward layers. For the encoder of these models, there are 12 layers in total and the lower layers are self-attention layers while the upper layers are feed forward layers. We start from an encoder with 6 self-attention layers and 6 feed forward layers. Then, we increase the number of self-attention layers and decrease the number of feed forward layers. We tested if stacking more feedforward layers to make deeper encoders is beneficial. We also tried to modify the architecture of the stacked feed forward layers, such as removing residual connections or using an identity mapping [he2016identity]. However, these modifications were not helpful and we did not observe CER reduction compared to the 11-layer self-attention 1-layer feed forward encoder.