When Can Self-Attention Be Replaced by Feed Forward Layers?
Table 2: Word error rate (WER) of the experiments on SWBD. The evaluation sets are eval 2000 SWBD/callhome
['Number of Layers Self-attention', 'Number of Layers Feed forward', 'WER SWBD', 'WER Callhome']
We further test our argument on a larger and more complicated dataset, SWBD. The encoder with 10 self-attention layers have worse results than the encoders with 11 and 12 self-attention layers. Also, the 12-layer self-attention encoder has higher word error rates (WER) than the 11-layer encoder. However, the encoder with 10 self-attention layers and 2 feed forward layers, which has 12 layers in total, gives the lowest WERs. The 9 self-attention layers + 3 feed forward layers encoder yields worse WERs. Thus, the 10th self-attention layer is crucial in learning contextual information. Upon the 10th self-attention layer feed forward layers are sufficient in learning further abstract representations.