Transformer with Depth-Wise LSTM
Table 5: Results of Deep Transformers. Std and Ours mean the standard Transformer with residual connection and our approach with depth-wise LSTM respectively.
['Layers', 'En-De Std', 'En-De Ours', 'Cs-En Std', 'Cs-En Ours']
To examine whether the depth-wise LSTM has the ability to ensure the convergence of deep Transformers and how it performs with deep Transformers. We conduct experiments on the WMT 14 English to German task and the WMT 15 Czech to English task following \newcitebapna2018training,xu2020lipschitz, and compare our approach with the Transformer in which residual connections are not normalized by layer normalization.