Analyzing Word Translation of Transformer Layers
Table 4: Effects of Encoder / Decoder Depth.
['Depth Encoder', 'Depth Decoder', 'BLEU', 'Time Train', 'Time Decode (/s)', 'Time Speed up']
The decoding speed is measured on the test set which contains 3,003 sentences with a beam size of 4. “Speed up” stands for the decoding acceleration compared to the 6-layer Transformer. Specifically, the Transformer with 10 encoder layers and 2 decoder layers is 2.32 times as fast as the 6-layer Transformer while achieving a slightly higher BLEU.