Deep Neural Machine Translation with Linear Associative Unit
Table 4: BLEU scores of DeepLAU and DeepGRU with different model sizes.
['SYSTEM', '( [ITALIC] Lenc, [ITALIC] LDec)', 'width', 'AVE.']
By comparing row 3 to row 7, we see that when LEnc and LDec are set to 2, the average BLEU scores achieved by DeepGRU and DeepLAU are 34.68 and 37.65, respectively. LAU can bring an improvement of 2.97 in terms of BLEU. After increasing the model depth to 4 (row 4 and row 6), the improvement is enlarged to 4.91. When DeepGRU is trained with larger depth (say, 4), the training becomes more difficult and the performance falls behind its shallow partner. While for DeepLAU, as can be see in row 9, with increasing the depth even to LEnc=8 and LDec=6 we can still obtain growth by 0.04 BLEU score. Compared to previous short-cut connection methods (row 5 and row 6), The LAU still achieve meaningful improvements over F-F connections and Residual connections by +1.35 and +2.57 BLEU points respectively.