Multi-branch Attentive TransformerThis work is conducted at Microsoft Research Asia.
Table 4: Results on WMT’14 En→De translation.
['[ITALIC] Na/ [ITALIC] d/ [ITALIC] dh', '#Param (M)', '[ITALIC] ρ', 'BLEU']
The standard baseline is 29.13 BLEU score, and the model contains 209.8M parameters. We implement an MAT with Na=2,dh=12288 and another with Na=3,dh=10240. We can see that our MAT also works for the large-scale dataset. When Na=2, we can obtain 29.90 BLEU score. When increasing Na to 3, we can get 29.85, which is comparable with the the Na=2 variant. Besides, for large-scale datasets, the drop branch technique is important too. ρ=0.2 works best of all settings.