Multi-branch Attentive TransformerThis work is conducted at Microsoft Research Asia.
Table 7: Results of En→De with larger models. For WMT’14, both multi-bleu (left) and sacreBLEU (right) are reported.
['[ITALIC] Na/ [ITALIC] d/ [ITALIC] dh', '#Param (M)', '[ITALIC] ρ', 'WMT14', 'WMT18', 'WMT19']
Compared with the standard big transformer model of architecture 1/1024/4096, in terms of sacreBLEU, MAT with ρ=0.2 can improve the baseline by 1.0, 1.0 and 1.1 points on WMT14, WMT18 and WMT19 test sets respectively. Specially, on WMT’14 En→De, in terms of multi-bleu, we achieve 30.8 BLEU score, setting a new record on this work under the supervised setting.