Multiscale Collaborative Deep Models for Neural Machine Translation
Table 5: Ablation study on IWSLT14 En→De task.
['[BOLD] Model', '[BOLD] BLEU']
Ablation Study. We conduct ablation study to investigate the performance of each component of our model. This method achieves a 29.45 BLEU, which is lower than the best result. We additionally modify the implementation of the contextual collaboration cell Q(⋅) as \textscFfn(⋅), which shows that the performance is reduced by 0.5 BLEU. (2) Removing Cxt-Enc Attention and/or contextual collaboration makes the BLEU score drop by ∼0.7, which suggests that multiscale collaboration helps in constructing extremely deep models. (3) Considering that the deep Msc introduces more parameters, we also train another two Msc models with about the same or double number of parameters: with 18/36 layers, embedding size 512 and Ffn layer dimension 1024. These models underperform the deeper 72-layer model, which shows that the number of parameters is not the key to the improvement.