Deep Neural Machine Translation with Weakly-Recurrent Units
Table 4: Ablation experiments on SR-NMT with 4 layers. BLEU scores are computed after one training stage. While removing multi attention we still keep one attention model in the last layer. The system without layer normalization and highway connections failed to converge.
['[BOLD] Model', '[BOLD] BLEU', 'Δ']
Another important observation is the additivity of the contributions from all the components, in fact when two or three components are removed at once, the drop in performance is roughly the sum of the drops caused by the single components. Finally, the removal of layer normalization and highway connections, while keeping multi attention, causes a gradient explosion that prevents the SR-NMT system from converging.