Exploiting Sentential Context for Neural Machine Translation
Table 2: Case-sensitive BLEU scores on WMT14 En⇒De and En⇒Fr test sets. “↑/⇑”: significant over Transformer counterpart (p<0.05/0.01), tested by bootstrap resampling.
['[BOLD] Model', '[BOLD] En⇒De', '[BOLD] En⇒Fr']
As seen, exploiting deep sentential context representation consistently improves translation performance across language pairs and model architectures, demonstrating the necessity and effectiveness of modeling sentential context for NMT. Among them, Transformer-Base with deep sentential context achieves comparable performance with the vanilla Transformer-Big, with only less than half of the parameters (114.4M vs. 264.1M, not shown in the table). Furthermore, Deep (Tam) consistently outperforms Deep (RNN) in the Transformer-Big configuration. One possible reason is that the big models benefit more from the improved gradient flow with the transparent attention Bapna et al.