Does Multi-Encoder Help? A Case Study on Context-Aware Neural Machine Translation
Table 3: The BLEU scores [%] of different context-aware models with three context inputs. We use dropout=0.1 and dropout=0.3 respectively.
['[BOLD] System', '[BOLD] System', '[BOLD] Zh-En  [ITALIC] p=0.1', '[BOLD] Zh-En  [ITALIC] p=0.3', '[BOLD] Fr-En  [ITALIC] p=0.1', '[BOLD] Fr-En  [ITALIC] p=0.3', '[BOLD] En-De  [ITALIC] p=0.1', '[BOLD] En-De  [ITALIC] p=0.3', '[BOLD] En-Ru  [ITALIC] p=0.1', '[BOLD] En-Ru  [ITALIC] p=0.3']
We see, first of all, that all multi-encoder models, including both Inside and Outside approaches outperform the sentence-level baselines by a large margin on the Zh-En and En-De datasets with a small p value of dropout. Also, there are modest BLEU improvements on the Fr-En and En-Ru tasks. When the models are regularized by a larger dropout, all systems obtain substantial improvements - but the gaps between sentence-level and multi-encoder systems decrease significantly. We deduce that if the context-aware systems rely on the contextual information from the preceding sentence, the performance of Random and Fixed should dramatically decrease due to the incorrect context. A possible explanation is that the context encoder does not only model the context. Instead, it acts more like a noise generator to provide additional supervised signals to train the sentence-level model.