Does Multi-Encoder Help? A Case Study on Context-Aware Neural Machine Translation
Table 5: Comparison of Outside Context and Gaussian-noise methods on three tasks, with dropout=0.3, σ = 0.3.
['[BOLD] System', '[BOLD] Zh-En', '[BOLD] Fr-En', '[BOLD] En-De', '[BOLD] En-Ru']
Inspired by Outside integration manner, we designed a simple yet effective method to regularize the training process: A Gaussian noise is added to the encoder output instead of the embedding Cheng et al. We sample a vector ϵ∼N(0,σ2I) from a Gaussian distribution with variance σ2, where σ is a hyper-parameter. Moreover, a natural question is whether further improvement can be achieved by combining the Context with the Gaussian-noise method. The observation here convinced the assumption again that the context-encoder plays a similar role with the noise generator.