Does Multi-Encoder Help? A Case Study on Context-Aware Neural Machine Translation
Table 4: The BLEU scores [%] of context-aware systems with two inference schemas. Aware represents the inference process matches the training. Agnostic represents that models ignore context encoder during inference.
['[BOLD] System', '[BOLD] Inside Aware', '[BOLD] Inside Agnostic', '[BOLD] Outside Aware', '[BOLD] Outside Agnostic']
To verify the assumption of robust training, we followed the work Srivastava et al. We turned off the context-encoder during the inference process, and made the inference system perform as the sentence-level baseline. This confirms that the information extracted by context-encoder just plays a role like introducing randomness into training (e.g., dropout), which is a popular method used in robust statistics. We argue that three types of context provide noise signals to disturb the distribution of the sentence-level encoder output. The BLEU improvements of both Outside and Inside are mainly due to the richer noise signals which can effectively alleviate the overfitting.