Does Multi-Encoder Help? A Case Study on Context-Aware Neural Machine Translation
Table 2: Comparison of context-aware model with two training strategies on En-De task. WS represents weight-sharing and TS represents two-stage training.
['[BOLD] System', '[BOLD] Layers', '[BOLD] WS', '[BOLD] TS', '[BOLD] BLEU']
Weight sharing Voita et al. two-stage training Zhang et al. strategies have been proven essential to build strong context-aware systems. The former shared the first N-1 blocks of context encoder with the source encoder, and the latter first trained a standard sentence-level Transformer and finetuned the document-level Transformer with an extra context-encoder. We first evaluated the importance of two training strategies for multi-encoder systems. The model with two-stage training performs slightly better than the weight-sharing system in terms of BLEU. To our surprise, the context-encoder with a single-layer can compete with a six-layers model. We suspect that this is because the training data is limited and we do not need a sophisticated model to fit it. Therefore, we choose the two-stage training and single-layer context-encoder for all experiments in the remainder of this paper.