Learning Contextualized Sentence Representations for Document-Level Neural Machine Translation
Table 8: BLEU scores for the ablation study of the fine-tuning model on the two corpora. “Pre+Next+Input Embedding” is the method that inputs the sum of the outputs from the pre- and next-encoder and input embeddings into the fine-tuning model. “Input Embedding” only uses input embeddings of the pre-training model for the input to the fine-tuning model.
['[BOLD] Methods', '[BOLD] 900k', '[BOLD] 2.8M']
Here we empirically investigate the impact of these three parts on the fine-tuning model via ablation study. and +0.66 BLEU points over baseline on the two corpora. When we add the outputs from both the pre-encoder and next-encoder, we obtain further improvements of +0.5 and +0.62 BLEU points over the input word embeddings. This clearly suggests that the pretrained two encoders learn additional contextual information.