Learning Contextualized Sentence Representations for Document-Level Neural Machine Translation
Table 4: Comparison between the joint training method and the explicit contextual integration method on Chinese-English translation.
['[BOLD] Methods', '[BOLD] AVG']
In our joint training method, during the testing phase, we do not explicitly use any contextual information for the current source sentence as the jointly trained NMT model implicitly learns the potential context information by learning to predict the preceding / succeeding source sentence during the training phase. In order to study how much potential context information can be captured by this implicit method, we conducted a comparison experiment. We obtain the representations of the preceding / succeeding source sentences via the pre-encoder and next-encoder and integrate them into the encoder of the current sentence by using self-attention that treats the encoder outputs of the current sentence as q and the encoder outputs of the previous / next sentence as k and v. We refer to this method as “Explicit Contextual Integration”. The explicit contextual integration method is better than our joint training method by only 0.26 BLEU points. Comparing this with the improvement of 0.84 BLEU points achieved by our joint training method over the baseline, we find that the joint training method is able to learn sufficient contextual information for translation in an implicit way.