Toward Making the Most of Context in Neural Machine Translation
Table 3: Ablation study on modeling context on TED Zh-En development set. ”Doc” means using a entire document as a sequence for input or output. BLEU{}_{\rm doc} indicates the document-level BLEU score calculated on the concatenation of all output sentences.
['[BOLD] Model', 'BLEU (BLEU{}_{\\rm doc})']
First of all, using the entire document as input and output directly cannot even generate document translation with the same number of sentences as source document, which is much worse than sentence-level baseline and our model in terms of document-level BLEU. For source context modeling, only casting the whole source document as an input sequence (Doc2Sent) does not work. Meanwhile, reset word positions and introducing segment embedding for each sentence alleviate this problem, which verifies one of our motivations that we should focus more on local sentences. Moreover, the gains by the segment-level relative attention and gated context fusion mechanism demonstrate retrieving and integrating source global context are useful for document translation. As for target context, employing Transformer-XL decoder to exploit target historically global context also leads to better performance on document translation. This is somewhat contrasted to [zhang2018improving] claiming that using target context leading to error propagation. In the end, by jointly modeling both source and target contexts, our final model can obtain the best performance.