When and Why is Document-level Context Usefulin Neural Machine Translation?
Tabelle 3: Comparison of document-level model architectures and complexity.
['Approach', 'Context Encoder Architecture', 'Context Encoder #layers', '[BOLD] en-it Bleu [%]', '[BOLD] en-it Ter [%]', '[BOLD] en-de Bleu [%]', '[BOLD] en-de Ter [%]']
Model Architecture The tested methods are equal or closest to: Single-Encoder: \newciteagrawal2018contextual Integration outside the decoder: \newcitevoita2018context without sharing the encoder hidden layers over current/context sentences Integration inside the decoder Sequential attention : Decoder integration of \newcitezhang2018improving with the order of attentions (current/context) switched Parallel attention: Gating version of \newcitebawden2018evaluating Context encoding without any sequential modeling (the last row) shows indeed comparable performance to using a full 6-layer encoder. This simplified encoding eases the memory-intensive document-level training by having 22% fewer model parameters, which allows us to adopt a larger batch size without accumulating gradients. For the remainder of this paper, we stick to using the multi-encoder approach with parallel attention components in the decoder and restricting the context encoding to only word embeddings.