Non-Parametric Adaptation for Neural Machine Translation
Table 6: Comparison of test translation quality (BLEU) with different memory architectures. All models are trained on the Dense N-Gram Multi-Domain dataset. CSTM corresponds to the proposed Conditional Source Target Memory. CTM corresponds to Conditional Target Memory, where we ignore the retrieved sources while encoding the retrieved targets, and directly attend the encoding of the current source, X. TM corresponds to encoding the retrieved targets without any context.
['Model', 'newstest 14', 'IWSLT 2015', 'OpenSub', 'JRC-Acquis']
We first remove the retrieved sources, Xi, from the CSTM, resulting in an architecture where the encoding of a retrieved target, Yi, only incorporates information from the source X, represented by the row CTM in the table. This results in a clear drop in performance on all datasets. We ablate further by removing the attention to the original source X, resulting in a slightly smaller drop in performance (represented by TM). These experiments indicate that incorporating context from the sources significantly contributes to performance, by allowing the model to distinguish between relevant context and noise.