Context-Aware Neural Machine Translation Learns Anaphora Resolution
Table 1: Automatic evaluation: BLEU. Significant differences at p<0.01 are in bold.
['[BOLD] model', '[BOLD] BLEU']
We use the traditional automatic metric BLEU on a general test set to get an estimate of the overall performance of the discourse-aware model, before turning to more targeted evaluation in the next section. The ‘baseline’ is the discourse-agnostic version of the Transformer. As another baseline we use the standard Transformer applied to the concatenation of the previous and source sentences, as proposed by \newcitetiedemann_neural_2017. \newcitetiedemann_neural_2017 only used a special symbol to mark where the context sentence ends and the source sentence begins. This technique performed badly with the non-recurrent Transformer architecture in preliminary experiments, resulting in a substantial degradation of performance (over 1 BLEU). Instead, we use a binary flag at every word position in our concatenation baseline telling the encoder whether the word belongs to the context sentence or to the source sentence.