On the use of BERT for Neural Machine Translation
Table 4: FT: initialize NMT encoder with BERT and finetune; Freeze: fix NMT encoder parameters to BERT parameters; Emb: fix encoder embeddding layer with BERT contextual word embeddings.
['[EMPTY]', 'news14', 'news18', 'iwslt15', 'wiki', 'kde', 'OpenSub']
As expected, freezing the encoder with BERT parameters lead to a significant decrease in translation quality. However, other BERT+NMT architectures mostly improve over the baseline both on in-domain and out-of-domain test sets. We conclude, that the information encoded by BERT is useful but not sufficient to perform the translation task. We believe, that the role of the NMT encoder is to encode both information specific to source sentence, but also information specific to the target sentence (which is missing in BERT training).