Towards Making the Most of BERT in Neural Machine Translation
Table 2: Ablation of asymptotic distillation on the encoder and the decoder of NMT.
['Models', 'En→De BLEU']
As BERT contains bidirectional information, pre-training decoder may lead inconsistencies between the training and the inference. The GPT-2 Transformer uses constrained self-attention where every token can only attend to context to its left, thus it is natural to introduce GPT-2 to the NMT decoder. While there are still no more significant gains obtained in our experiments. One possible reason is that the decoder is not a typical language model, which contains the information from source attention. We will leave this issue in the future study. We compare BERT with GPT-2Radford et al. on WMT 2014 English-German corpus. The experiments suggest that bidirectional information plays an important role in the encoder of NMT models. While for the decoder part, GPT-2 is a more priority choice. In the following part, we choose BERT as the pre-trained LM and apply only for the encoder part.