Towards Making the Most of BERT in Neural Machine Translation
Table 6: Results on WMT14 English-German with different layers of BERT.
['Models', 'En→De BLEU']
We implement asymptotic distilling by applying auxiliary L2 loss between a specific NMT encoder layer and a specific BERT layer. In our experiments, we add the supervision signal to the 3th encoder layer. Intuitively, the last layer of the pre-trained LM model is impacted by the LM target-related objective (i.e. masked language model and next sentence prediction) during pre-training and hence the last layer is biased to the LM targets.