Towards Making the Most of BERT in Neural Machine Translation
Table 4: Results on WMT14 English-German with different feeding strategies. ‘+’ indicates average pooling and ‘@’ indicates dynamic switch.
['Models', 'En→De BLEU']
We then compare different ways to combine the embedding vector and the BERT features which will be fed into the Transformer encoder. The model cannot get meaningful results due to gradient vanish problem. This also suggests that good initialization help to train the deep model. In the third row, we replace the NMT encoder with BERT and keep the BERT parameters frozen during fine-tuning, the performance lags behind the baseline, which indicates the importance of the original NMT encoder. According to the above experimental results, we combine both the BERT and NMT encoder. In the fourth row, the average pooling method obtains a gain of 1.3 BLEU score over the baseline model showing the power of combination. Finally, the dynamic switch strategy keep the balance between BERT and NMT and achieve a substantial improvement of 0.9 BLEU score over the average pooling approach.