Towards Making the Most of BERT in Neural Machine Translation
Table 5: Results on WMT14 English-German with rate-scheduled learning. ηlm=1 indicates the fine-tuning approach and ηlm=0 indicates the frozen feature-based approach.
['Models', 'En→De BLEU']
For ηlm=1, the model draws back to the traditional fine-tuning approach, while for ηlm=0, the model is exactly the feature-based approach. The results show that slanted triangular learning rates policy is a more promising strategy for fine-tuning models. We find that changing ηlm during the training phase provides better results than fixed values with a similar or even smaller number of epochs.