Towards Making the Most of BERT in Neural Machine Translation
Table 3: Ablation of using different data size with asymptotic distillation.
['[EMPTY]', 'Transformer', 'Fine-tuning', 'AD']
We conduct experiments on the performance of asymptotic distillation model on different amounts of training data. The experiments is in line with our intuition that with the increasing training data, the gains of fine-tuning will gradually diminish. While with the asymptotic distillation, we achieve continuous improvements.