Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgetting
Table 2: Comparison of different model initialization strategies: pretrained initialization (PI) and Random Initialization (RI). We report median over 5 runs.
['[BOLD] Method', '[BOLD] CoLA', '[BOLD] STS', '[BOLD] MRPC', '[BOLD] RTE', '[BOLD] Avg']
It is interesting to see whether the choice of initialization strategies would have an impact on the performance of our RecAdam method. It shows that RecAdam, with both initialization strategies, can outperform the vanilla fine-tuning method on all the four tasks. For the target task STS, the model with pretrained initialization can achieve the same result as random initialization. For the other tasks (e.g., CoLA, MRPC, RTE), Random initialize the model would be our best choice. It is because the model would benefit from a larger parameter search space with random initialization. In contrast, with pretrained initialization, the search space would be limited to around the pretraining model, making it harder for the model to learn the new tasks.