LightPAFF: A Two-Stage Distillation Framework for Pre-training and Fine-tuning
Table 3: Results of LightPAFF on GPT-2 tasks in terms of perplexity (ppl). “w/ FT” and “w/o FT” mean the pre-trained model is evaluated on downstream tasks with and without fine-tuning. “GPT-2” represents the model released by Radford et al. (2019) and is taken as the teacher model, “Transformer small” and “GPT-2 small” share the same model structure and number of parameters, with the only difference that “Transformer small” has no GPT-2 pre-training, “LightPAFF” is our method applied on “GPT-2 small”.
['[EMPTY]', '#Param', 'WikiText-2 w/o FT', 'WikiText-2 w/ FT', 'PTB w/o FT', 'PTB w/ FT', 'WikiText103 w/o FT', 'WikiText103 w/ FT']
Accuracy of It can be seen that lightPAFF achieves large improvements than the models under the same size of parameters without pre-training (Transformer small) and with pre-training (GPT-2 small), while approaches the perplexity of the GPT-2 teacher model in the downstream tasks but with only 1/5× number of parameters.