LightPAFF: A Two-Stage Distillation Framework for Pre-training and Fine-tuning
Table 8: The comparison of accuracy, and average number of input tokens in the pre-training task between BERT, GPT-2 and MASS.
['Model', 'Accuracy', 'Input Tokens']
We test the three models on a held-out dataset. In particular, we compute the accuracy of token predictions. It can be seen that BERT achieves the highest accuracy while GPT-2 the lowest. Since the three models are of similar complexity (e.g., with similar architecture and similar number of parameters), our hypothesis is that the pre-training task of BERT is easiest and that of GPT-2 is the most difficult, which is verified intuitively as below.