Similarity Analysis of Contextual Word Representation Models
Table 1: Freezing top/bottom 4/6 layers of BERT and XLNet during fine-tuning.
['[EMPTY]', 'Froze', 'SST-2', 'MNLI', 'QNLI', 'QQP']
On all of the tasks except QQP, freezing the bottom layers resulted in better performance than freezing the top layers. One interesting observation is that as we increase the number of bottom layers for freezing to six, the performance marginally degrades while saving a lot more computation. Surprisingly, on SST-2 and QNLI, freezing the bottom six layers resulted in better or equal performance than not freezing any layers of both models. With freezing the bottom six layers, one can save back-propagation computation by more than 50%.