1 Introduction
Table 3: L0-close fine-tuning results: layers excluded from fine-tuning, corresponding number of parameters remaining to fine-tune, and the fine-tuning performance on the MRPC task (F1 score); other GLUE tasks show similar patterns. We report the mean and standard deviation across 10 independent runs.
['Layers excluded from fine-tuning', 'Task-specific parameter storage', 'F1 score']
Informed by different layers’ sensitivity to fine-tuning, we performed fine-tuning experiments by progressively excluding: (1) key projection layers in self-attention across all encoder stack layers, (2) the penultimate and ultimate encoder stacks, and (3) the word embedding layer. Each of these exclusions independently or all three combined do not substantially degrade performance, while reducing the number of parameters to fine-tune by up to 40% (from 109M to 66M). Layer-wise sparsity levels of supermasks also demonstrate systematic trends (1) across GLUE tasks, WQ, WK and WI tend to be sparser than WV, WD and WO, and (2) shallower encoder stack layers are sparser than deeper ones. Optimization of only task-specific layers does not lead to successful fine-tuning. For instance, for the MRPC task, freezing parameter weights in the pre-trained model and optimizing the task-specific last layer alone yields a non-performing model. Across 10 independent runs, the model consistently predicts all 1’s for the paraphrase classification task, yielding an F1 score of 81.2. Thus, it is critical to fine-tune layers in the pre-trained model and not just the task-specific layers alone.