What Would Elsa Do? Freezing Layers During Transformer Fine-Tuning
Table 2: Reproduced results of BERT and RoBERTa on the development sets.
['Model', 'CoLA', 'SST-2', 'MRPC', 'STS-B', 'QQP', 'MNLI', 'QNLI', 'RTE']
Our fine-tuning procedure closely resembles those of BERT and RoBERTa. For hyperparameter tuning, the best learning rate is different for each task, and all of the original authors choose one between 1×10−5 and 5×10−5; thus, we perform line search over the interval with a step size of 1×10−5.