Revisiting Few-sample BERT Fine-tuning
Table 6: Average Test performance with standard deviation on four small datasets with four different pre-trained models. For each setting, the better numbers are bolded and are in blue if the improvement is statistically significant.
['[EMPTY]', 'RTE Standard', 'RTE Re-init', 'MRPC Standard', 'MRPC Re-init', 'STS-B Standard', 'STS-B Re-init', 'CoLA Standard', 'CoLA Re-init']
We study more recent pre-trained contexual embedding models beyond BERTLarge. We investigate whether Re-init provides better fine-tuning initialization in XLNetLarge Yang et al. RoBERTaLarge Liu et al. XLNet is an autoregressive language model trained by learning all permutations of natural language sentences. RoBERTa is similar to BERT in terms of model architecture but is only pre-trained on the mask language modeling task only, but for longer and on more data. BART is a sequence-to-sequence model trained as a denoising autoencoder. ELECTRA is a BERT-like model trained to distinguish tokens generated by masked language model from tokens drawn from the natural distribution. Together, they represent a diverse range of modeling choices in pre-training, including different model architectures, objectives, data, and training strategies. We use Adam with debiasing to fine-tune these models on RTE, MRPC, STS-B, and COLA, using the hyperparameters that are either described in the paper or in the official repository of each model. The experimental setup is kept the same as our other experiments. We observe that several models suffer from high instability on these datasets and in most cases, Re-init can reduce the performance variance. We observe that for some models, like XLNetLarge or RoBERTaLarge, Re-init can improve the average performance and reduce variance. However, the behavior of Re-init varies significantly across different models and Re-init have less significant improvement for ELECTRALarge and BARTLarge. Further study of the entire model family requires significant computational resources, and we leave it as an important direction for future work.