Revisiting Few-sample BERT Fine-tuning
Table 2: Mean test performance and standard deviation on four datasets. Numbers that are statistically significantly better than the standard setting (left column) are in blue and underlined. The results of Re-init and Longer are copied from Table 1. All experiments use Adam with debiasing (Section 4). Except Longer, all methods are trained with three epochs.
['[EMPTY]', 'Standard', 'Int. Task', 'LLRD', 'Mixout', 'Pre-trained WD', 'WD', 'Re-init', 'Longer']
Task, Mixout, Pre-trained WD, and WD are generally higher when using the Adam with debiasing. However, we observe less pronounced benefits for all surveyed methods compared to results originally reported. At times, these methods do not outperform the standard baselines or simply training longer. Using additional annotated data for intermediate task training continues to be effective, leading to consistent improvement over the average performance across all datasets. LLRD and Mixout show less consistent performance impact. We observe no noticeable improvement using pre-trained weight decay and conventional weight decay in improving or stabilizing BERT fine-tuning in our experiments, contrary to existing work This indicates that these methods potentially ease the optimization difficulty brought by the debiasing omission in BERTAdam, and when we add the debiasing, the positive effects are reduced. However, the relatively large margin between our results and previously published results indicates an improvement. More important, our focus is the relative improvement, or lack of improvement compared to simply training longer.