Parameter-Efficient Transfer Learning for NLP
Table 2: Test accuracy for additional classification tasks. In these experiments we transfer from the BERTBASE model. For each task and algorithm, the model with the best validation set accuracy is chosen. We report the mean test accuracy and s.e.m. across runs with different random seeds.
['Dataset baseline', '5cmNo BERT 5cmBERTBASE', '[EMPTY]', '[EMPTY]', '[EMPTY]']
The AutoML baseline demonstrates that the BERT models are competitive. This baseline explores thousands of models, yet the BERT models perform better on average. We see similar pattern of results to GLUE. The performance of adapter-tuning is close to full fine-tuning (0.4% behind). Fine-tuning requires 17× the number of parameters to BERTBASE to solve all tasks. Variable fine-tuning performs slightly better than fine-tuning, whilst training fewer layers. The optimal setting of variable fine-tuning results in training 52% of the network on average per task, reducing the total to 9.9× parameters. Adapters, however, offer a much more compact model. They introduce 1.14% new parameters per task, resulting in 1.19× parameters for all 17 tasks. Removing the adapters from the layers 0−4 on MNLI barely affects performance. This indicates that adapters perform well because they automatically prioritize higher layers. One intuition is that the lower layers extract lower-level features that are shared among tasks, while the higher layers build features that are unique to different tasks.