Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Table 10: Comparison of different alternative fine-tuning methods that only update a subset of the model’s parameters. For adapter layers, d refers to the inner dimensionality of the adapters.
['Fine-tuning method', 'GLUE', 'CNNDM', 'SQuAD', 'SGLUE', 'EnDe', 'EnFr', 'EnRo']
For adapter layers, we report the performance using an inner dimensionality d of 32, 128, 512, 2048. Pursuant with past results [houlsby2019parameter; bapna2019simple] we find that lower-resource tasks like SQuAD work well with a small value of d whereas higher resource tasks require a large dimensionality to achieve reasonable performance. This suggests that adapter layers could be a promising technique for fine-tuning on fewer parameters as long as the dimensionality is scaled appropriately to the task size. Note that in our case we treat GLUE and SuperGLUE each as a single “task” by concatenating their constituent datasets, so although they comprise some low-resource datasets the combined dataset is large enough that it necessitates a large value of d. We found that gradual unfreezing caused a minor degradation in performance across all tasks, though it did provide some speedup during fine-tuning. Better results may be attainable by more carefully tuning the unfreezing schedule.