Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Table 6: Though tapt is effective (Table 5), it is harmful when applied across tasks. These findings illustrate differences in task distributions within a domain.
['BioMed', 'RCT', 'ChemProt']
For instance, we further pretrain the LM using the RCT unlabeled data, fine-tune it with the ChemProt labeled data, and observe the effect. We refer to this setting as Transfer-tapt. We see that tapt optimizes for single task performance, to the detriment of cross-task transfer. These results demonstrate that data distributions of tasks within a given domain might differ. Further, this could also explain why adapting only to a broad domain is not sufficient, and why tapt after dapt is effective.