Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Table 3: Comparison of RoBERTa (RoBa.) and dapt to adaptation to an irrelevant domain (¬ dapt). Reported results are test macro-F1, except for ChemProt and RCT, for which we report micro-F1, following Beltagy et al. (2019). We report averages across five random seeds, with standard deviations as subscripts. † indicates high-resource settings. Best task performance is boldfaced. See §3.3 for our choice of irrelevant domains.
['[BOLD] Dom.', '[BOLD] Task', 'RoBa.', '[BOLD] dapt', '¬dapt']
We observe that dapt improves over RoBERTa in all domains. For BioMed, CS, and Reviews, we see consistent improvements over RoBERTa, demonstrating the benefit of dapt when the target domain is more distant from RoBERTa’s source domain. The pattern is consistent across high- and low- resource settings. Although dapt does not increase performance on AGNews, the benefit we observe in HyperPartisan suggests that dapt may be useful even for tasks that align more closely with RoBERTa’s source domain. For each task, dapt significantly outperforms adapting to an irrelevant domain, suggesting the importance of pretraining on domain-relevant data. Furthermore, we generally observe that ¬dapt results in worse performance than even RoBERTa on end-tasks. Taken together, these results indicate that in most settings, exposure to more data without considering domain relevance is detrimental to end-task performance. However, there are two tasks (SciERC and ACL-ARC) in which ¬dapt marginally improves performance over RoBERTa. This may suggest that in some cases, continued pretraining on any additional data is useful, as noted in Baevski et al.