Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Table 8: Mean test set micro-F1 (for ChemProt and RCT) and macro-F1 (for ACL-ARC), across five random seeds, with standard deviations as subscripts, comparing rand-tapt (with 50 candidates) and knn-tapt selection. Neighbors of the task data are selected from the domain data.
['Pretraining', 'BioMed ChemProt', 'BioMed RCT-500', 'CS ACL-ARC']
that knn-tapt outperforms tapt for all cases. rand-tapt is generally worse than knn-tapt, but within a standard deviation arising from 5 seeds for RCT and ACL-ARC. As we increase k, knn-tapt performance steadily increases, and approaches that of dapt. Future work might consider a closer study of knn-tapt, more sophisticated data selection methods, and the tradeoff between the diversity and task relevance of selected examples.