Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Table 14: Hyperparameters for RoBERTa text classifier.
['[BOLD] Hyperparameter', '[BOLD] Assignment']
Similar to dapt, task-adaptive pretraining consists of a second phase of pretraining RoBERTa, but only on the available task-specific training data. In contrast to dapt, which we train for 12.5K steps, we perform tapt for 100 epochs. We artificially augment each dataset by randomly masking different words (using the masking probability of 0.15) across epochs.