Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing
Table 9. Comparison of PubMedBERT performance on BLURB using standard and adversarial pretraining.
['1pt1.5pt1.5pt', 'PubMedBERT', '+ adversarial']
Adversarial pretraining has been shown to be highly effective in boosting performance in general-domain applications Surprisingly, adversarial pretraining generally leads to a slight degradation in performance, with some exceptions such as sentence similarity (BIOSSES). We hypothesize that the reason may be similar to what we observe in pretraining with full texts. Namely, adversarial training is most useful if the pretraining corpus is more diverse and relatively out-domain compared to the application tasks. We leave a more thorough evaluation of adversarial pretraining to future work.