Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing
Table 7. Evaluation of the impact of vocabulary and whole word masking on the performance of PubMedBERT on BLURB.
['1pt1.5pt1.5pt', 'Wiki + Books Word Piece', 'Wiki + Books Whole Word', 'PubMed Word Piece', 'PubMed Whole Word']
To assess the impact of pretraining options on downstream applications, we conduct several ablation studies using PubMedBERT as a running example. Using the original BERT vocabulary derived from Wikipedia & BookCorpus (by continual pretraining from the original BERT), the results are significantly worse than using an in-domain vocabulary from PubMed. Additionally, WWM leads to consistent improvement across the board, regardless of the vocabulary in use.