Inexpensive Domain Adaptation of Pretrained Language Models: Case Studies on Biomedical NER and Covid-19 QA
Table 3: Top: NER ablation study. Drop in dev set F1 (w.r.t. GreenBioBERT) when using non-aligned or randomly initialized word vectors instead of aligned word vectors. Bottom: Results (%) on Deepset-AI Covid-QA. EM (exact match) and F1 are evaluated with the SQuAD scorer. “substr”: Predictions that are a substring of the gold answer. Much higher than EM, because many gold answers are not minimal answer spans (see Appendix, “Notes on Covid-QA”, for an example).
['[EMPTY]', 'domain adaptation corpus size', 'EM', 'F1', 'substr']
To test whether the improvements over general-domain BERT are due to the aligned Word2Vec vectors, or just to the availability of additional vectors in general, we perform an ablation study where we replace the aligned vectors with their non-aligned counterparts (by setting W=1 in Eq. Interestingly, the small CORD-19 corpus is enough to achieve this result (compare “CORD-19 only” and “CORD-19+PubMed+PMC”), presumably because it is specific to the target domain and contains the Covid-QA context papers.