An Empirical Study of Multi-Task Learning on BERT for Biomedical Text Mining
Table 4: Test results of MT-BERT-Fine-Tune models on clinical tasks.
['Model', 'ClinicalSTS', 'i2b2 2010 re', 'MedNLI', 'ShARe/CLEFE', '[ITALIC] Avg']
First, we would like to compare multi-task learning on BERT variants: BioBERT, ClinicalBERT, and BlueBERT. When compared the MTL models using BERT model pretrained on PubMed only (rows 2 and 3) and on the combination of PubMed and clinical notes (row 4), it shows the impact of using clinical notes during the pretraining process. This observation is consistently as shown in Peng et al. On the other hand, MT-ClinicalBERT-Fine-Tune, which used ClinicalBERT during the pretraining, drops ∼1.6% across the tasks. The differences between ClinicalBERT and BlueBERT are at least in 2-fold. (1) ClinicalBERT used “cased” text while BlueBERT used “uncased” text; and (2) the number of epochs to continuously pretrained the model. Given that there are limited details of pretraining ClinicalBERT, further investigation may be necessary.