Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets
Table 1: BLUE tasks
['Corpus', 'Train', 'Dev', 'Test', 'Task', 'Metrics', 'Domain', 'Avg']
Here, we rely on preexisting datasets because they have been widely used by the BioNLP community as shared tasks. When comparing BERT pre-trained using the base settings against that using the large settings, it is a bit surprising that BERT-Base is better than BERT-Large except in relation extraction and document classification tasks. In addition, BERT-Large pre-trained on PubMed and MIMIC is worse than other models overall. This is partially because the MIMIC-III data are relatively smaller than the PubMed abstracts and, thus, cannot pre-train the large model sufficiently.