Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing
Table 10. Comparison of linear layers vs recurrent neural networks for task-specific fine-tuning in named entity recognition (entity-level F1) and relation extraction (micro F1), all using the standard PubMedBERT.
['1pt1.5pt1.5pt Task-Specific Model', 'Linear Layer', 'Bi-LSTM']
With the advent of BERT models and the self-attention mechanism, the utility of explicit sequential modeling becomes questionable. The top layer in the BERT model already captures many non-linear dependencies across the entire text span. Therefore, it’s conceivable that even a linear layer on top can perform competitively. The use of a bidirectional LSTM (Bi-LSTM) does not lead to any gain compared to linear layer.