Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing
Table 12. Evaluation of the impact of entity dummification and relation encoding in relation extraction, all using PubMedBERT. With entity dummification, the entity mentions in question are anonymized using entity type tags such as $DRUG or $GENE. With entity marker, special tags marking the start and end of an entity are appended to the entity mentions in question. Relation encoding is derived from the special [CLS] token appended to the beginning of the text or the special entity start token, or by concatenating the contextual representation of the entity mentions in question.
['1pt1.5pt1.5pt Input text', 'Classification Encoding', 'ChemProt', 'DDI']
Simply using the original text indeed exposes the neural methods to significant overfitting risk. Using [CLS] with the original text is the worst choice, as the relation encoding has a hard time to distinguish which entities in the text span are in question. Dummification remains the most reliable method, which works for either relation encoding method. Interestingly, using entity markers leads to slightly better results in both datasets, as it appears to prevent overfitting while preserving useful entity information. We leave it to future work to study whether this would generalize to all relation extraction tasks.