SenseBERT: Driving Some Sense into BERT
Table 4: Results on the GLUE benchmark test set.
['[EMPTY]', '[BOLD] Score', '[BOLD] CoLA', '[BOLD] SST-2', '[BOLD] MRPC', '[BOLD] STS-B', '[BOLD] QQP', '[BOLD] MNLI', '[BOLD] QNLI', '[BOLD] RTE']
The General Language Understanding Evaluation (GLUE; Wang et al. It consists of 9 different NLP tasks, covering different linguistic phenomena. We evaluate our model on GLUE, in order to verify that SenseBERT gains its lexical semantic knowledge without compromising performance on other downstream tasks. Due to slight differences in the data used for pretraining BERT and SenseBERT (BookCorpus is not publicly available), we trained a BERT\textscbase model with the same data used for our models. BERT\textscbase and SenseBERT\textscbase were both finetuned using the exact same procedures and hyperparameters. Indeed, SenseBERT performs on par with BERT, achieving an overall score of 77.9, compared to 77.5 achieved by BERT\textscbase.