Transfer Fine-Tuning: A BERT Case Study
Table 3: GLUE test results scored by the GLUE evaluation server. The best scores are represented in bold and scores higher than those of BERT-base are underlined.
['ModelTask', 'Semantic Equivalence MRPC', 'Semantic Equivalence STS-B', 'Semantic Equivalence QQP', 'NLI MNLI (m/mm)', 'NLI RTE', 'NLI QNLI', 'Single-Sent. SST', 'Single-Sent. CoLA']
The first set of columns shows the results of semantic equivalence assessment tasks. Our model outperformed BERT-base on MRPC (+0.9 points) and STS-B (+2.7 points). Furthermore, it outperformed even BERT-large by 0.6 points on MRPC and by 1.4 points on STS-B, despite BERT-large having 3.1 times more parameters than our model. \newcitebert described that the next-sentence prediction task in BERT’s pre-training aims to train a model that understands sentence relations. Herein, we argue that such relations are effective at generating representations broadly transferable to various NLP tasks, but are too generic to generate representations for semantic equivalence assessment tasks. Our method allows semantic relations between sentences and phrases that are directly useful for this class of tasks to be learned. Our model presents moderate improvements on most NLI tasks, which supports hypothesis H3. We consider this is because the majority of NLI tasks that require inferences in one-direction, contrary to bi-directional entailment relations of paraphrases, are uni-directional. Our model presents a slightly lower score on SST compared to BERT-base and performed poorly on CoLA.