Transfer Fine-Tuning: A BERT Case Study
Table 6: Test results on semantic equivalence assessment and NLI tasks scored by the GLUE evaluation server. The best scores for each task are represented in bold. The scores higher than those of BERT counterparts (against BERT-base and BERT-large, respectively) are underlined. Our models with simple feature generation (Simple Transfer Fine-Tuning) consistently outperformed the BERT models and achieved the best scores for six out of seven tasks.
['ModelTask', 'Semantic Equivalence MRPC', 'Semantic Equivalence STS-B', 'Semantic Equivalence QQP', 'NLI MNLI (m/mm)', 'NLI RTE', 'NLI QNLI']
The results and findings are summarized as follows. Our model with simple feature generation (Simple Transfer Fine-Tuning) on BERT-base outperformed BERT on both semantic equivalent assessment and NLI tasks. Furthermore, it performed on-par against BERT-large on MRPC and outperformed it on STS-B and RTE, despite BERT-large having 3.1 times more parameters than our model. The same trend was confirmed on the model trained on BERT-large, where our model outperformed BERT-large on all the tasks except QNLI. Simple Transfer Fine-Tuning also outperformed our model with elaborate feature generation (Transfer Fine-Tuning) on all semantic equivalent assessment and NLI tasks except MRPC. This result implies that elaborate feature generation tightly fits the model to paraphrase identification while sacrifices its generality to relevant tasks. Further investigation will be our future work. Simple Transfer Fine-Tuning achieved higher scores on STS-B, RTE, and QNLI tasks than models trained either with only sentential (+sentence) or phrasal paraphrase (+3way-PP [Simple Feature]) classification. Simple feature generation improves the performance of the model trained with only phrasal paraphrase classification; +3way-PP [Simple Feature] outperformed +3way-PP [Elaborate Feature] on all tasks except RTE.