Transfer Fine-Tuning: A BERT Case Study
Table 7: Results of the ablation study where the best scores are represented in bold and scores higher than those of BERT-base are underlined. The third row shows performances when conducting only sentential paraphrase classification (+sentence) and the fourth row shows those when conducting only phrasal paraphrase classification with elaborate feature generation (+3way-PP [Elaborate Feature]), as reported in Table 5. The last row shows performances when conducing phrasal paraphrase classification with simple feature generation (+3way-PP [Simple Feature]). Results indicate that sentential and phrasal paraphrase classification complementarily contributes to Simple Transfer Fine-Tuning modeling.
['ModelTask', 'Semantic Equivalence MRPC', 'Semantic Equivalence STS-B', 'Semantic Equivalence QQP', 'NLI MNLI (m/mm)', 'NLI RTE', 'NLI QNLI']
The results and findings are summarized as follows. Our model with simple feature generation (Simple Transfer Fine-Tuning) on BERT-base outperformed BERT on both semantic equivalent assessment and NLI tasks. Furthermore, it performed on-par against BERT-large on MRPC and outperformed it on STS-B and RTE, despite BERT-large having 3.1 times more parameters than our model. The same trend was confirmed on the model trained on BERT-large, where our model outperformed BERT-large on all the tasks except QNLI. Simple Transfer Fine-Tuning also outperformed our model with elaborate feature generation (Transfer Fine-Tuning) on all semantic equivalent assessment and NLI tasks except MRPC. This result implies that elaborate feature generation tightly fits the model to paraphrase identification while sacrifices its generality to relevant tasks. Further investigation will be our future work. Simple Transfer Fine-Tuning achieved higher scores on STS-B, RTE, and QNLI tasks than models trained either with only sentential (+sentence) or phrasal paraphrase (+3way-PP [Simple Feature]) classification. Simple feature generation improves the performance of the model trained with only phrasal paraphrase classification; +3way-PP [Simple Feature] outperformed +3way-PP [Elaborate Feature] on all tasks except RTE.