How to Fine-Tune BERT for Text Classification?
Table 6: Test error rates (%) on eight text classification datasets. The results without ∗ of previous models are the results reported on their papers. / means not reported. ∗ means the results are from our implementation since the Sogou dataset is different from theirs. BERT-Feat means “BERT as features”. BERT-FiT means “BERT + Fine-Tuning”. BERT-ITPT-FiT means “BERT + withIn-Task Pre-Training + Fine-Tuning”. BERT-IDPT-FiT means “BERT + In-Domain Pre-Training + Fine-Tuning”. BERT-CDPT-FiT means “BERT + Cross-Domain Pre-Training + Fine-Tuning”.
['Model', 'IMDb', 'Yelp P.', 'Yelp F.', 'TREC', 'Yah. A.', 'AG', 'DBP', 'Sogou', 'Avg. Δ']
In addition to being slightly worse than BERT-Feat on DBpedia dataset, BERT-FiT outperforms BERT-Feat on the other seven datasets. Moreover, all of the three further pre-training models are better than BERT-FiT model. Using BERT-Feat as a reference, we calculate the average percentage increase of other BERT-FiT models on each dataset. BERT-IDPT-FiT performs best, with an average error rate reduce by 18.57%.