How to Fine-Tune BERT for Text Classification?
Table 5: Performance of in-domain and cross-domain further pre-training on seven datasets. Each was further pre-trained for 100k steps. The first column indicates the different further pre-training dataset. “all sentiment” means the dataset consists of all the training datasets in sentiment domain. “all” means the dataset consists of all the seven training datasets. Note that some of the data in Yelp P. and Yelp F. are overlapping, e.g., part of the data in the test set of Yelp P. will appear in the training set of Yelp F., so we remove this part of data from the training sets during further pre-training.
['Domain Dataset', 'sentiment IMDb', 'sentiment Yelp P.', 'sentiment Yelp F.', 'question TREC', 'question Yah. A.', 'topic AG’s News', 'topic DBPedia']
Generally, in-domain pretraining can bring better performance than within-task pretraining. On the small sentence-level TREC dataset, within-task pre-training do harm to the performance while in-domain pre-training which utilizes Yah. A. corpus can achieve better results on TREC. It is reasonable since BERT is already trained on a general domain. We implement BERT-Feat through using the feature from BERT model as the input embedding of the biLSTM with self-attention Lin et al.