Towards Non-task-specific Distillation of BERT via Sentence Representation Approximation
Table 3: The accuracy scores evaluated on the SST-2 validation set. The models are trained with different proportions of the training data.
['Models', '20%', '30%', '50%', '100%']
BERTLARGE trained on the corresponding data stands for the teacher model of each BiLSTMKD. With no doubt, all the models can achieve better results using more training data, while BERT performs the best. BERT even successfully predicts 91.9% of validation samples under only 20% training data. Comparing with the pure BiLSTM models, the BiLSTMKD models slightly improve the performances by 1%∼2%, whereas BiLSTMSRA outperforms the best BiLSTM model as well as the BiLSTMKD trained with 20% and 30% percent data respectively. Besides, similar to BERT, the difference of accuracy between BiLSTMSRA trained with 20% and the one using 100% corpus is relatively small. This phenomenon indicates that our model converges faster and is less dependent on the amount of training data for downstream tasks.