Towards Non-task-specific Distillation of BERT via Sentence Representation Approximation
Table 5: Results of untuned sentence representing models on QQP dataset.
['Models', 'Acc', 'F1']
It should be noted that, in this table, ELMO, BERTBASE (CLS) and BERTBASE (averaged) are introduced as the comparison basis, since they can give the SOTA untuned sentence representations for the similarity measurement. The comparison mainly focuses on the performances of our proposed BiLSTMSRA and BiLSTMKD. For a thorough comparison, we define the training objective of BiLSTMKD as fitting the cosine similarity score of the sentence pair directly given by the pre-trained BERT, which means both the teacher BERT and KD do not utilize the labels of QQP dataset. It can be seen that, even though the training goal of BiLSTMKD is more direct than BiLSTMSRA, our BiLSTMSRA outperforms the former on both the metrics, and meanwhile achieves scores closed to those of BERTBASE. Besides, we can also observe that, for sentence similarity quantification, averaging the context word embeddings as the sentence representation (ELMO and BERTBASE (averaged)) works better than taking the final hidden state corresponding to the [CLS] token (BERTBASE (CLS)).