Distilling Transformers into Simple Neural Networks with Unlabeled Transfer DataDraft under review.
Table 12: Different training schedules for distillation with BERT. ‘|’ indicates different stages and ‘+’ indicates joint optimization. The first block shows distillation results with BERT Base and the second block with BERT Large. The training scheme in the first column optimizes the representation loss in the first stage, and then jointly optimizes the logit loss and cross-entropy loss in the second stage with gradual unfreezing of the layers. The one in the second column optimizes all the loss functions jointly. Whereas the last one first optimizes the representation and logit loss over unlabeled data, followed by fine-tuning on labeled data.
['Dataset', 'RRL\xa0|', 'RRL+', 'RRL+RLL\xa0|']
We had proposed three different schedules for distilling the student model based on joint and stagewise optimization with gradual unfreezing. We observe that the stagewise schedule (RRL |RLL+RCE in table) where we first optimize the loss RRL (cf. RLL (cf.