Structured Pruning of a BERT-based Question Answering Model
Table 5: pruning RoBERTa on NQ, continued distillation training on NQ
['model', '% attn', '% ff', 'LA', 'SA', 'time']
Combining distillation and pruning Distillation may be combined with structured pruning in two distinct ways. First, in the final phase of continued training (after the model has been pruned) the cross-entropy objective function may be replaced by a distillation objective function in which the unpruned model is the teacher and the pruned model is the student. Alternately, the pruning phase itself may be driven by the distillation objective rather than the cross-entropy objective. The 2.9 F1-point gain of pruned×2 is especially notable. Timings are median of 5 decoding runs over the entire NQ developement set on an NVidia V100 using 16-bit floating point with batch size 64. In this experiment, matrices were forced to have sizes that are round numbers, resulting in small changes (<1%) in reported pruning fractions. We also include for comparison RoBERTa-base model than has been similarly distilled, using RoBERTa-large as a teacher.