Movement Pruning: Adaptive Sparsity by Fine-Tuning
Table 3: Distillation-augmented performances for selected high sparsity levels. All pruning methods benefit from distillation signal further enhancing the ratio Performance VS Model Size.
['[EMPTY]', 'BERT base fine-tuned', 'Remaining Weights (%)', 'MaP', '[ITALIC] L0 Regu', 'MvP', 'soft MvP']
The training objective is a linear combination of the training loss and a knowledge distillation loss on the output distributions. Overall, we observe that the relative comparisons of the pruning methods remain unchanged while the performances are strictly increased. When combined with distillation, soft movement pruning yields the strongest performances across all pruning methods and studied datasets: it reaches 95% of BERT-base with only a fraction of the weights in the encoder (∼5% on SQuAD and MNLI).