Movement Pruning: Adaptive Sparsity by Fine-Tuning
Table 2: Performance at high sparsity levels. (Soft) movement pruning outperforms current state-of-the art pruning methods at different high sparsity levels.
['[EMPTY]', 'BERT base fine-tuned', 'Remaining Weights (%)', 'MaP', '[ITALIC] L0 Regu', 'MvP', 'soft MvP']
Magnitude pruning on SQuAD achieves 54.5 F1 with 3% of the weights compared to 73.6 F1 with L0 regularization, 76.3 F1 for movement pruning, and 79.9 F1 with soft movement pruning. These experiments indicate that in high sparsity regimes, importance scores derived from the movement accumulated during fine-tuning induce significantly better pruned models compared to absolute values.