Structured Pruning of Large Language Models
(a) SRU
['[BOLD] Parameters 35M', '[BOLD] Parameters (100%)', '[BOLD] FLOP 1.24', '[BOLD] AGP (unstructured) -', '[BOLD] AGP (structured) -', '[BOLD] Dense Model -']
The results conform to our expectations and to the results reported in previous work – pruning a large model is consistently better than training a small dense model from scratch. Furthermore, FLOP exceeds the performance of the unstructured AGP method at all sparsity levels tested. For instance, we achieve a loss of 0.01 bits-per-character (BPC) (less than 1% relative performance) using 30% of the parameters, while the AGP baseline has a loss of 0.04 BPC. Again, both pruning methods significantly outperform training small dense models from scratch. Our method achieves results on par with the unstructured pruning baseline, being marginally worse at 90% sparsity but slightly better at 80% sparsity.