Structured Pruning of Large Language Models
Table 4: Compression on downstream fine-tuning
['[BOLD] Parameters 125M', '[BOLD] Parameters (100%)', '[BOLD] SST2 92.43', '[BOLD] MRPC 90.9', '[BOLD] STS-B 90.22', '[BOLD] QNLI 89.77', '[BOLD] Average  [BOLD] 90.83']
We are able to conserve nearly 99% of the performance while reducing the number of parameters by 35%. Our target sparsity level is limited by the fact that the embedding layers consist of a significant portion of the remaining parameters. We believe that higher levels of sparsity could be obtained by also factorizing the embedding layer, similar to Lan et al.