---
license: mit
---


Please refer to the [SepLLM paper - ICML 2025](https://arxiv.org/abs/2412.12094) and our [`GitHub repository`](https://github.com/HKUDS/SepLLM)  for using this model.

To use the checkpoint of this model, you must install the `transformers-4.38.0.post1+sepllm-py3-none-any.whl` released from our [`GitHub repository`](https://github.com/HKUDS/SepLLM). Below are the reference script for testing and a sample of test results. We conducted testing using `lm_eval==0.4.0`.


```
CUDA_LAUNCH_BLOCKING=1
lm_eval --model hf \
	--model_args pretrained=Gausson/pythia-160m-deduped-n64ht-SepLLM \
	--tasks  arc_challenge,arc_easy,lambada_openai,logiqa,piqa,sciq,winogrande,wsc,wikitext  \
	--num_fewshot 5 \
	--device cuda:0\
	--batch_size 32
```

```
hf (pretrained=Gausson/pythia-160m-deduped-n64ht-SepLLM), gen_kwargs: (), limit: None, num_fewshot: 5, batch_size: 32
|    Tasks     |Version|Filter|n-shot|    Metric     | Value |   |Stderr|
|--------------|-------|------|-----:|---------------|------:|---|-----:|
|arc_challenge |Yaml   |none  |     5|acc            | 0.2133|±  |0.0120|
|              |       |none  |     5|acc_norm       | 0.2389|±  |0.0125|
|arc_easy      |Yaml   |none  |     5|acc            | 0.4735|±  |0.0102|
|              |       |none  |     5|acc_norm       | 0.4474|±  |0.0102|
|lambada_openai|Yaml   |none  |     5|perplexity     |33.4119|±  |1.1735|
|              |       |none  |     5|acc            | 0.3287|±  |0.0065|
|logiqa        |Yaml   |none  |     5|acc            | 0.2243|±  |0.0164|
|              |       |none  |     5|acc_norm       | 0.2734|±  |0.0175|
|piqa          |Yaml   |none  |     5|acc            | 0.6409|±  |0.0112|
|              |       |none  |     5|acc_norm       | 0.6431|±  |0.0112|
|sciq          |Yaml   |none  |     5|acc            | 0.8130|±  |0.0123|
|              |       |none  |     5|acc_norm       | 0.7950|±  |0.0128|
|wikitext      |Yaml   |none  |     5|word_perplexity|29.1903|   |      |
|              |       |none  |     5|byte_perplexity| 1.8793|   |      |
|              |       |none  |     5|bits_per_byte  | 0.9102|   |      |
|winogrande    |Yaml   |none  |     5|acc            | 0.5051|±  |0.0141|
|wsc           |Yaml   |none  |     5|acc            | 0.4038|±  |0.0483|
```

If you find our work helpful, please consider giving us a star ⭐ @ our  [`GitHub repository`](https://github.com/HKUDS/SepLLM) and citing our paper. We greatly appreciate your support 😄
```
@inproceedings{chen2025sepllm,
  title={{SepLLM: Accelerate Large Language Models by Compressing One Segment into One Separator}},
  author={Chen, Guoxuan and Shi, Han and Li, Jiawei and Gao, Yihang and Ren, Xiaozhe and Chen, Yimeng and Jiang, Xin and Li, Zhenguo and Liu, Weiyang and Huang, Chao},
  booktitle={International Conference on Machine Learning},
  year={2025},
  note={Also available at arXiv:2412.12094}
}
```