Hugging Face's logo

RichardErkhov
/
instruction-pretrain_-_InstructLM-1.3B-gguf

GGUF
Model card Files
xet
Community
RichardErkhov commited on
Commit
ad1c206
·
verified ·
1 Parent(s): 0963e4a

uploaded readme

Browse files
Files changed (1) hide show
  1. README.md +131 -0
README.md ADDED
@@ -0,0 +1,131 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Quantization made by Richard Erkhov.
2
+
3
+ [Github](https://github.com/RichardErkhov)
4
+
5
+ [Discord](https://discord.gg/pvy7H8DZMG)
6
+
7
+ [Request more models](https://github.com/RichardErkhov/quant_request)
8
+
9
+
10
+ InstructLM-1.3B - GGUF
11
+ - Model creator: https://huggingface.co/instruction-pretrain/
12
+ - Original model: https://huggingface.co/instruction-pretrain/InstructLM-1.3B/
13
+
14
+
15
+ | Name | Quant method | Size |
16
+ | ---- | ---- | ---- |
17
+ | [InstructLM-1.3B.Q2_K.gguf](https://huggingface.co/RichardErkhov/instruction-pretrain_-_InstructLM-1.3B-gguf/blob/main/InstructLM-1.3B.Q2_K.gguf) | Q2_K | 0.49GB |
18
+ | [InstructLM-1.3B.IQ3_XS.gguf](https://huggingface.co/RichardErkhov/instruction-pretrain_-_InstructLM-1.3B-gguf/blob/main/InstructLM-1.3B.IQ3_XS.gguf) | IQ3_XS | 0.54GB |
19
+ | [InstructLM-1.3B.IQ3_S.gguf](https://huggingface.co/RichardErkhov/instruction-pretrain_-_InstructLM-1.3B-gguf/blob/main/InstructLM-1.3B.IQ3_S.gguf) | IQ3_S | 0.57GB |
20
+ | [InstructLM-1.3B.Q3_K_S.gguf](https://huggingface.co/RichardErkhov/instruction-pretrain_-_InstructLM-1.3B-gguf/blob/main/InstructLM-1.3B.Q3_K_S.gguf) | Q3_K_S | 0.56GB |
21
+ | [InstructLM-1.3B.IQ3_M.gguf](https://huggingface.co/RichardErkhov/instruction-pretrain_-_InstructLM-1.3B-gguf/blob/main/InstructLM-1.3B.IQ3_M.gguf) | IQ3_M | 0.58GB |
22
+ | [InstructLM-1.3B.Q3_K.gguf](https://huggingface.co/RichardErkhov/instruction-pretrain_-_InstructLM-1.3B-gguf/blob/main/InstructLM-1.3B.Q3_K.gguf) | Q3_K | 0.62GB |
23
+ | [InstructLM-1.3B.Q3_K_M.gguf](https://huggingface.co/RichardErkhov/instruction-pretrain_-_InstructLM-1.3B-gguf/blob/main/InstructLM-1.3B.Q3_K_M.gguf) | Q3_K_M | 0.62GB |
24
+ | [InstructLM-1.3B.Q3_K_L.gguf](https://huggingface.co/RichardErkhov/instruction-pretrain_-_InstructLM-1.3B-gguf/blob/main/InstructLM-1.3B.Q3_K_L.gguf) | Q3_K_L | 0.67GB |
25
+ | [InstructLM-1.3B.IQ4_XS.gguf](https://huggingface.co/RichardErkhov/instruction-pretrain_-_InstructLM-1.3B-gguf/blob/main/InstructLM-1.3B.IQ4_XS.gguf) | IQ4_XS | 0.69GB |
26
+ | [InstructLM-1.3B.Q4_0.gguf](https://huggingface.co/RichardErkhov/instruction-pretrain_-_InstructLM-1.3B-gguf/blob/main/InstructLM-1.3B.Q4_0.gguf) | Q4_0 | 0.72GB |
27
+ | [InstructLM-1.3B.IQ4_NL.gguf](https://huggingface.co/RichardErkhov/instruction-pretrain_-_InstructLM-1.3B-gguf/blob/main/InstructLM-1.3B.IQ4_NL.gguf) | IQ4_NL | 0.73GB |
28
+ | [InstructLM-1.3B.Q4_K_S.gguf](https://huggingface.co/RichardErkhov/instruction-pretrain_-_InstructLM-1.3B-gguf/blob/main/InstructLM-1.3B.Q4_K_S.gguf) | Q4_K_S | 0.73GB |
29
+ | [InstructLM-1.3B.Q4_K.gguf](https://huggingface.co/RichardErkhov/instruction-pretrain_-_InstructLM-1.3B-gguf/blob/main/InstructLM-1.3B.Q4_K.gguf) | Q4_K | 0.77GB |
30
+ | [InstructLM-1.3B.Q4_K_M.gguf](https://huggingface.co/RichardErkhov/instruction-pretrain_-_InstructLM-1.3B-gguf/blob/main/InstructLM-1.3B.Q4_K_M.gguf) | Q4_K_M | 0.77GB |
31
+ | [InstructLM-1.3B.Q4_1.gguf](https://huggingface.co/RichardErkhov/instruction-pretrain_-_InstructLM-1.3B-gguf/blob/main/InstructLM-1.3B.Q4_1.gguf) | Q4_1 | 0.8GB |
32
+ | [InstructLM-1.3B.Q5_0.gguf](https://huggingface.co/RichardErkhov/instruction-pretrain_-_InstructLM-1.3B-gguf/blob/main/InstructLM-1.3B.Q5_0.gguf) | Q5_0 | 0.87GB |
33
+ | [InstructLM-1.3B.Q5_K_S.gguf](https://huggingface.co/RichardErkhov/instruction-pretrain_-_InstructLM-1.3B-gguf/blob/main/InstructLM-1.3B.Q5_K_S.gguf) | Q5_K_S | 0.87GB |
34
+ | [InstructLM-1.3B.Q5_K.gguf](https://huggingface.co/RichardErkhov/instruction-pretrain_-_InstructLM-1.3B-gguf/blob/main/InstructLM-1.3B.Q5_K.gguf) | Q5_K | 0.89GB |
35
+ | [InstructLM-1.3B.Q5_K_M.gguf](https://huggingface.co/RichardErkhov/instruction-pretrain_-_InstructLM-1.3B-gguf/blob/main/InstructLM-1.3B.Q5_K_M.gguf) | Q5_K_M | 0.89GB |
36
+ | [InstructLM-1.3B.Q5_1.gguf](https://huggingface.co/RichardErkhov/instruction-pretrain_-_InstructLM-1.3B-gguf/blob/main/InstructLM-1.3B.Q5_1.gguf) | Q5_1 | 0.95GB |
37
+ | [InstructLM-1.3B.Q6_K.gguf](https://huggingface.co/RichardErkhov/instruction-pretrain_-_InstructLM-1.3B-gguf/blob/main/InstructLM-1.3B.Q6_K.gguf) | Q6_K | 1.03GB |
38
+ | [InstructLM-1.3B.Q8_0.gguf](https://huggingface.co/RichardErkhov/instruction-pretrain_-_InstructLM-1.3B-gguf/blob/main/InstructLM-1.3B.Q8_0.gguf) | Q8_0 | 1.33GB |
39
+
40
+
41
+
42
+
43
+ Original model description:
44
+ ---
45
+ license: apache-2.0
46
+ datasets:
47
+ - tiiuae/falcon-refinedweb
48
+ - instruction-pretrain/ft-instruction-synthesizer-collection
49
+ language:
50
+ - en
51
+ ---
52
+ # Instruction Pre-Training: Language Models are Supervised Multitask Learners
53
+ This repo contains the **general models pre-trained from scratch** in our paper [Instruction Pre-Training: Language Models are Supervised Multitask Learners](https://huggingface.co/papers/2406.14491).
54
+
55
+ We explore supervised multitask pre-training by proposing ***Instruction Pre-Training***, a framework that scalably augments massive raw corpora with instruction-response pairs to pre-train language models. The instruction-response pairs are generated by an efficient instruction synthesizer built on open-source models. In our experiments, we synthesize 200M instruction-response pairs covering 40+ task categories to verify the effectiveness of *Instruction Pre-Training*. Instruction Pre-Training* outperforms *Vanilla Pre-training* in both general pre-training from scratch and domain-adaptive continual pre-training. **In pre-training from scratch, *Instruction Pre-Training* not only improves pre-trained base models but also benefits more from further instruction tuning.** In continual pre-training, *Instruction Pre-Training* enables Llama3-8B to be comparable to or even outperform Llama3-70B.
56
+
57
+ <p align='center'>
58
+ <img src="https://cdn-uploads.huggingface.co/production/uploads/66711d2ee12fa6cc5f5dfc89/vRdsFIVQptbNaGiZ18Lih.png" width="400">
59
+ </p>
60
+
61
+ ## Resources
62
+ **🤗 We share our data and models with example usages, feel free to open any issues or discussions! 🤗**
63
+
64
+ - Context-Based Instruction Synthesizer: [instruction-synthesizer](https://huggingface.co/instruction-pretrain/instruction-synthesizer)
65
+ - Fine-Tuning Data for the Synthesizer: [ft-instruction-synthesizer-collection](https://huggingface.co/datasets/instruction-pretrain/ft-instruction-synthesizer-collection)
66
+ - General Models Pre-Trained from Scratch:
67
+ - [InstructLM-500M](https://huggingface.co/instruction-pretrain/InstructLM-500M)
68
+ - [InstructLM-1.3B](https://huggingface.co/instruction-pretrain/InstructLM-1.3B)
69
+ - Domain-Specific Models Pre-Trained from Llama3-8B:
70
+ - [Finance-Llama3-8B](https://huggingface.co/instruction-pretrain/finance-Llama3-8B)
71
+ - [Biomedicine-Llama3-8B](https://huggingface.co/instruction-pretrain/medicine-Llama3-8B)
72
+ - General Instruction-Augmented Corpora: [general-instruction-augmented-corpora](https://huggingface.co/datasets/instruction-pretrain/general-instruction-augmented-corpora)
73
+ - Domain-Specific Instruction-Augmented Corpora (no finance data to avoid ethical issues): [medicine-instruction-augmented-corpora](https://huggingface.co/datasets/instruction-pretrain/medicine-instruction-augmented-corpora)
74
+
75
+ ## General Pre-Training From Scratch
76
+ We augment the [RefinedWeb corproa](https://huggingface.co/datasets/tiiuae/falcon-refinedweb) with instruction-response pairs generated by our [context-based instruction synthesizer](https://huggingface.co/instruction-pretrain/instruction-synthesizer) to pre-train general langauge models from scratch.
77
+
78
+ To evaluate our general base model using the [lm-evaluation-harness framework](https://github.com/EleutherAI/lm-evaluation-harness)
79
+
80
+ 1. Setup dependencies:
81
+ ```bash
82
+ git clone https://github.com/EleutherAI/lm-evaluation-harness
83
+ cd lm-evaluation-harness
84
+ pip install -e .
85
+ ```
86
+
87
+ 2. Evalaute:
88
+ ```bash
89
+ MODEL=instruction-pretrain/InstructLM-1.3B
90
+ add_bos_token=True # this flag is needed because lm-eval-harness set add_bos_token to False by default, but ours require add_bos_token to be True
91
+
92
+ accelerate launch -m lm_eval --model hf \
93
+ --model_args pretrained=${MODEL},add_bos_token=${add_bos_token},dtype=float16 \
94
+ --gen_kwargs do_sample=False \
95
+ --tasks piqa,hellaswag,winogrande \
96
+ --batch_size auto \
97
+ --num_fewshot 0
98
+
99
+ accelerate launch -m lm_eval --model hf \
100
+ --model_args pretrained=${MODEL},add_bos_token=${add_bos_token},dtype=float16 \
101
+ --gen_kwargs do_sample=False \
102
+ --tasks social_iqa,ai2_arc,openbookqa,boolq,mmlu \
103
+ --batch_size auto \
104
+ --num_fewshot 5
105
+ ```
106
+
107
+ ## Citation
108
+ If you find our work helpful, please cite us:
109
+
110
+ Instruction Pre-Training
111
+ ```bibtex
112
+ @article{cheng2024instruction,
113
+ title={Instruction Pre-Training: Language Models are Supervised Multitask Learners},
114
+ author={Cheng, Daixuan and Gu, Yuxian and Huang, Shaohan and Bi, Junyu and Huang, Minlie and Wei, Furu},
115
+ journal={arXiv preprint arXiv:2406.14491},
116
+ year={2024}
117
+ }
118
+ ```
119
+
120
+ [AdaptLLM](https://huggingface.co/papers/2309.09530)
121
+ ```bibtex
122
+ @inproceedings{
123
+ cheng2024adapting,
124
+ title={Adapting Large Language Models via Reading Comprehension},
125
+ author={Daixuan Cheng and Shaohan Huang and Furu Wei},
126
+ booktitle={The Twelfth International Conference on Learning Representations},
127
+ year={2024},
128
+ url={https://openreview.net/forum?id=y886UXPEZ0}
129
+ }
130
+ ```
131
+