Hugging Face's logo

Sean13
/
llama-8b-instruct-dpo-full

Text Generation
Transformers
TensorBoard
Safetensors
llama
Generated from Trainer
trl
dpo
conversational
text-generation-inference
Model card Files
xet
Metrics Community
Sean13 commited on
Commit
118db30
·
verified ·
1 Parent(s): e643abb

Model save

Browse files
Files changed (10) hide show
  1. README.md +67 -0
  2. all_results.json +9 -0
  3. generation_config.json +12 -0
  4. model-00001-of-00004.safetensors +1 -1
  5. model-00002-of-00004.safetensors +1 -1
  6. model-00003-of-00004.safetensors +1 -1
  7. model-00004-of-00004.safetensors +1 -1
  8. runs/Sep23_16-36-00_pm-d04f/events.out.tfevents.1758616585.pm-d04f.4163239.0 +2 -2
  9. train_results.json +9 -0
  10. trainer_state.json +1596 -0
README.md ADDED
@@ -0,0 +1,67 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: transformers
3
+ model_name: llama-8b-instruct-dpo-full
4
+ tags:
5
+ - generated_from_trainer
6
+ - trl
7
+ - dpo
8
+ licence: license
9
+ ---
10
+
11
+ # Model Card for llama-8b-instruct-dpo-full
12
+
13
+ This model is a fine-tuned version of [None](https://huggingface.co/None).
14
+ It has been trained using [TRL](https://github.com/huggingface/trl).
15
+
16
+ ## Quick start
17
+
18
+ ```python
19
+ from transformers import pipeline
20
+
21
+ question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
22
+ generator = pipeline("text-generation", model="Sean13/llama-8b-instruct-dpo-full", device="cuda")
23
+ output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0]
24
+ print(output["generated_text"])
25
+ ```
26
+
27
+ ## Training procedure
28
+
29
+
30
+
31
+ This model was trained with DPO, a method introduced in [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://huggingface.co/papers/2305.18290).
32
+
33
+ ### Framework versions
34
+
35
+ - TRL: 0.12.2
36
+ - Transformers: 4.46.3
37
+ - Pytorch: 2.7.1
38
+ - Datasets: 4.0.0
39
+ - Tokenizers: 0.20.3
40
+
41
+ ## Citations
42
+
43
+ Cite DPO as:
44
+
45
+ ```bibtex
46
+ @inproceedings{rafailov2023direct,
47
+ title = {{Direct Preference Optimization: Your Language Model is Secretly a Reward Model}},
48
+ author = {Rafael Rafailov and Archit Sharma and Eric Mitchell and Christopher D. Manning and Stefano Ermon and Chelsea Finn},
49
+ year = 2023,
50
+ booktitle = {Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023},
51
+ url = {http://papers.nips.cc/paper_files/paper/2023/hash/a85b405ed65c6477a4fe8302b5e06ce7-Abstract-Conference.html},
52
+ editor = {Alice Oh and Tristan Naumann and Amir Globerson and Kate Saenko and Moritz Hardt and Sergey Levine},
53
+ }
54
+ ```
55
+
56
+ Cite TRL as:
57
+
58
+ ```bibtex
59
+ @misc{vonwerra2022trl,
60
+ title = {{TRL: Transformer Reinforcement Learning}},
61
+ author = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallouédec},
62
+ year = 2020,
63
+ journal = {GitHub repository},
64
+ publisher = {GitHub},
65
+ howpublished = {\url{https://github.com/huggingface/trl}}
66
+ }
67
+ ```
all_results.json ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "epoch": 1.0,
3
+ "total_flos": 0.0,
4
+ "train_loss": 0.5544820035624708,
5
+ "train_runtime": 9953.5203,
6
+ "train_samples": 59875,
7
+ "train_samples_per_second": 6.015,
8
+ "train_steps_per_second": 0.094
9
+ }
generation_config.json ADDED
@@ -0,0 +1,12 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token_id": 128000,
3
+ "do_sample": true,
4
+ "eos_token_id": [
5
+ 128001,
6
+ 128009
7
+ ],
8
+ "max_length": 4096,
9
+ "temperature": 0.6,
10
+ "top_p": 0.9,
11
+ "transformers_version": "4.46.3"
12
+ }
model-00001-of-00004.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:9777e98ef039af03c5e2139a007f5fc48e14ff8ab0545a8f769a3946feb64a98
3
  size 4976698672
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:246704731b07c31fb903df856b19a63d189ac28ae9801e489cca5e03d83da849
3
  size 4976698672
model-00002-of-00004.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:2e15ccb9a427ac802508503affa8f62feb58198845d74d410611f41be2dbf373
3
  size 4999802720
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:607cc01f6398770352b6309023c44156bbb615c692f64de59d99add119e786ee
3
  size 4999802720
model-00003-of-00004.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:8472acdbd198ac54552ea167046cb8b956e78e5308cfbce087d81909fa45bd13
3
  size 4915916176
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:10137a3b34a8a45946c1948c80fd927debb8035d7610720ab9be3f4a3e417e04
3
  size 4915916176
model-00004-of-00004.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:90a50880f8c66d19b576d8af5761766d7a704089ba02a3001fd92037e9083431
3
  size 1168138808
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:26c3ec087adcab42eecc960922abbe8880fd6a2b579251d05c4297a80adff704
3
  size 1168138808
runs/Sep23_16-36-00_pm-d04f/events.out.tfevents.1758616585.pm-d04f.4163239.0 CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:ea13e7212dc69c7a89d933ff7b346c9e62a4a02c77943f89b386e0f88d35dd0d
3
- size 67986
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d80e3b7d3a4b2eb6057ea2dbdb326e2b83948a527ca83af632b0cc60b2b854ba
3
+ size 78024
train_results.json ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "epoch": 1.0,
3
+ "total_flos": 0.0,
4
+ "train_loss": 0.5544820035624708,
5
+ "train_runtime": 9953.5203,
6
+ "train_samples": 59875,
7
+ "train_samples_per_second": 6.015,
8
+ "train_steps_per_second": 0.094
9
+ }
trainer_state.json ADDED
@@ -0,0 +1,1596 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "best_metric": null,
3
+ "best_model_checkpoint": null,
4
+ "epoch": 1.0,
5
+ "eval_steps": 100,
6
+ "global_step": 936,
7
+ "is_hyper_param_search": false,
8
+ "is_local_process_zero": true,
9
+ "is_world_process_zero": true,
10
+ "log_history": [
11
+ {
12
+ "epoch": 0.0010683760683760685,
13
+ "grad_norm": 5.542751238453549,
14
+ "learning_rate": 3.1914893617021273e-09,
15
+ "logits/chosen": -0.16796875,
16
+ "logits/rejected": -0.3359375,
17
+ "logps/chosen": -105.0,
18
+ "logps/rejected": -119.0,
19
+ "loss": 0.6914,
20
+ "rewards/accuracies": 0.0,
21
+ "rewards/chosen": 0.0,
22
+ "rewards/margins": 0.0,
23
+ "rewards/rejected": 0.0,
24
+ "step": 1
25
+ },
26
+ {
27
+ "epoch": 0.010683760683760684,
28
+ "grad_norm": 5.981793978419444,
29
+ "learning_rate": 3.191489361702128e-08,
30
+ "logits/chosen": -0.373046875,
31
+ "logits/rejected": -0.341796875,
32
+ "logps/chosen": -144.0,
33
+ "logps/rejected": -144.0,
34
+ "loss": 0.6925,
35
+ "rewards/accuracies": 0.3055555522441864,
36
+ "rewards/chosen": -0.0011444091796875,
37
+ "rewards/margins": -0.0002422332763671875,
38
+ "rewards/rejected": -0.000904083251953125,
39
+ "step": 10
40
+ },
41
+ {
42
+ "epoch": 0.021367521367521368,
43
+ "grad_norm": 5.578450607295297,
44
+ "learning_rate": 6.382978723404255e-08,
45
+ "logits/chosen": -0.443359375,
46
+ "logits/rejected": -0.40625,
47
+ "logps/chosen": -152.0,
48
+ "logps/rejected": -152.0,
49
+ "loss": 0.6929,
50
+ "rewards/accuracies": 0.21250000596046448,
51
+ "rewards/chosen": -0.0003757476806640625,
52
+ "rewards/margins": -0.00022792816162109375,
53
+ "rewards/rejected": -0.00014495849609375,
54
+ "step": 20
55
+ },
56
+ {
57
+ "epoch": 0.03205128205128205,
58
+ "grad_norm": 5.334397741773421,
59
+ "learning_rate": 9.574468085106384e-08,
60
+ "logits/chosen": -0.4375,
61
+ "logits/rejected": -0.421875,
62
+ "logps/chosen": -176.0,
63
+ "logps/rejected": -176.0,
64
+ "loss": 0.6926,
65
+ "rewards/accuracies": 0.26249998807907104,
66
+ "rewards/chosen": -0.000797271728515625,
67
+ "rewards/margins": -0.0012969970703125,
68
+ "rewards/rejected": 0.000499725341796875,
69
+ "step": 30
70
+ },
71
+ {
72
+ "epoch": 0.042735042735042736,
73
+ "grad_norm": 5.495670488086666,
74
+ "learning_rate": 1.276595744680851e-07,
75
+ "logits/chosen": -0.50390625,
76
+ "logits/rejected": -0.44140625,
77
+ "logps/chosen": -165.0,
78
+ "logps/rejected": -162.0,
79
+ "loss": 0.6927,
80
+ "rewards/accuracies": 0.3375000059604645,
81
+ "rewards/chosen": 0.0001430511474609375,
82
+ "rewards/margins": 0.0002040863037109375,
83
+ "rewards/rejected": -6.246566772460938e-05,
84
+ "step": 40
85
+ },
86
+ {
87
+ "epoch": 0.053418803418803416,
88
+ "grad_norm": 5.614534340749398,
89
+ "learning_rate": 1.5957446808510638e-07,
90
+ "logits/chosen": -0.46484375,
91
+ "logits/rejected": -0.3984375,
92
+ "logps/chosen": -149.0,
93
+ "logps/rejected": -147.0,
94
+ "loss": 0.6928,
95
+ "rewards/accuracies": 0.32499998807907104,
96
+ "rewards/chosen": -0.0003833770751953125,
97
+ "rewards/margins": -0.00052642822265625,
98
+ "rewards/rejected": 0.000141143798828125,
99
+ "step": 50
100
+ },
101
+ {
102
+ "epoch": 0.0641025641025641,
103
+ "grad_norm": 5.400523674987204,
104
+ "learning_rate": 1.9148936170212767e-07,
105
+ "logits/chosen": -0.466796875,
106
+ "logits/rejected": -0.421875,
107
+ "logps/chosen": -135.0,
108
+ "logps/rejected": -132.0,
109
+ "loss": 0.6922,
110
+ "rewards/accuracies": 0.38749998807907104,
111
+ "rewards/chosen": 0.000469207763671875,
112
+ "rewards/margins": 0.0019683837890625,
113
+ "rewards/rejected": -0.00150299072265625,
114
+ "step": 60
115
+ },
116
+ {
117
+ "epoch": 0.07478632478632478,
118
+ "grad_norm": 5.56521185233665,
119
+ "learning_rate": 2.2340425531914892e-07,
120
+ "logits/chosen": -0.341796875,
121
+ "logits/rejected": -0.33984375,
122
+ "logps/chosen": -179.0,
123
+ "logps/rejected": -173.0,
124
+ "loss": 0.6918,
125
+ "rewards/accuracies": 0.4000000059604645,
126
+ "rewards/chosen": 0.00177001953125,
127
+ "rewards/margins": 0.001708984375,
128
+ "rewards/rejected": 6.246566772460938e-05,
129
+ "step": 70
130
+ },
131
+ {
132
+ "epoch": 0.08547008547008547,
133
+ "grad_norm": 5.1781495329861995,
134
+ "learning_rate": 2.553191489361702e-07,
135
+ "logits/chosen": -0.40625,
136
+ "logits/rejected": -0.35546875,
137
+ "logps/chosen": -139.0,
138
+ "logps/rejected": -146.0,
139
+ "loss": 0.6911,
140
+ "rewards/accuracies": 0.512499988079071,
141
+ "rewards/chosen": 0.0019683837890625,
142
+ "rewards/margins": 0.0035400390625,
143
+ "rewards/rejected": -0.00156402587890625,
144
+ "step": 80
145
+ },
146
+ {
147
+ "epoch": 0.09615384615384616,
148
+ "grad_norm": 6.098617800411569,
149
+ "learning_rate": 2.872340425531915e-07,
150
+ "logits/chosen": -0.4453125,
151
+ "logits/rejected": -0.412109375,
152
+ "logps/chosen": -151.0,
153
+ "logps/rejected": -154.0,
154
+ "loss": 0.6911,
155
+ "rewards/accuracies": 0.4375,
156
+ "rewards/chosen": 0.00153350830078125,
157
+ "rewards/margins": 0.002899169921875,
158
+ "rewards/rejected": -0.001373291015625,
159
+ "step": 90
160
+ },
161
+ {
162
+ "epoch": 0.10683760683760683,
163
+ "grad_norm": 5.6077020917115705,
164
+ "learning_rate": 2.9996241442585123e-07,
165
+ "logits/chosen": -0.408203125,
166
+ "logits/rejected": -0.375,
167
+ "logps/chosen": -138.0,
168
+ "logps/rejected": -138.0,
169
+ "loss": 0.6898,
170
+ "rewards/accuracies": 0.4375,
171
+ "rewards/chosen": -0.00506591796875,
172
+ "rewards/margins": 0.003265380859375,
173
+ "rewards/rejected": -0.00836181640625,
174
+ "step": 100
175
+ },
176
+ {
177
+ "epoch": 0.10683760683760683,
178
+ "eval_logits/chosen": -0.482421875,
179
+ "eval_logits/rejected": -0.4609375,
180
+ "eval_logps/chosen": -175.0,
181
+ "eval_logps/rejected": -169.0,
182
+ "eval_loss": 0.6892210841178894,
183
+ "eval_rewards/accuracies": 0.5322580933570862,
184
+ "eval_rewards/chosen": -0.000949859619140625,
185
+ "eval_rewards/margins": 0.006622314453125,
186
+ "eval_rewards/rejected": -0.007568359375,
187
+ "eval_runtime": 101.5044,
188
+ "eval_samples_per_second": 19.319,
189
+ "eval_steps_per_second": 0.611,
190
+ "step": 100
191
+ },
192
+ {
193
+ "epoch": 0.11752136752136752,
194
+ "grad_norm": 5.842441721129293,
195
+ "learning_rate": 2.9973279301399443e-07,
196
+ "logits/chosen": -0.482421875,
197
+ "logits/rejected": -0.51171875,
198
+ "logps/chosen": -159.0,
199
+ "logps/rejected": -156.0,
200
+ "loss": 0.6881,
201
+ "rewards/accuracies": 0.6000000238418579,
202
+ "rewards/chosen": -0.00445556640625,
203
+ "rewards/margins": 0.009765625,
204
+ "rewards/rejected": -0.01422119140625,
205
+ "step": 110
206
+ },
207
+ {
208
+ "epoch": 0.1282051282051282,
209
+ "grad_norm": 5.972847664297803,
210
+ "learning_rate": 2.9929475029988037e-07,
211
+ "logits/chosen": -0.404296875,
212
+ "logits/rejected": -0.35546875,
213
+ "logps/chosen": -164.0,
214
+ "logps/rejected": -168.0,
215
+ "loss": 0.6867,
216
+ "rewards/accuracies": 0.6499999761581421,
217
+ "rewards/chosen": -0.01043701171875,
218
+ "rewards/margins": 0.01416015625,
219
+ "rewards/rejected": -0.024658203125,
220
+ "step": 120
221
+ },
222
+ {
223
+ "epoch": 0.1388888888888889,
224
+ "grad_norm": 5.896436267236366,
225
+ "learning_rate": 2.986488960192327e-07,
226
+ "logits/chosen": -0.515625,
227
+ "logits/rejected": -0.458984375,
228
+ "logps/chosen": -157.0,
229
+ "logps/rejected": -158.0,
230
+ "loss": 0.6849,
231
+ "rewards/accuracies": 0.675000011920929,
232
+ "rewards/chosen": -0.0196533203125,
233
+ "rewards/margins": 0.023193359375,
234
+ "rewards/rejected": -0.04296875,
235
+ "step": 130
236
+ },
237
+ {
238
+ "epoch": 0.14957264957264957,
239
+ "grad_norm": 6.691562977403101,
240
+ "learning_rate": 2.977961291721137e-07,
241
+ "logits/chosen": -0.47265625,
242
+ "logits/rejected": -0.453125,
243
+ "logps/chosen": -166.0,
244
+ "logps/rejected": -166.0,
245
+ "loss": 0.6825,
246
+ "rewards/accuracies": 0.637499988079071,
247
+ "rewards/chosen": -0.0390625,
248
+ "rewards/margins": 0.01611328125,
249
+ "rewards/rejected": -0.05517578125,
250
+ "step": 140
251
+ },
252
+ {
253
+ "epoch": 0.16025641025641027,
254
+ "grad_norm": 6.375544406935963,
255
+ "learning_rate": 2.9673763677155654e-07,
256
+ "logits/chosen": -0.59375,
257
+ "logits/rejected": -0.5546875,
258
+ "logps/chosen": -175.0,
259
+ "logps/rejected": -174.0,
260
+ "loss": 0.6788,
261
+ "rewards/accuracies": 0.637499988079071,
262
+ "rewards/chosen": -0.053466796875,
263
+ "rewards/margins": 0.032470703125,
264
+ "rewards/rejected": -0.0859375,
265
+ "step": 150
266
+ },
267
+ {
268
+ "epoch": 0.17094017094017094,
269
+ "grad_norm": 6.0354745276000195,
270
+ "learning_rate": 2.9547489219129667e-07,
271
+ "logits/chosen": -0.44921875,
272
+ "logits/rejected": -0.41015625,
273
+ "logps/chosen": -168.0,
274
+ "logps/rejected": -173.0,
275
+ "loss": 0.6774,
276
+ "rewards/accuracies": 0.6000000238418579,
277
+ "rewards/chosen": -0.10302734375,
278
+ "rewards/margins": 0.030029296875,
279
+ "rewards/rejected": -0.1328125,
280
+ "step": 160
281
+ },
282
+ {
283
+ "epoch": 0.18162393162393162,
284
+ "grad_norm": 5.884420729829068,
285
+ "learning_rate": 2.9400965311490175e-07,
286
+ "logits/chosen": -0.478515625,
287
+ "logits/rejected": -0.4375,
288
+ "logps/chosen": -172.0,
289
+ "logps/rejected": -183.0,
290
+ "loss": 0.6727,
291
+ "rewards/accuracies": 0.6000000238418579,
292
+ "rewards/chosen": -0.11328125,
293
+ "rewards/margins": 0.03369140625,
294
+ "rewards/rejected": -0.1474609375,
295
+ "step": 170
296
+ },
297
+ {
298
+ "epoch": 0.19230769230769232,
299
+ "grad_norm": 7.410775468910537,
300
+ "learning_rate": 2.923439590891556e-07,
301
+ "logits/chosen": -0.419921875,
302
+ "logits/rejected": -0.314453125,
303
+ "logps/chosen": -174.0,
304
+ "logps/rejected": -181.0,
305
+ "loss": 0.6675,
306
+ "rewards/accuracies": 0.6625000238418579,
307
+ "rewards/chosen": -0.16015625,
308
+ "rewards/margins": 0.0673828125,
309
+ "rewards/rejected": -0.2275390625,
310
+ "step": 180
311
+ },
312
+ {
313
+ "epoch": 0.202991452991453,
314
+ "grad_norm": 6.515019365826615,
315
+ "learning_rate": 2.9048012868510087e-07,
316
+ "logits/chosen": -0.47265625,
317
+ "logits/rejected": -0.455078125,
318
+ "logps/chosen": -201.0,
319
+ "logps/rejected": -201.0,
320
+ "loss": 0.6656,
321
+ "rewards/accuracies": 0.6499999761581421,
322
+ "rewards/chosen": -0.26171875,
323
+ "rewards/margins": 0.048828125,
324
+ "rewards/rejected": -0.310546875,
325
+ "step": 190
326
+ },
327
+ {
328
+ "epoch": 0.21367521367521367,
329
+ "grad_norm": 6.905819636676354,
330
+ "learning_rate": 2.884207562706925e-07,
331
+ "logits/chosen": -0.53125,
332
+ "logits/rejected": -0.484375,
333
+ "logps/chosen": -197.0,
334
+ "logps/rejected": -202.0,
335
+ "loss": 0.6539,
336
+ "rewards/accuracies": 0.7124999761581421,
337
+ "rewards/chosen": -0.333984375,
338
+ "rewards/margins": 0.09130859375,
339
+ "rewards/rejected": -0.423828125,
340
+ "step": 200
341
+ },
342
+ {
343
+ "epoch": 0.21367521367521367,
344
+ "eval_logits/chosen": -0.515625,
345
+ "eval_logits/rejected": -0.50390625,
346
+ "eval_logps/chosen": -208.0,
347
+ "eval_logps/rejected": -211.0,
348
+ "eval_loss": 0.6502740979194641,
349
+ "eval_rewards/accuracies": 0.6693548560142517,
350
+ "eval_rewards/chosen": -0.33203125,
351
+ "eval_rewards/margins": 0.09375,
352
+ "eval_rewards/rejected": -0.42578125,
353
+ "eval_runtime": 104.1433,
354
+ "eval_samples_per_second": 18.83,
355
+ "eval_steps_per_second": 0.595,
356
+ "step": 200
357
+ },
358
+ {
359
+ "epoch": 0.22435897435897437,
360
+ "grad_norm": 7.284890119175988,
361
+ "learning_rate": 2.8616870839955444e-07,
362
+ "logits/chosen": -0.4453125,
363
+ "logits/rejected": -0.419921875,
364
+ "logps/chosen": -195.0,
365
+ "logps/rejected": -199.0,
366
+ "loss": 0.6428,
367
+ "rewards/accuracies": 0.574999988079071,
368
+ "rewards/chosen": -0.337890625,
369
+ "rewards/margins": 0.07763671875,
370
+ "rewards/rejected": -0.416015625,
371
+ "step": 210
372
+ },
373
+ {
374
+ "epoch": 0.23504273504273504,
375
+ "grad_norm": 7.8876506549073655,
376
+ "learning_rate": 2.837271198208662e-07,
377
+ "logits/chosen": -0.4921875,
378
+ "logits/rejected": -0.451171875,
379
+ "logps/chosen": -205.0,
380
+ "logps/rejected": -218.0,
381
+ "loss": 0.6373,
382
+ "rewards/accuracies": 0.737500011920929,
383
+ "rewards/chosen": -0.392578125,
384
+ "rewards/margins": 0.150390625,
385
+ "rewards/rejected": -0.54296875,
386
+ "step": 220
387
+ },
388
+ {
389
+ "epoch": 0.24572649572649571,
390
+ "grad_norm": 7.5346301038681815,
391
+ "learning_rate": 2.810993891159332e-07,
392
+ "logits/chosen": -0.451171875,
393
+ "logits/rejected": -0.451171875,
394
+ "logps/chosen": -198.0,
395
+ "logps/rejected": -216.0,
396
+ "loss": 0.6317,
397
+ "rewards/accuracies": 0.6625000238418579,
398
+ "rewards/chosen": -0.44140625,
399
+ "rewards/margins": 0.1435546875,
400
+ "rewards/rejected": -0.5859375,
401
+ "step": 230
402
+ },
403
+ {
404
+ "epoch": 0.2564102564102564,
405
+ "grad_norm": 9.738739769272485,
406
+ "learning_rate": 2.7828917396751476e-07,
407
+ "logits/chosen": -0.51171875,
408
+ "logits/rejected": -0.466796875,
409
+ "logps/chosen": -242.0,
410
+ "logps/rejected": -266.0,
411
+ "loss": 0.6113,
412
+ "rewards/accuracies": 0.737500011920929,
413
+ "rewards/chosen": -0.6171875,
414
+ "rewards/margins": 0.25,
415
+ "rewards/rejected": -0.8671875,
416
+ "step": 240
417
+ },
418
+ {
419
+ "epoch": 0.2670940170940171,
420
+ "grad_norm": 10.033627685900031,
421
+ "learning_rate": 2.7530038606849427e-07,
422
+ "logits/chosen": -0.60546875,
423
+ "logits/rejected": -0.609375,
424
+ "logps/chosen": -246.0,
425
+ "logps/rejected": -258.0,
426
+ "loss": 0.6031,
427
+ "rewards/accuracies": 0.7124999761581421,
428
+ "rewards/chosen": -0.73828125,
429
+ "rewards/margins": 0.1982421875,
430
+ "rewards/rejected": -0.9375,
431
+ "step": 250
432
+ },
433
+ {
434
+ "epoch": 0.2777777777777778,
435
+ "grad_norm": 11.941033651367999,
436
+ "learning_rate": 2.721371856769793e-07,
437
+ "logits/chosen": -0.52734375,
438
+ "logits/rejected": -0.49609375,
439
+ "logps/chosen": -249.0,
440
+ "logps/rejected": -282.0,
441
+ "loss": 0.601,
442
+ "rewards/accuracies": 0.6625000238418579,
443
+ "rewards/chosen": -0.8203125,
444
+ "rewards/margins": 0.3125,
445
+ "rewards/rejected": -1.1328125,
446
+ "step": 260
447
+ },
448
+ {
449
+ "epoch": 0.28846153846153844,
450
+ "grad_norm": 13.603960073243194,
451
+ "learning_rate": 2.6880397582540925e-07,
452
+ "logits/chosen": -0.47265625,
453
+ "logits/rejected": -0.474609375,
454
+ "logps/chosen": -258.0,
455
+ "logps/rejected": -324.0,
456
+ "loss": 0.5688,
457
+ "rewards/accuracies": 0.824999988079071,
458
+ "rewards/chosen": -1.0078125,
459
+ "rewards/margins": 0.55859375,
460
+ "rewards/rejected": -1.5625,
461
+ "step": 270
462
+ },
463
+ {
464
+ "epoch": 0.29914529914529914,
465
+ "grad_norm": 17.74830669348706,
466
+ "learning_rate": 2.65305396191733e-07,
467
+ "logits/chosen": -0.6015625,
468
+ "logits/rejected": -0.6015625,
469
+ "logps/chosen": -276.0,
470
+ "logps/rejected": -302.0,
471
+ "loss": 0.5952,
472
+ "rewards/accuracies": 0.6499999761581421,
473
+ "rewards/chosen": -1.2578125,
474
+ "rewards/margins": 0.30078125,
475
+ "rewards/rejected": -1.5546875,
476
+ "step": 280
477
+ },
478
+ {
479
+ "epoch": 0.30982905982905984,
480
+ "grad_norm": 24.594314426555332,
481
+ "learning_rate": 2.6164631664118594e-07,
482
+ "logits/chosen": -0.498046875,
483
+ "logits/rejected": -0.51953125,
484
+ "logps/chosen": -312.0,
485
+ "logps/rejected": -358.0,
486
+ "loss": 0.5688,
487
+ "rewards/accuracies": 0.75,
488
+ "rewards/chosen": -1.421875,
489
+ "rewards/margins": 0.478515625,
490
+ "rewards/rejected": -1.8984375,
491
+ "step": 290
492
+ },
493
+ {
494
+ "epoch": 0.32051282051282054,
495
+ "grad_norm": 14.533810491314245,
496
+ "learning_rate": 2.578318304476571e-07,
497
+ "logits/chosen": -0.53125,
498
+ "logits/rejected": -0.50390625,
499
+ "logps/chosen": -274.0,
500
+ "logps/rejected": -336.0,
501
+ "loss": 0.5817,
502
+ "rewards/accuracies": 0.737500011920929,
503
+ "rewards/chosen": -1.2578125,
504
+ "rewards/margins": 0.6171875,
505
+ "rewards/rejected": -1.875,
506
+ "step": 300
507
+ },
508
+ {
509
+ "epoch": 0.32051282051282054,
510
+ "eval_logits/chosen": -0.5546875,
511
+ "eval_logits/rejected": -0.5859375,
512
+ "eval_logps/chosen": -320.0,
513
+ "eval_logps/rejected": -360.0,
514
+ "eval_loss": 0.5601654052734375,
515
+ "eval_rewards/accuracies": 0.725806474685669,
516
+ "eval_rewards/chosen": -1.453125,
517
+ "eval_rewards/margins": 0.458984375,
518
+ "eval_rewards/rejected": -1.90625,
519
+ "eval_runtime": 101.0819,
520
+ "eval_samples_per_second": 19.4,
521
+ "eval_steps_per_second": 0.613,
522
+ "step": 300
523
+ },
524
+ {
525
+ "epoch": 0.3311965811965812,
526
+ "grad_norm": 18.918431944309962,
527
+ "learning_rate": 2.538672472040813e-07,
528
+ "logits/chosen": -0.52734375,
529
+ "logits/rejected": -0.51953125,
530
+ "logps/chosen": -284.0,
531
+ "logps/rejected": -312.0,
532
+ "loss": 0.5932,
533
+ "rewards/accuracies": 0.6875,
534
+ "rewards/chosen": -1.3671875,
535
+ "rewards/margins": 0.265625,
536
+ "rewards/rejected": -1.6328125,
537
+ "step": 310
538
+ },
539
+ {
540
+ "epoch": 0.3418803418803419,
541
+ "grad_norm": 14.267120841783509,
542
+ "learning_rate": 2.49758085431725e-07,
543
+ "logits/chosen": -0.51171875,
544
+ "logits/rejected": -0.53125,
545
+ "logps/chosen": -316.0,
546
+ "logps/rejected": -370.0,
547
+ "loss": 0.5631,
548
+ "rewards/accuracies": 0.7749999761581421,
549
+ "rewards/chosen": -1.4296875,
550
+ "rewards/margins": 0.61328125,
551
+ "rewards/rejected": -2.046875,
552
+ "step": 320
553
+ },
554
+ {
555
+ "epoch": 0.3525641025641026,
556
+ "grad_norm": 14.352005125875994,
557
+ "learning_rate": 2.455100648986533e-07,
558
+ "logits/chosen": -0.51171875,
559
+ "logits/rejected": -0.5,
560
+ "logps/chosen": -304.0,
561
+ "logps/rejected": -368.0,
562
+ "loss": 0.5696,
563
+ "rewards/accuracies": 0.7875000238418579,
564
+ "rewards/chosen": -1.4296875,
565
+ "rewards/margins": 0.6640625,
566
+ "rewards/rejected": -2.09375,
567
+ "step": 330
568
+ },
569
+ {
570
+ "epoch": 0.36324786324786323,
571
+ "grad_norm": 18.099939601206643,
572
+ "learning_rate": 2.4112909865807054e-07,
573
+ "logits/chosen": -0.45703125,
574
+ "logits/rejected": -0.46875,
575
+ "logps/chosen": -268.0,
576
+ "logps/rejected": -320.0,
577
+ "loss": 0.5542,
578
+ "rewards/accuracies": 0.699999988079071,
579
+ "rewards/chosen": -1.34375,
580
+ "rewards/margins": 0.4453125,
581
+ "rewards/rejected": -1.78125,
582
+ "step": 340
583
+ },
584
+ {
585
+ "epoch": 0.37393162393162394,
586
+ "grad_norm": 17.348733753912757,
587
+ "learning_rate": 2.366212848176164e-07,
588
+ "logits/chosen": -0.451171875,
589
+ "logits/rejected": -0.494140625,
590
+ "logps/chosen": -278.0,
591
+ "logps/rejected": -336.0,
592
+ "loss": 0.5355,
593
+ "rewards/accuracies": 0.737500011920929,
594
+ "rewards/chosen": -1.3359375,
595
+ "rewards/margins": 0.55078125,
596
+ "rewards/rejected": -1.890625,
597
+ "step": 350
598
+ },
599
+ {
600
+ "epoch": 0.38461538461538464,
601
+ "grad_norm": 19.652291890765426,
602
+ "learning_rate": 2.319928980510752e-07,
603
+ "logits/chosen": -0.44921875,
604
+ "logits/rejected": -0.45703125,
605
+ "logps/chosen": -324.0,
606
+ "logps/rejected": -382.0,
607
+ "loss": 0.509,
608
+ "rewards/accuracies": 0.7124999761581421,
609
+ "rewards/chosen": -1.6875,
610
+ "rewards/margins": 0.58203125,
611
+ "rewards/rejected": -2.265625,
612
+ "step": 360
613
+ },
614
+ {
615
+ "epoch": 0.3952991452991453,
616
+ "grad_norm": 20.132269692683796,
617
+ "learning_rate": 2.2725038086431226e-07,
618
+ "logits/chosen": -0.54296875,
619
+ "logits/rejected": -0.59375,
620
+ "logps/chosen": -364.0,
621
+ "logps/rejected": -452.0,
622
+ "loss": 0.5538,
623
+ "rewards/accuracies": 0.75,
624
+ "rewards/chosen": -1.9921875,
625
+ "rewards/margins": 0.8671875,
626
+ "rewards/rejected": -2.859375,
627
+ "step": 370
628
+ },
629
+ {
630
+ "epoch": 0.405982905982906,
631
+ "grad_norm": 18.69330687021763,
632
+ "learning_rate": 2.2240033462759625e-07,
633
+ "logits/chosen": -0.515625,
634
+ "logits/rejected": -0.5390625,
635
+ "logps/chosen": -358.0,
636
+ "logps/rejected": -438.0,
637
+ "loss": 0.5581,
638
+ "rewards/accuracies": 0.737500011920929,
639
+ "rewards/chosen": -1.8828125,
640
+ "rewards/margins": 0.8671875,
641
+ "rewards/rejected": -2.75,
642
+ "step": 380
643
+ },
644
+ {
645
+ "epoch": 0.4166666666666667,
646
+ "grad_norm": 14.86773297251084,
647
+ "learning_rate": 2.1744951038678905e-07,
648
+ "logits/chosen": -0.59375,
649
+ "logits/rejected": -0.5859375,
650
+ "logps/chosen": -352.0,
651
+ "logps/rejected": -422.0,
652
+ "loss": 0.5397,
653
+ "rewards/accuracies": 0.800000011920929,
654
+ "rewards/chosen": -1.796875,
655
+ "rewards/margins": 0.73046875,
656
+ "rewards/rejected": -2.53125,
657
+ "step": 390
658
+ },
659
+ {
660
+ "epoch": 0.42735042735042733,
661
+ "grad_norm": 16.11636675276784,
662
+ "learning_rate": 2.124047994661941e-07,
663
+ "logits/chosen": -0.462890625,
664
+ "logits/rejected": -0.51953125,
665
+ "logps/chosen": -334.0,
666
+ "logps/rejected": -410.0,
667
+ "loss": 0.5376,
668
+ "rewards/accuracies": 0.6875,
669
+ "rewards/chosen": -1.7265625,
670
+ "rewards/margins": 0.73828125,
671
+ "rewards/rejected": -2.453125,
672
+ "step": 400
673
+ },
674
+ {
675
+ "epoch": 0.42735042735042733,
676
+ "eval_logits/chosen": -0.5703125,
677
+ "eval_logits/rejected": -0.60546875,
678
+ "eval_logps/chosen": -374.0,
679
+ "eval_logps/rejected": -434.0,
680
+ "eval_loss": 0.5276495218276978,
681
+ "eval_rewards/accuracies": 0.7419354915618896,
682
+ "eval_rewards/chosen": -1.984375,
683
+ "eval_rewards/margins": 0.68359375,
684
+ "eval_rewards/rejected": -2.65625,
685
+ "eval_runtime": 102.5557,
686
+ "eval_samples_per_second": 19.121,
687
+ "eval_steps_per_second": 0.605,
688
+ "step": 400
689
+ },
690
+ {
691
+ "epoch": 0.43803418803418803,
692
+ "grad_norm": 20.3755079841164,
693
+ "learning_rate": 2.0727322387614335e-07,
694
+ "logits/chosen": -0.55859375,
695
+ "logits/rejected": -0.59765625,
696
+ "logps/chosen": -350.0,
697
+ "logps/rejected": -462.0,
698
+ "loss": 0.5063,
699
+ "rewards/accuracies": 0.824999988079071,
700
+ "rewards/chosen": -1.8671875,
701
+ "rewards/margins": 0.98828125,
702
+ "rewards/rejected": -2.859375,
703
+ "step": 410
704
+ },
705
+ {
706
+ "epoch": 0.44871794871794873,
707
+ "grad_norm": 16.255946037832317,
708
+ "learning_rate": 2.0206192653867534e-07,
709
+ "logits/chosen": -0.62109375,
710
+ "logits/rejected": -0.63671875,
711
+ "logps/chosen": -358.0,
712
+ "logps/rejected": -420.0,
713
+ "loss": 0.5411,
714
+ "rewards/accuracies": 0.675000011920929,
715
+ "rewards/chosen": -1.8828125,
716
+ "rewards/margins": 0.66015625,
717
+ "rewards/rejected": -2.546875,
718
+ "step": 420
719
+ },
720
+ {
721
+ "epoch": 0.4594017094017094,
722
+ "grad_norm": 26.473804402421884,
723
+ "learning_rate": 1.9677816134490948e-07,
724
+ "logits/chosen": -0.53515625,
725
+ "logits/rejected": -0.5625,
726
+ "logps/chosen": -370.0,
727
+ "logps/rejected": -442.0,
728
+ "loss": 0.5102,
729
+ "rewards/accuracies": 0.762499988079071,
730
+ "rewards/chosen": -2.015625,
731
+ "rewards/margins": 0.71484375,
732
+ "rewards/rejected": -2.734375,
733
+ "step": 430
734
+ },
735
+ {
736
+ "epoch": 0.4700854700854701,
737
+ "grad_norm": 15.37793095158405,
738
+ "learning_rate": 1.9142928305795636e-07,
739
+ "logits/chosen": -0.6328125,
740
+ "logits/rejected": -0.61328125,
741
+ "logps/chosen": -350.0,
742
+ "logps/rejected": -456.0,
743
+ "loss": 0.5329,
744
+ "rewards/accuracies": 0.762499988079071,
745
+ "rewards/chosen": -1.8203125,
746
+ "rewards/margins": 1.0234375,
747
+ "rewards/rejected": -2.84375,
748
+ "step": 440
749
+ },
750
+ {
751
+ "epoch": 0.4807692307692308,
752
+ "grad_norm": 20.55002112327052,
753
+ "learning_rate": 1.8602273707541883e-07,
754
+ "logits/chosen": -0.53515625,
755
+ "logits/rejected": -0.59375,
756
+ "logps/chosen": -292.0,
757
+ "logps/rejected": -398.0,
758
+ "loss": 0.5083,
759
+ "rewards/accuracies": 0.7124999761581421,
760
+ "rewards/chosen": -1.5234375,
761
+ "rewards/margins": 0.9765625,
762
+ "rewards/rejected": -2.5,
763
+ "step": 450
764
+ },
765
+ {
766
+ "epoch": 0.49145299145299143,
767
+ "grad_norm": 21.12850644767001,
768
+ "learning_rate": 1.8056604906573417e-07,
769
+ "logits/chosen": -0.59375,
770
+ "logits/rejected": -0.60546875,
771
+ "logps/chosen": -370.0,
772
+ "logps/rejected": -442.0,
773
+ "loss": 0.5182,
774
+ "rewards/accuracies": 0.7250000238418579,
775
+ "rewards/chosen": -2.0,
776
+ "rewards/margins": 0.796875,
777
+ "rewards/rejected": -2.796875,
778
+ "step": 460
779
+ },
780
+ {
781
+ "epoch": 0.5021367521367521,
782
+ "grad_norm": 29.0073411549113,
783
+ "learning_rate": 1.7506681449278225e-07,
784
+ "logits/chosen": -0.52734375,
785
+ "logits/rejected": -0.59375,
786
+ "logps/chosen": -358.0,
787
+ "logps/rejected": -460.0,
788
+ "loss": 0.5065,
789
+ "rewards/accuracies": 0.7250000238418579,
790
+ "rewards/chosen": -2.234375,
791
+ "rewards/margins": 0.88671875,
792
+ "rewards/rejected": -3.125,
793
+ "step": 470
794
+ },
795
+ {
796
+ "epoch": 0.5128205128205128,
797
+ "grad_norm": 19.23467713867048,
798
+ "learning_rate": 1.6953268804334256e-07,
799
+ "logits/chosen": -0.58984375,
800
+ "logits/rejected": -0.61328125,
801
+ "logps/chosen": -322.0,
802
+ "logps/rejected": -442.0,
803
+ "loss": 0.5089,
804
+ "rewards/accuracies": 0.762499988079071,
805
+ "rewards/chosen": -1.796875,
806
+ "rewards/margins": 1.03125,
807
+ "rewards/rejected": -2.828125,
808
+ "step": 480
809
+ },
810
+ {
811
+ "epoch": 0.5235042735042735,
812
+ "grad_norm": 16.595660350841946,
813
+ "learning_rate": 1.6397137297211436e-07,
814
+ "logits/chosen": -0.625,
815
+ "logits/rejected": -0.69921875,
816
+ "logps/chosen": -370.0,
817
+ "logps/rejected": -498.0,
818
+ "loss": 0.493,
819
+ "rewards/accuracies": 0.824999988079071,
820
+ "rewards/chosen": -2.03125,
821
+ "rewards/margins": 1.203125,
822
+ "rewards/rejected": -3.234375,
823
+ "step": 490
824
+ },
825
+ {
826
+ "epoch": 0.5341880341880342,
827
+ "grad_norm": 20.927345136231715,
828
+ "learning_rate": 1.5839061037913393e-07,
829
+ "logits/chosen": -0.6328125,
830
+ "logits/rejected": -0.6796875,
831
+ "logps/chosen": -372.0,
832
+ "logps/rejected": -466.0,
833
+ "loss": 0.5119,
834
+ "rewards/accuracies": 0.737500011920929,
835
+ "rewards/chosen": -1.96875,
836
+ "rewards/margins": 0.9140625,
837
+ "rewards/rejected": -2.890625,
838
+ "step": 500
839
+ },
840
+ {
841
+ "epoch": 0.5341880341880342,
842
+ "eval_logits/chosen": -0.59375,
843
+ "eval_logits/rejected": -0.63671875,
844
+ "eval_logps/chosen": -394.0,
845
+ "eval_logps/rejected": -472.0,
846
+ "eval_loss": 0.505980908870697,
847
+ "eval_rewards/accuracies": 0.7540322542190552,
848
+ "eval_rewards/chosen": -2.1875,
849
+ "eval_rewards/margins": 0.84375,
850
+ "eval_rewards/rejected": -3.03125,
851
+ "eval_runtime": 100.2109,
852
+ "eval_samples_per_second": 19.569,
853
+ "eval_steps_per_second": 0.619,
854
+ "step": 500
855
+ },
856
+ {
857
+ "epoch": 0.5448717948717948,
858
+ "grad_norm": 18.129746863835578,
859
+ "learning_rate": 1.5279816843451148e-07,
860
+ "logits/chosen": -0.55859375,
861
+ "logits/rejected": -0.59765625,
862
+ "logps/chosen": -394.0,
863
+ "logps/rejected": -496.0,
864
+ "loss": 0.4897,
865
+ "rewards/accuracies": 0.7749999761581421,
866
+ "rewards/chosen": -2.125,
867
+ "rewards/margins": 1.03125,
868
+ "rewards/rejected": -3.15625,
869
+ "step": 510
870
+ },
871
+ {
872
+ "epoch": 0.5555555555555556,
873
+ "grad_norm": 19.22223381123346,
874
+ "learning_rate": 1.4720183156548853e-07,
875
+ "logits/chosen": -0.6015625,
876
+ "logits/rejected": -0.65234375,
877
+ "logps/chosen": -358.0,
878
+ "logps/rejected": -456.0,
879
+ "loss": 0.518,
880
+ "rewards/accuracies": 0.75,
881
+ "rewards/chosen": -2.125,
882
+ "rewards/margins": 0.9140625,
883
+ "rewards/rejected": -3.03125,
884
+ "step": 520
885
+ },
886
+ {
887
+ "epoch": 0.5662393162393162,
888
+ "grad_norm": 22.855310205942654,
889
+ "learning_rate": 1.4160938962086611e-07,
890
+ "logits/chosen": -0.5546875,
891
+ "logits/rejected": -0.62890625,
892
+ "logps/chosen": -394.0,
893
+ "logps/rejected": -482.0,
894
+ "loss": 0.5381,
895
+ "rewards/accuracies": 0.699999988079071,
896
+ "rewards/chosen": -2.296875,
897
+ "rewards/margins": 0.91015625,
898
+ "rewards/rejected": -3.203125,
899
+ "step": 530
900
+ },
901
+ {
902
+ "epoch": 0.5769230769230769,
903
+ "grad_norm": 19.08554032740922,
904
+ "learning_rate": 1.3602862702788565e-07,
905
+ "logits/chosen": -0.453125,
906
+ "logits/rejected": -0.40625,
907
+ "logps/chosen": -278.0,
908
+ "logps/rejected": -354.0,
909
+ "loss": 0.5235,
910
+ "rewards/accuracies": 0.7250000238418579,
911
+ "rewards/chosen": -1.640625,
912
+ "rewards/margins": 0.68359375,
913
+ "rewards/rejected": -2.328125,
914
+ "step": 540
915
+ },
916
+ {
917
+ "epoch": 0.5876068376068376,
918
+ "grad_norm": 28.544577577873742,
919
+ "learning_rate": 1.3046731195665748e-07,
920
+ "logits/chosen": -0.65234375,
921
+ "logits/rejected": -0.62890625,
922
+ "logps/chosen": -332.0,
923
+ "logps/rejected": -452.0,
924
+ "loss": 0.5096,
925
+ "rewards/accuracies": 0.7749999761581421,
926
+ "rewards/chosen": -1.8671875,
927
+ "rewards/margins": 1.15625,
928
+ "rewards/rejected": -3.03125,
929
+ "step": 550
930
+ },
931
+ {
932
+ "epoch": 0.5982905982905983,
933
+ "grad_norm": 22.66887603925732,
934
+ "learning_rate": 1.2493318550721774e-07,
935
+ "logits/chosen": -0.68359375,
936
+ "logits/rejected": -0.68359375,
937
+ "logps/chosen": -298.0,
938
+ "logps/rejected": -386.0,
939
+ "loss": 0.483,
940
+ "rewards/accuracies": 0.762499988079071,
941
+ "rewards/chosen": -1.640625,
942
+ "rewards/margins": 0.78125,
943
+ "rewards/rejected": -2.421875,
944
+ "step": 560
945
+ },
946
+ {
947
+ "epoch": 0.6089743589743589,
948
+ "grad_norm": 31.07787330787091,
949
+ "learning_rate": 1.1943395093426584e-07,
950
+ "logits/chosen": -0.6953125,
951
+ "logits/rejected": -0.7109375,
952
+ "logps/chosen": -362.0,
953
+ "logps/rejected": -524.0,
954
+ "loss": 0.5388,
955
+ "rewards/accuracies": 0.762499988079071,
956
+ "rewards/chosen": -1.8515625,
957
+ "rewards/margins": 1.5234375,
958
+ "rewards/rejected": -3.375,
959
+ "step": 570
960
+ },
961
+ {
962
+ "epoch": 0.6196581196581197,
963
+ "grad_norm": 21.597913965377327,
964
+ "learning_rate": 1.1397726292458113e-07,
965
+ "logits/chosen": -0.45703125,
966
+ "logits/rejected": -0.498046875,
967
+ "logps/chosen": -370.0,
968
+ "logps/rejected": -492.0,
969
+ "loss": 0.5308,
970
+ "rewards/accuracies": 0.699999988079071,
971
+ "rewards/chosen": -2.140625,
972
+ "rewards/margins": 1.25,
973
+ "rewards/rejected": -3.390625,
974
+ "step": 580
975
+ },
976
+ {
977
+ "epoch": 0.6303418803418803,
978
+ "grad_norm": 18.471152729641705,
979
+ "learning_rate": 1.085707169420437e-07,
980
+ "logits/chosen": -0.546875,
981
+ "logits/rejected": -0.60546875,
982
+ "logps/chosen": -326.0,
983
+ "logps/rejected": -456.0,
984
+ "loss": 0.5017,
985
+ "rewards/accuracies": 0.862500011920929,
986
+ "rewards/chosen": -1.703125,
987
+ "rewards/margins": 1.1953125,
988
+ "rewards/rejected": -2.890625,
989
+ "step": 590
990
+ },
991
+ {
992
+ "epoch": 0.6410256410256411,
993
+ "grad_norm": 17.332734942907393,
994
+ "learning_rate": 1.0322183865509053e-07,
995
+ "logits/chosen": -0.55078125,
996
+ "logits/rejected": -0.578125,
997
+ "logps/chosen": -378.0,
998
+ "logps/rejected": -480.0,
999
+ "loss": 0.4721,
1000
+ "rewards/accuracies": 0.824999988079071,
1001
+ "rewards/chosen": -2.15625,
1002
+ "rewards/margins": 1.0859375,
1003
+ "rewards/rejected": -3.234375,
1004
+ "step": 600
1005
+ },
1006
+ {
1007
+ "epoch": 0.6410256410256411,
1008
+ "eval_logits/chosen": -0.62109375,
1009
+ "eval_logits/rejected": -0.66796875,
1010
+ "eval_logps/chosen": -386.0,
1011
+ "eval_logps/rejected": -472.0,
1012
+ "eval_loss": 0.4912582337856293,
1013
+ "eval_rewards/accuracies": 0.7540322542190552,
1014
+ "eval_rewards/chosen": -2.109375,
1015
+ "eval_rewards/margins": 0.92578125,
1016
+ "eval_rewards/rejected": -3.046875,
1017
+ "eval_runtime": 101.7619,
1018
+ "eval_samples_per_second": 19.27,
1019
+ "eval_steps_per_second": 0.609,
1020
+ "step": 600
1021
+ },
1022
+ {
1023
+ "epoch": 0.6517094017094017,
1024
+ "grad_norm": 21.737025901875963,
1025
+ "learning_rate": 9.793807346132465e-08,
1026
+ "logits/chosen": -0.671875,
1027
+ "logits/rejected": -0.73046875,
1028
+ "logps/chosen": -386.0,
1029
+ "logps/rejected": -520.0,
1030
+ "loss": 0.4818,
1031
+ "rewards/accuracies": 0.8125,
1032
+ "rewards/chosen": -2.046875,
1033
+ "rewards/margins": 1.4453125,
1034
+ "rewards/rejected": -3.5,
1035
+ "step": 610
1036
+ },
1037
+ {
1038
+ "epoch": 0.6623931623931624,
1039
+ "grad_norm": 22.482467028173893,
1040
+ "learning_rate": 9.272677612385667e-08,
1041
+ "logits/chosen": -0.58984375,
1042
+ "logits/rejected": -0.63671875,
1043
+ "logps/chosen": -346.0,
1044
+ "logps/rejected": -450.0,
1045
+ "loss": 0.5352,
1046
+ "rewards/accuracies": 0.75,
1047
+ "rewards/chosen": -1.9765625,
1048
+ "rewards/margins": 1.0390625,
1049
+ "rewards/rejected": -3.015625,
1050
+ "step": 620
1051
+ },
1052
+ {
1053
+ "epoch": 0.6730769230769231,
1054
+ "grad_norm": 19.020366926200907,
1055
+ "learning_rate": 8.75952005338059e-08,
1056
+ "logits/chosen": -0.63671875,
1057
+ "logits/rejected": -0.65625,
1058
+ "logps/chosen": -388.0,
1059
+ "logps/rejected": -480.0,
1060
+ "loss": 0.5084,
1061
+ "rewards/accuracies": 0.75,
1062
+ "rewards/chosen": -2.203125,
1063
+ "rewards/margins": 1.0390625,
1064
+ "rewards/rejected": -3.234375,
1065
+ "step": 630
1066
+ },
1067
+ {
1068
+ "epoch": 0.6837606837606838,
1069
+ "grad_norm": 19.944182815388004,
1070
+ "learning_rate": 8.255048961321087e-08,
1071
+ "logits/chosen": -0.51171875,
1072
+ "logits/rejected": -0.578125,
1073
+ "logps/chosen": -346.0,
1074
+ "logps/rejected": -448.0,
1075
+ "loss": 0.4934,
1076
+ "rewards/accuracies": 0.7875000238418579,
1077
+ "rewards/chosen": -1.875,
1078
+ "rewards/margins": 1.03125,
1079
+ "rewards/rejected": -2.90625,
1080
+ "step": 640
1081
+ },
1082
+ {
1083
+ "epoch": 0.6944444444444444,
1084
+ "grad_norm": 21.645476262079082,
1085
+ "learning_rate": 7.759966537240372e-08,
1086
+ "logits/chosen": -0.5625,
1087
+ "logits/rejected": -0.59765625,
1088
+ "logps/chosen": -338.0,
1089
+ "logps/rejected": -428.0,
1090
+ "loss": 0.4807,
1091
+ "rewards/accuracies": 0.699999988079071,
1092
+ "rewards/chosen": -1.8046875,
1093
+ "rewards/margins": 0.875,
1094
+ "rewards/rejected": -2.671875,
1095
+ "step": 650
1096
+ },
1097
+ {
1098
+ "epoch": 0.7051282051282052,
1099
+ "grad_norm": 16.0313495658248,
1100
+ "learning_rate": 7.274961913568772e-08,
1101
+ "logits/chosen": -0.640625,
1102
+ "logits/rejected": -0.71875,
1103
+ "logps/chosen": -380.0,
1104
+ "logps/rejected": -504.0,
1105
+ "loss": 0.4991,
1106
+ "rewards/accuracies": 0.7875000238418579,
1107
+ "rewards/chosen": -2.15625,
1108
+ "rewards/margins": 1.1875,
1109
+ "rewards/rejected": -3.359375,
1110
+ "step": 660
1111
+ },
1112
+ {
1113
+ "epoch": 0.7158119658119658,
1114
+ "grad_norm": 22.645278285314497,
1115
+ "learning_rate": 6.800710194892483e-08,
1116
+ "logits/chosen": -0.59765625,
1117
+ "logits/rejected": -0.62109375,
1118
+ "logps/chosen": -338.0,
1119
+ "logps/rejected": -444.0,
1120
+ "loss": 0.459,
1121
+ "rewards/accuracies": 0.75,
1122
+ "rewards/chosen": -1.875,
1123
+ "rewards/margins": 0.9921875,
1124
+ "rewards/rejected": -2.859375,
1125
+ "step": 670
1126
+ },
1127
+ {
1128
+ "epoch": 0.7264957264957265,
1129
+ "grad_norm": 31.044395342713987,
1130
+ "learning_rate": 6.33787151823836e-08,
1131
+ "logits/chosen": -0.67578125,
1132
+ "logits/rejected": -0.69140625,
1133
+ "logps/chosen": -402.0,
1134
+ "logps/rejected": -486.0,
1135
+ "loss": 0.4782,
1136
+ "rewards/accuracies": 0.75,
1137
+ "rewards/chosen": -2.453125,
1138
+ "rewards/margins": 0.890625,
1139
+ "rewards/rejected": -3.34375,
1140
+ "step": 680
1141
+ },
1142
+ {
1143
+ "epoch": 0.7371794871794872,
1144
+ "grad_norm": 21.636040060435434,
1145
+ "learning_rate": 5.887090134192947e-08,
1146
+ "logits/chosen": -0.6328125,
1147
+ "logits/rejected": -0.68359375,
1148
+ "logps/chosen": -364.0,
1149
+ "logps/rejected": -478.0,
1150
+ "loss": 0.5046,
1151
+ "rewards/accuracies": 0.762499988079071,
1152
+ "rewards/chosen": -2.15625,
1153
+ "rewards/margins": 1.1015625,
1154
+ "rewards/rejected": -3.25,
1155
+ "step": 690
1156
+ },
1157
+ {
1158
+ "epoch": 0.7478632478632479,
1159
+ "grad_norm": 23.459759628210772,
1160
+ "learning_rate": 5.448993510134669e-08,
1161
+ "logits/chosen": -0.640625,
1162
+ "logits/rejected": -0.703125,
1163
+ "logps/chosen": -370.0,
1164
+ "logps/rejected": -464.0,
1165
+ "loss": 0.5024,
1166
+ "rewards/accuracies": 0.699999988079071,
1167
+ "rewards/chosen": -2.125,
1168
+ "rewards/margins": 0.9140625,
1169
+ "rewards/rejected": -3.046875,
1170
+ "step": 700
1171
+ },
1172
+ {
1173
+ "epoch": 0.7478632478632479,
1174
+ "eval_logits/chosen": -0.640625,
1175
+ "eval_logits/rejected": -0.68359375,
1176
+ "eval_logps/chosen": -410.0,
1177
+ "eval_logps/rejected": -508.0,
1178
+ "eval_loss": 0.48169082403182983,
1179
+ "eval_rewards/accuracies": 0.7620967626571655,
1180
+ "eval_rewards/chosen": -2.34375,
1181
+ "eval_rewards/margins": 1.0546875,
1182
+ "eval_rewards/rejected": -3.40625,
1183
+ "eval_runtime": 101.5003,
1184
+ "eval_samples_per_second": 19.32,
1185
+ "eval_steps_per_second": 0.611,
1186
+ "step": 700
1187
+ },
1188
+ {
1189
+ "epoch": 0.7585470085470085,
1190
+ "grad_norm": 36.82125571876781,
1191
+ "learning_rate": 5.024191456827498e-08,
1192
+ "logits/chosen": -0.60546875,
1193
+ "logits/rejected": -0.62109375,
1194
+ "logps/chosen": -374.0,
1195
+ "logps/rejected": -524.0,
1196
+ "loss": 0.45,
1197
+ "rewards/accuracies": 0.800000011920929,
1198
+ "rewards/chosen": -2.109375,
1199
+ "rewards/margins": 1.421875,
1200
+ "rewards/rejected": -3.53125,
1201
+ "step": 710
1202
+ },
1203
+ {
1204
+ "epoch": 0.7692307692307693,
1205
+ "grad_norm": 27.66542286225275,
1206
+ "learning_rate": 4.613275279591866e-08,
1207
+ "logits/chosen": -0.5390625,
1208
+ "logits/rejected": -0.6015625,
1209
+ "logps/chosen": -376.0,
1210
+ "logps/rejected": -506.0,
1211
+ "loss": 0.4911,
1212
+ "rewards/accuracies": 0.800000011920929,
1213
+ "rewards/chosen": -2.203125,
1214
+ "rewards/margins": 1.2578125,
1215
+ "rewards/rejected": -3.46875,
1216
+ "step": 720
1217
+ },
1218
+ {
1219
+ "epoch": 0.7799145299145299,
1220
+ "grad_norm": 20.659365211408293,
1221
+ "learning_rate": 4.21681695523429e-08,
1222
+ "logits/chosen": -0.490234375,
1223
+ "logits/rejected": -0.5859375,
1224
+ "logps/chosen": -366.0,
1225
+ "logps/rejected": -520.0,
1226
+ "loss": 0.5169,
1227
+ "rewards/accuracies": 0.8500000238418579,
1228
+ "rewards/chosen": -2.125,
1229
+ "rewards/margins": 1.515625,
1230
+ "rewards/rejected": -3.640625,
1231
+ "step": 730
1232
+ },
1233
+ {
1234
+ "epoch": 0.7905982905982906,
1235
+ "grad_norm": 21.422600073203874,
1236
+ "learning_rate": 3.8353683358814044e-08,
1237
+ "logits/chosen": -0.6640625,
1238
+ "logits/rejected": -0.6953125,
1239
+ "logps/chosen": -404.0,
1240
+ "logps/rejected": -466.0,
1241
+ "loss": 0.4791,
1242
+ "rewards/accuracies": 0.675000011920929,
1243
+ "rewards/chosen": -2.40625,
1244
+ "rewards/margins": 0.75390625,
1245
+ "rewards/rejected": -3.15625,
1246
+ "step": 740
1247
+ },
1248
+ {
1249
+ "epoch": 0.8012820512820513,
1250
+ "grad_norm": 19.194254846262428,
1251
+ "learning_rate": 3.469460380826697e-08,
1252
+ "logits/chosen": -0.6328125,
1253
+ "logits/rejected": -0.66796875,
1254
+ "logps/chosen": -358.0,
1255
+ "logps/rejected": -490.0,
1256
+ "loss": 0.4905,
1257
+ "rewards/accuracies": 0.8374999761581421,
1258
+ "rewards/chosen": -2.015625,
1259
+ "rewards/margins": 1.3359375,
1260
+ "rewards/rejected": -3.359375,
1261
+ "step": 750
1262
+ },
1263
+ {
1264
+ "epoch": 0.811965811965812,
1265
+ "grad_norm": 25.458201673338998,
1266
+ "learning_rate": 3.1196024174590747e-08,
1267
+ "logits/chosen": -0.6796875,
1268
+ "logits/rejected": -0.71875,
1269
+ "logps/chosen": -366.0,
1270
+ "logps/rejected": -482.0,
1271
+ "loss": 0.4949,
1272
+ "rewards/accuracies": 0.7875000238418579,
1273
+ "rewards/chosen": -2.046875,
1274
+ "rewards/margins": 1.125,
1275
+ "rewards/rejected": -3.171875,
1276
+ "step": 760
1277
+ },
1278
+ {
1279
+ "epoch": 0.8226495726495726,
1280
+ "grad_norm": 20.734699064397134,
1281
+ "learning_rate": 2.7862814323020706e-08,
1282
+ "logits/chosen": -0.41796875,
1283
+ "logits/rejected": -0.4765625,
1284
+ "logps/chosen": -384.0,
1285
+ "logps/rejected": -516.0,
1286
+ "loss": 0.4961,
1287
+ "rewards/accuracies": 0.7749999761581421,
1288
+ "rewards/chosen": -2.265625,
1289
+ "rewards/margins": 1.265625,
1290
+ "rewards/rejected": -3.53125,
1291
+ "step": 770
1292
+ },
1293
+ {
1294
+ "epoch": 0.8333333333333334,
1295
+ "grad_norm": 18.04180153459493,
1296
+ "learning_rate": 2.4699613931505702e-08,
1297
+ "logits/chosen": -0.59765625,
1298
+ "logits/rejected": -0.66015625,
1299
+ "logps/chosen": -366.0,
1300
+ "logps/rejected": -488.0,
1301
+ "loss": 0.4956,
1302
+ "rewards/accuracies": 0.8125,
1303
+ "rewards/chosen": -1.96875,
1304
+ "rewards/margins": 1.28125,
1305
+ "rewards/rejected": -3.25,
1306
+ "step": 780
1307
+ },
1308
+ {
1309
+ "epoch": 0.844017094017094,
1310
+ "grad_norm": 32.45910697379444,
1311
+ "learning_rate": 2.1710826032485284e-08,
1312
+ "logits/chosen": -0.703125,
1313
+ "logits/rejected": -0.75,
1314
+ "logps/chosen": -364.0,
1315
+ "logps/rejected": -498.0,
1316
+ "loss": 0.4759,
1317
+ "rewards/accuracies": 0.7875000238418579,
1318
+ "rewards/chosen": -1.9921875,
1319
+ "rewards/margins": 1.2578125,
1320
+ "rewards/rejected": -3.25,
1321
+ "step": 790
1322
+ },
1323
+ {
1324
+ "epoch": 0.8547008547008547,
1325
+ "grad_norm": 23.332014663194055,
1326
+ "learning_rate": 1.8900610884066816e-08,
1327
+ "logits/chosen": -0.6875,
1328
+ "logits/rejected": -0.7265625,
1329
+ "logps/chosen": -344.0,
1330
+ "logps/rejected": -454.0,
1331
+ "loss": 0.5006,
1332
+ "rewards/accuracies": 0.762499988079071,
1333
+ "rewards/chosen": -1.90625,
1334
+ "rewards/margins": 0.99609375,
1335
+ "rewards/rejected": -2.90625,
1336
+ "step": 800
1337
+ },
1338
+ {
1339
+ "epoch": 0.8547008547008547,
1340
+ "eval_logits/chosen": -0.640625,
1341
+ "eval_logits/rejected": -0.6875,
1342
+ "eval_logps/chosen": -398.0,
1343
+ "eval_logps/rejected": -492.0,
1344
+ "eval_loss": 0.47812220454216003,
1345
+ "eval_rewards/accuracies": 0.7661290168762207,
1346
+ "eval_rewards/chosen": -2.21875,
1347
+ "eval_rewards/margins": 1.03125,
1348
+ "eval_rewards/rejected": -3.25,
1349
+ "eval_runtime": 102.4136,
1350
+ "eval_samples_per_second": 19.148,
1351
+ "eval_steps_per_second": 0.605,
1352
+ "step": 800
1353
+ },
1354
+ {
1355
+ "epoch": 0.8653846153846154,
1356
+ "grad_norm": 18.92181329955703,
1357
+ "learning_rate": 1.627288017913383e-08,
1358
+ "logits/chosen": -0.6171875,
1359
+ "logits/rejected": -0.59765625,
1360
+ "logps/chosen": -416.0,
1361
+ "logps/rejected": -568.0,
1362
+ "loss": 0.4802,
1363
+ "rewards/accuracies": 0.7250000238418579,
1364
+ "rewards/chosen": -2.453125,
1365
+ "rewards/margins": 1.546875,
1366
+ "rewards/rejected": -4.0,
1367
+ "step": 810
1368
+ },
1369
+ {
1370
+ "epoch": 0.8760683760683761,
1371
+ "grad_norm": 20.86644592519866,
1372
+ "learning_rate": 1.3831291600445571e-08,
1373
+ "logits/chosen": -0.578125,
1374
+ "logits/rejected": -0.640625,
1375
+ "logps/chosen": -320.0,
1376
+ "logps/rejected": -448.0,
1377
+ "loss": 0.4838,
1378
+ "rewards/accuracies": 0.75,
1379
+ "rewards/chosen": -1.7734375,
1380
+ "rewards/margins": 1.2421875,
1381
+ "rewards/rejected": -3.015625,
1382
+ "step": 820
1383
+ },
1384
+ {
1385
+ "epoch": 0.8867521367521367,
1386
+ "grad_norm": 28.147793595276735,
1387
+ "learning_rate": 1.1579243729307486e-08,
1388
+ "logits/chosen": -0.54296875,
1389
+ "logits/rejected": -0.57421875,
1390
+ "logps/chosen": -342.0,
1391
+ "logps/rejected": -482.0,
1392
+ "loss": 0.466,
1393
+ "rewards/accuracies": 0.8125,
1394
+ "rewards/chosen": -1.90625,
1395
+ "rewards/margins": 1.375,
1396
+ "rewards/rejected": -3.28125,
1397
+ "step": 830
1398
+ },
1399
+ {
1400
+ "epoch": 0.8974358974358975,
1401
+ "grad_norm": 27.32641526927714,
1402
+ "learning_rate": 9.519871314899092e-09,
1403
+ "logits/chosen": -0.64453125,
1404
+ "logits/rejected": -0.64453125,
1405
+ "logps/chosen": -384.0,
1406
+ "logps/rejected": -472.0,
1407
+ "loss": 0.523,
1408
+ "rewards/accuracies": 0.762499988079071,
1409
+ "rewards/chosen": -2.234375,
1410
+ "rewards/margins": 0.875,
1411
+ "rewards/rejected": -3.109375,
1412
+ "step": 840
1413
+ },
1414
+ {
1415
+ "epoch": 0.9081196581196581,
1416
+ "grad_norm": 24.25909713533937,
1417
+ "learning_rate": 7.656040910844358e-09,
1418
+ "logits/chosen": -0.61328125,
1419
+ "logits/rejected": -0.625,
1420
+ "logps/chosen": -386.0,
1421
+ "logps/rejected": -510.0,
1422
+ "loss": 0.4749,
1423
+ "rewards/accuracies": 0.8125,
1424
+ "rewards/chosen": -2.1875,
1425
+ "rewards/margins": 1.2734375,
1426
+ "rewards/rejected": -3.453125,
1427
+ "step": 850
1428
+ },
1429
+ {
1430
+ "epoch": 0.9188034188034188,
1431
+ "grad_norm": 17.801297109293913,
1432
+ "learning_rate": 5.990346885098235e-09,
1433
+ "logits/chosen": -0.66796875,
1434
+ "logits/rejected": -0.71875,
1435
+ "logps/chosen": -366.0,
1436
+ "logps/rejected": -492.0,
1437
+ "loss": 0.4755,
1438
+ "rewards/accuracies": 0.7749999761581421,
1439
+ "rewards/chosen": -1.9765625,
1440
+ "rewards/margins": 1.25,
1441
+ "rewards/rejected": -3.234375,
1442
+ "step": 860
1443
+ },
1444
+ {
1445
+ "epoch": 0.9294871794871795,
1446
+ "grad_norm": 17.682232799756985,
1447
+ "learning_rate": 4.525107808703349e-09,
1448
+ "logits/chosen": -0.63671875,
1449
+ "logits/rejected": -0.65625,
1450
+ "logps/chosen": -392.0,
1451
+ "logps/rejected": -520.0,
1452
+ "loss": 0.4814,
1453
+ "rewards/accuracies": 0.7875000238418579,
1454
+ "rewards/chosen": -2.265625,
1455
+ "rewards/margins": 1.265625,
1456
+ "rewards/rejected": -3.53125,
1457
+ "step": 870
1458
+ },
1459
+ {
1460
+ "epoch": 0.9401709401709402,
1461
+ "grad_norm": 18.928250676279468,
1462
+ "learning_rate": 3.2623632284434264e-09,
1463
+ "logits/chosen": -0.57421875,
1464
+ "logits/rejected": -0.64453125,
1465
+ "logps/chosen": -436.0,
1466
+ "logps/rejected": -568.0,
1467
+ "loss": 0.518,
1468
+ "rewards/accuracies": 0.737500011920929,
1469
+ "rewards/chosen": -2.578125,
1470
+ "rewards/margins": 1.3515625,
1471
+ "rewards/rejected": -3.921875,
1472
+ "step": 880
1473
+ },
1474
+ {
1475
+ "epoch": 0.9508547008547008,
1476
+ "grad_norm": 27.67410747954945,
1477
+ "learning_rate": 2.203870827886295e-09,
1478
+ "logits/chosen": -0.62890625,
1479
+ "logits/rejected": -0.671875,
1480
+ "logps/chosen": -400.0,
1481
+ "logps/rejected": -548.0,
1482
+ "loss": 0.4949,
1483
+ "rewards/accuracies": 0.7749999761581421,
1484
+ "rewards/chosen": -2.265625,
1485
+ "rewards/margins": 1.578125,
1486
+ "rewards/rejected": -3.84375,
1487
+ "step": 890
1488
+ },
1489
+ {
1490
+ "epoch": 0.9615384615384616,
1491
+ "grad_norm": 21.853464310555925,
1492
+ "learning_rate": 1.3511039807673208e-09,
1493
+ "logits/chosen": -0.625,
1494
+ "logits/rejected": -0.66796875,
1495
+ "logps/chosen": -414.0,
1496
+ "logps/rejected": -532.0,
1497
+ "loss": 0.4661,
1498
+ "rewards/accuracies": 0.824999988079071,
1499
+ "rewards/chosen": -2.328125,
1500
+ "rewards/margins": 1.2109375,
1501
+ "rewards/rejected": -3.546875,
1502
+ "step": 900
1503
+ },
1504
+ {
1505
+ "epoch": 0.9615384615384616,
1506
+ "eval_logits/chosen": -0.64453125,
1507
+ "eval_logits/rejected": -0.69140625,
1508
+ "eval_logps/chosen": -398.0,
1509
+ "eval_logps/rejected": -496.0,
1510
+ "eval_loss": 0.4771551191806793,
1511
+ "eval_rewards/accuracies": 0.7701612710952759,
1512
+ "eval_rewards/chosen": -2.234375,
1513
+ "eval_rewards/margins": 1.046875,
1514
+ "eval_rewards/rejected": -3.28125,
1515
+ "eval_runtime": 101.9594,
1516
+ "eval_samples_per_second": 19.233,
1517
+ "eval_steps_per_second": 0.608,
1518
+ "step": 900
1519
+ },
1520
+ {
1521
+ "epoch": 0.9722222222222222,
1522
+ "grad_norm": 44.172231527520715,
1523
+ "learning_rate": 7.052497001196367e-10,
1524
+ "logits/chosen": -0.63671875,
1525
+ "logits/rejected": -0.66796875,
1526
+ "logps/chosen": -396.0,
1527
+ "logps/rejected": -540.0,
1528
+ "loss": 0.5173,
1529
+ "rewards/accuracies": 0.75,
1530
+ "rewards/chosen": -2.359375,
1531
+ "rewards/margins": 1.4375,
1532
+ "rewards/rejected": -3.796875,
1533
+ "step": 910
1534
+ },
1535
+ {
1536
+ "epoch": 0.9829059829059829,
1537
+ "grad_norm": 24.37711330194768,
1538
+ "learning_rate": 2.672069860055359e-10,
1539
+ "logits/chosen": -0.640625,
1540
+ "logits/rejected": -0.71484375,
1541
+ "logps/chosen": -396.0,
1542
+ "logps/rejected": -656.0,
1543
+ "loss": 0.4902,
1544
+ "rewards/accuracies": 0.800000011920929,
1545
+ "rewards/chosen": -2.171875,
1546
+ "rewards/margins": 2.578125,
1547
+ "rewards/rejected": -4.75,
1548
+ "step": 920
1549
+ },
1550
+ {
1551
+ "epoch": 0.9935897435897436,
1552
+ "grad_norm": 27.894111829625263,
1553
+ "learning_rate": 3.758557414877961e-11,
1554
+ "logits/chosen": -0.63671875,
1555
+ "logits/rejected": -0.703125,
1556
+ "logps/chosen": -358.0,
1557
+ "logps/rejected": -460.0,
1558
+ "loss": 0.4718,
1559
+ "rewards/accuracies": 0.75,
1560
+ "rewards/chosen": -2.046875,
1561
+ "rewards/margins": 1.0,
1562
+ "rewards/rejected": -3.046875,
1563
+ "step": 930
1564
+ },
1565
+ {
1566
+ "epoch": 1.0,
1567
+ "step": 936,
1568
+ "total_flos": 0.0,
1569
+ "train_loss": 0.5544820035624708,
1570
+ "train_runtime": 9953.5203,
1571
+ "train_samples_per_second": 6.015,
1572
+ "train_steps_per_second": 0.094
1573
+ }
1574
+ ],
1575
+ "logging_steps": 10,
1576
+ "max_steps": 936,
1577
+ "num_input_tokens_seen": 0,
1578
+ "num_train_epochs": 1,
1579
+ "save_steps": 100,
1580
+ "stateful_callbacks": {
1581
+ "TrainerControl": {
1582
+ "args": {
1583
+ "should_epoch_stop": false,
1584
+ "should_evaluate": false,
1585
+ "should_log": false,
1586
+ "should_save": true,
1587
+ "should_training_stop": true
1588
+ },
1589
+ "attributes": {}
1590
+ }
1591
+ },
1592
+ "total_flos": 0.0,
1593
+ "train_batch_size": 4,
1594
+ "trial_name": null,
1595
+ "trial_params": null
1596
+ }