Update README.md
Browse files
README.md
CHANGED
|
@@ -111,12 +111,23 @@ model-index:
|
|
| 111 |
source:
|
| 112 |
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=fblgit/UNA-SimpleSmaug-34b-v1beta
|
| 113 |
name: Open LLM Leaderboard
|
|
|
|
|
|
|
| 114 |
---
|
| 115 |
|
| 116 |
# UNA-SimpleSmaug-34b-v1beta
|
| 117 |
|
| 118 |
Scoring 04-February-2024 #1 34B model, outperforming its original base model Smaug-34B-v0.1 with `77.41` 😎
|
| 119 |
-
Oh, btw.. this one went thru SFT so the abacus inside Smaug is back to normal.. so you can further train/dpo him .. RESET
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 120 |
|
| 121 |

|
| 122 |
Applied UNA only on the Attention, not on the MLP's
|
|
@@ -132,7 +143,17 @@ Results: Improving mathematican and reasoning capabilities without degrading and
|
|
| 132 |
**And enjoy our ModelSimilarities tool detector** https://github.com/fblgit/model-similarity where we confirmed numerically the bloodties of the model.
|
| 133 |
## Evals
|
| 134 |
|
| 135 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 136 |
```
|
| 137 |
| Task |Version| Metric |Value |
|
| 138 |
|-------------|------:|--------|----------------:|
|
|
@@ -155,13 +176,4 @@ To abacusai for making Smaug-34B, the Bagel, and all the magic behind the base m
|
|
| 155 |
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
|
| 156 |
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_fblgit__UNA-SimpleSmaug-34b-v1beta)
|
| 157 |
|
| 158 |
-
| Metric |Value|
|
| 159 |
-
|---------------------------------|----:|
|
| 160 |
-
|Avg. |77.41|
|
| 161 |
-
|AI2 Reasoning Challenge (25-Shot)|74.57|
|
| 162 |
-
|HellaSwag (10-Shot) |86.74|
|
| 163 |
-
|MMLU (5-Shot) |76.68|
|
| 164 |
-
|TruthfulQA (0-shot) |70.17|
|
| 165 |
-
|Winogrande (5-shot) |83.82|
|
| 166 |
-
|GSM8k (5-shot) |72.48|
|
| 167 |
|
|
|
|
| 111 |
source:
|
| 112 |
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=fblgit/UNA-SimpleSmaug-34b-v1beta
|
| 113 |
name: Open LLM Leaderboard
|
| 114 |
+
|
| 115 |
+
|
| 116 |
---
|
| 117 |
|
| 118 |
# UNA-SimpleSmaug-34b-v1beta
|
| 119 |
|
| 120 |
Scoring 04-February-2024 #1 34B model, outperforming its original base model Smaug-34B-v0.1 with `77.41` 😎
|
| 121 |
+
Oh, btw.. this one went thru SFT so the abacus inside Smaug is back to normal.. so you can further train/dpo him .. RESET!..
|
| 122 |
+
|
| 123 |
+
*UPDATES* March : Stills undisputed 34B King
|
| 124 |
+
Smaug 70B stills undisputed 70B King
|
| 125 |
+
|
| 126 |
+
====
|
| 127 |
+
And people wonders.. why there is no UNA of Hermes or Smaug 70B? << i dont think is worth the time to spend on a model that is widely known for not being too useful, likely UNA can fix some of the internal mess..
|
| 128 |
+
for Hermes, we spoke chitchat quick a couple times but nothing solid, but we would like to make a reborn of excellent models using UNA, just liek we did with UNA-Dolphin where we saw
|
| 129 |
+
relevant performance is short time.
|
| 130 |
+
===
|
| 131 |
|
| 132 |

|
| 133 |
Applied UNA only on the Attention, not on the MLP's
|
|
|
|
| 143 |
**And enjoy our ModelSimilarities tool detector** https://github.com/fblgit/model-similarity where we confirmed numerically the bloodties of the model.
|
| 144 |
## Evals
|
| 145 |
|
| 146 |
+
|
| 147 |
+
| Metric |Value|
|
| 148 |
+
|---------------------------------|----:|
|
| 149 |
+
|Avg. |77.41|
|
| 150 |
+
|AI2 Reasoning Challenge (25-Shot)|74.57|
|
| 151 |
+
|HellaSwag (10-Shot) |86.74|
|
| 152 |
+
|MMLU (5-Shot) |76.68|
|
| 153 |
+
|TruthfulQA (0-shot) |70.17|
|
| 154 |
+
|Winogrande (5-shot) |83.82|
|
| 155 |
+
|GSM8k (5-shot) |72.48|
|
| 156 |
+
|
| 157 |
```
|
| 158 |
| Task |Version| Metric |Value |
|
| 159 |
|-------------|------:|--------|----------------:|
|
|
|
|
| 176 |
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
|
| 177 |
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_fblgit__UNA-SimpleSmaug-34b-v1beta)
|
| 178 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 179 |
|