Update README.md
Browse files
README.md
CHANGED
|
@@ -263,7 +263,7 @@ Teapot LLM is fine-tuned from [flan-t5-base](https://huggingface.co/google/flan-
|
|
| 263 |
- [Hyperparameters] The model was trained with various learning rates and monitored to ensure task specific performance was learned without catastrophic forgetting.
|
| 264 |
|
| 265 |
### Evaluation
|
| 266 |
-
TeapotLLM is focused on in-context reasoning tasks, and therefore most benchmarks are not suitable for evaluation
|
| 267 |
#### Synthqa Evaluation
|
| 268 |
[Synthqa](https://huggingface.co/datasets/teapotai/synthqa) is a dataset focused on in-context QnA and information extraction tasks. We use the validation set to benchmark TeapotLLM against other models of similar size. All benchmarks were run using a Google Colab Notebook running on CPU with High Ram. Teapot significantly outperforms models of similar size, with low latency CPU inference and improved accuracy.
|
| 269 |
|
|
|
|
| 263 |
- [Hyperparameters] The model was trained with various learning rates and monitored to ensure task specific performance was learned without catastrophic forgetting.
|
| 264 |
|
| 265 |
### Evaluation
|
| 266 |
+
TeapotLLM is focused on in-context reasoning tasks, and therefore most benchmarks are not suitable for evaluation. We want TeapotLLM to be a practical tool for QnA and information extraction, so we have developed custom datasets to benchmark performance.
|
| 267 |
#### Synthqa Evaluation
|
| 268 |
[Synthqa](https://huggingface.co/datasets/teapotai/synthqa) is a dataset focused on in-context QnA and information extraction tasks. We use the validation set to benchmark TeapotLLM against other models of similar size. All benchmarks were run using a Google Colab Notebook running on CPU with High Ram. Teapot significantly outperforms models of similar size, with low latency CPU inference and improved accuracy.
|
| 269 |
|