zakerytclarke commited on
Commit
3f93979
·
verified ·
1 Parent(s): 2d29e48

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +1 -1
README.md CHANGED
@@ -263,7 +263,7 @@ Teapot LLM is fine-tuned from [flan-t5-base](https://huggingface.co/google/flan-
263
  - [Hyperparameters] The model was trained with various learning rates and monitored to ensure task specific performance was learned without catastrophic forgetting.
264
 
265
  ### Evaluation
266
- TeapotLLM is focused on in-context reasoning tasks, and therefore most benchmarks are not suitable for evaluation
267
  #### Synthqa Evaluation
268
  [Synthqa](https://huggingface.co/datasets/teapotai/synthqa) is a dataset focused on in-context QnA and information extraction tasks. We use the validation set to benchmark TeapotLLM against other models of similar size. All benchmarks were run using a Google Colab Notebook running on CPU with High Ram. Teapot significantly outperforms models of similar size, with low latency CPU inference and improved accuracy.
269
 
 
263
  - [Hyperparameters] The model was trained with various learning rates and monitored to ensure task specific performance was learned without catastrophic forgetting.
264
 
265
  ### Evaluation
266
+ TeapotLLM is focused on in-context reasoning tasks, and therefore most benchmarks are not suitable for evaluation. We want TeapotLLM to be a practical tool for QnA and information extraction, so we have developed custom datasets to benchmark performance.
267
  #### Synthqa Evaluation
268
  [Synthqa](https://huggingface.co/datasets/teapotai/synthqa) is a dataset focused on in-context QnA and information extraction tasks. We use the validation set to benchmark TeapotLLM against other models of similar size. All benchmarks were run using a Google Colab Notebook running on CPU with High Ram. Teapot significantly outperforms models of similar size, with low latency CPU inference and improved accuracy.
269