patrickvonplaten ybelkada commited on
Commit
ab4f8ab
·
verified ·
1 Parent(s): 9552e7b

Add instructions on how to run the model with transformers (#31)

Browse files

- Update README.md (f2b0fc04dc4a811841bab00bcc5986d4a83004b2)


Co-authored-by: Younes Belkada <[email protected]>

Files changed (1) hide show
  1. README.md +21 -0
README.md CHANGED
@@ -104,6 +104,27 @@ num1, num2):
104
  # return the sum
105
  ```
106
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
107
  ## Limitations
108
 
109
  The Codestral-22B-v0.1 does not have any moderation mechanisms. We're looking forward to engaging with the community on ways to
 
104
  # return the sum
105
  ```
106
 
107
+ ## Usage with transformers library
108
+
109
+ This model is also compatible with `transformers` library, first run `pip install -U transformers` then use the snippet below to quickly get started:
110
+
111
+ ```python
112
+ from transformers import AutoModelForCausalLM, AutoTokenizer
113
+
114
+ model_id = "mistralai/Codestral-22B-v0.1"
115
+ tokenizer = AutoTokenizer.from_pretrained(model_id)
116
+
117
+ model = AutoModelForCausalLM.from_pretrained(model_id)
118
+
119
+ text = "Hello my name is"
120
+ inputs = tokenizer(text, return_tensors="pt")
121
+
122
+ outputs = model.generate(**inputs, max_new_tokens=20)
123
+ print(tokenizer.decode(outputs[0], skip_special_tokens=True))
124
+ ```
125
+
126
+ By default, transformers will load the model in full precision. Therefore you might be interested to further reduce down the memory requirements to run the model through the optimizations we offer in HF ecosystem.
127
+
128
  ## Limitations
129
 
130
  The Codestral-22B-v0.1 does not have any moderation mechanisms. We're looking forward to engaging with the community on ways to