init commit

README.md

@@ -0,0 +1,99 @@
+---
+language: en
+datasets:
+- laion2b
+---
+
+# OpenFlamingo-3B (CLIP ViT-L/14, MPT-1B Instruct)
+
+[Blog post]() | [Code](https://github.com/mlfoundations/open_flamingo) | [Demo]()
+
+OpenFlamingo is an open source implementation of DeepMind's [Flamingo](https://www.deepmind.com/blog/tackling-multiple-tasks-with-a-single-visual-language-model) models. 
+This 3B-parameter model uses a [CLIP ViT-L/14](https://huggingface.co/openai/clip-vit-large-patch14) vision encoder and an instruction-tuned [MPT-1B](https://huggingface.co/mosaicml/mpt-1b-redpajama-200b-dolly) language model.
+
+## Model Details
+We follow the Flamingo modeling paradigm, outfitting the layers of a pretrained, frozen language model such that they cross-attend to visual features when decoding. Following Flamingo, we freeze the vision encoder and language model but train the connecting modules on web-scraped image-text sequences. Specifically, we use a mixture of [LAION-2B](https://arxiv.org/abs/2210.08402) and [Multimodal C4](https://arxiv.org/abs/2304.06939).  
+
+## Uses
+OpenFlamingo models process arbitrarily interleaved sequences of images and text to output text. This allows the models to accept in-context examples and undertake tasks like captioning, visual question answering, and image classification. 
+
+### Bias, Risks, and Limitations
+OpenFlamingo models inherit the risks of their parent models, especially the language model. As an open-source research effort, we highly value open, accessible, reproducible multimodal model research; however, it is crucial to be aware that these models are trained on web data, have not been finetuned for safety, and thus may produce unintended, inappropriate, unreliable, and/or inaccurate outputs. Please use caution before deploying OpenFlamingo models in real applications. We also hope that OpenFlamingo enables further safety and reliability research to address these issues.
+
+In an effort to mitigate current potential biases and harms, we have deployed a text content filter on model outputs in the OpenFlamingo demo. We continue to red-team the model to understand and improve its safety.
+
+## Evaluation
+<table>
+  <tr>
+    <th></th>
+    <th>0-shot</th>
+    <th>4-shot</th>
+    <th>8-shot</th>
+    <th>16-shot</th>
+    <th>32-shot</th>
+  </tr>
+  <tr>
+    <th>COCO (CIDEr)</th>
+    <td>74.4 (0.6)</td>
+    <td>82.7 (0.7)</td>
+    <td>87.8 (0.5)</td>
+    <td>91.9 (0.3)</td>
+    <td>94.8 (0.3)</td>
+  </tr>
+  <tr>
+    <th>VQAv2 (Accuracy)</th>
+    <td>43.6 (0.2)</td>
+    <td>45.7 (0.5)</td>
+    <td>46.1 (0.5)</td>
+    <td>45.3 (0.4)</td>
+    <td>44.3 (0.4)</td>
+  </tr>
+  <tr>
+    <th>Flickr-30K (CIDEr)</th>
+    <td>51.2 (0.2)</td>
+    <td>59.1 (0.3)</td>
+    <td>60.7 (0.6)</td>
+    <td>63.0 (0.4)</td>
+    <td>64.5 (1.3)</td>
+  </tr>
+  <tr>
+    <th>OK-VQA (Accuracy)</th>
+    <td>26.2 (0.3)</td>
+    <td>31.9 (0.2)</td>
+    <td>31.4 (0.4)</td>
+    <td>31.6 (0.3)</td>
+    <td>31.0 (0.1)</td>
+  </tr>
+  <tr>
+    <th>TextVQA (Accuracy)</th>
+    <td>23.1 (0.2)</td>
+    <td>28.1 (0.4)</td>
+    <td>29.1 (0.1)</td>
+    <td>29.1 (0.1)</td>
+    <td>28.5 (0.1)</td>
+  </tr>
+  <tr>
+    <th>Vizwiz (Accuracy)</th>
+    <td>18.0 (0.6)</td>
+    <td>22.0 (0.8)</td>
+    <td>28.8 (1.3)</td>
+    <td>35.5 (0.8)</td>
+    <td>41.3 (0.5)</td>
+  </tr>
+  <tr>
+    <th>ImageNet (Top-1 Accuracy)</th>
+    <td>-</td>
+    <td>-</td>
+    <td>-</td>
+    <td>-</td>
+    <td>-</td>
+  </tr>
+  <tr>
+    <th>Hateful Memes (ROC AUC)</th>
+    <td>-</td>
+    <td>-</td>
+    <td>-</td>
+    <td>-</td>
+    <td>-</td>
+  </tr>
+</table>

checkpoint.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9dc045d02933a46f012b52b96aa10dbaedec84594dd202ac4c3c6f2ece51dbe3
+size 4188086933