simplify the usage example in the README, point to other rad-dino version. update link to license
Browse files
README.md
CHANGED
|
@@ -1,6 +1,8 @@
|
|
| 1 |
---
|
| 2 |
-
library_name: transformers
|
| 3 |
license: other
|
|
|
|
|
|
|
|
|
|
| 4 |
---
|
| 5 |
|
| 6 |
# Model card for RAD-DINO
|
|
@@ -17,7 +19,7 @@ RAD-DINO-MAIRA-2 is the version of RAD-DINO used in [MAIRA-2: Grounded Radiology
|
|
| 17 |
|
| 18 |
- **Developed by:** Microsoft Health Futures
|
| 19 |
- **Model type:** Vision transformer
|
| 20 |
-
- **License:** MSRLA
|
| 21 |
- **Finetuned from model:** [`dinov2-base`](https://huggingface.co/facebook/dinov2-base)
|
| 22 |
|
| 23 |
## Uses
|
|
@@ -55,71 +57,13 @@ Underlying biases of the training datasets may not be well characterized.
|
|
| 55 |
|
| 56 |
## Getting started
|
| 57 |
|
| 58 |
-
Let us first write an auxiliary function to download a chest X-ray.
|
| 59 |
-
|
| 60 |
-
```python
|
| 61 |
-
>>> import requests
|
| 62 |
-
>>> from PIL import Image
|
| 63 |
-
>>> def download_sample_image() -> Image.Image:
|
| 64 |
-
... """Download chest X-ray with CC license."""
|
| 65 |
-
... base_url = "https://upload.wikimedia.org/wikipedia/commons"
|
| 66 |
-
... image_url = f"{base_url}/2/20/Chest_X-ray_in_influenza_and_Haemophilus_influenzae.jpg"
|
| 67 |
-
... headers = {"User-Agent": "RAD-DINO"}
|
| 68 |
-
... response = requests.get(image_url, headers=headers, stream=True)
|
| 69 |
-
... return Image.open(response.raw)
|
| 70 |
-
...
|
| 71 |
```
|
| 72 |
-
|
| 73 |
-
|
| 74 |
-
|
| 75 |
-
```python
|
| 76 |
-
>>> import torch
|
| 77 |
-
>>> from transformers import AutoModel
|
| 78 |
-
>>> from transformers import AutoImageProcessor
|
| 79 |
-
>>>
|
| 80 |
-
>>> # Download the model
|
| 81 |
-
>>> repo = "microsoft/rad-dino-maira-2"
|
| 82 |
-
>>> model = AutoModel.from_pretrained(repo)
|
| 83 |
-
>>>
|
| 84 |
-
>>> # The processor takes a PIL image, performs resizing, center-cropping, and
|
| 85 |
-
>>> # intensity normalization using stats from MIMIC-CXR, and returns a
|
| 86 |
-
>>> # dictionary with a PyTorch tensor ready for the encoder
|
| 87 |
-
>>> processor = AutoImageProcessor.from_pretrained(repo)
|
| 88 |
-
>>>
|
| 89 |
-
>>> # Download and preprocess a chest X-ray
|
| 90 |
-
>>> image = download_sample_image()
|
| 91 |
-
>>> image.size # (width, height)
|
| 92 |
-
(2765, 2505)
|
| 93 |
-
>>> inputs = processor(images=image, return_tensors="pt")
|
| 94 |
-
>>>
|
| 95 |
-
>>> # Encode the image!
|
| 96 |
-
>>> with torch.inference_mode():
|
| 97 |
-
>>> outputs = model(**inputs)
|
| 98 |
-
>>>
|
| 99 |
-
>>> # Look at the CLS embeddings
|
| 100 |
-
>>> cls_embeddings = outputs.pooler_output
|
| 101 |
-
>>> cls_embeddings.shape # (batch_size, num_channels)
|
| 102 |
-
torch.Size([1, 768])
|
| 103 |
```
|
| 104 |
|
| 105 |
-
|
| 106 |
-
We will use [`einops`](https://einops.rocks/) (install with `pip install einops`) for this.
|
| 107 |
-
|
| 108 |
-
```python
|
| 109 |
-
>>> def reshape_patch_embeddings(flat_tokens: torch.Tensor) -> torch.Tensor:
|
| 110 |
-
... """Reshape flat list of patch tokens into a nice grid."""
|
| 111 |
-
... from einops import rearrange
|
| 112 |
-
... image_size = processor.crop_size["height"]
|
| 113 |
-
... patch_size = model.config.patch_size
|
| 114 |
-
... embeddings_size = image_size // patch_size
|
| 115 |
-
... patches_grid = rearrange(flat_tokens, "b (h w) c -> b c h w", h=embeddings_size)
|
| 116 |
-
... return patches_grid
|
| 117 |
-
...
|
| 118 |
-
>>> flat_patch_embeddings = outputs.last_hidden_state[:, 1:] # first token is CLS
|
| 119 |
-
>>> reshaped_patch_embeddings = reshape_patch_embeddings(flat_patch_embeddings)
|
| 120 |
-
>>> reshaped_patch_embeddings.shape # (batch_size, num_channels, height, width)
|
| 121 |
-
torch.Size([1, 768, 37, 37])
|
| 122 |
-
```
|
| 123 |
|
| 124 |
## Training details
|
| 125 |
|
|
|
|
| 1 |
---
|
|
|
|
| 2 |
license: other
|
| 3 |
+
license_name: msrla
|
| 4 |
+
license_link: https://huggingface.co/microsoft/rad-dino-maira-2/blob/main/LICENSE
|
| 5 |
+
library_name: transformers
|
| 6 |
---
|
| 7 |
|
| 8 |
# Model card for RAD-DINO
|
|
|
|
| 19 |
|
| 20 |
- **Developed by:** Microsoft Health Futures
|
| 21 |
- **Model type:** Vision transformer
|
| 22 |
+
- **License:** [MSRLA](./LICENSE)
|
| 23 |
- **Finetuned from model:** [`dinov2-base`](https://huggingface.co/facebook/dinov2-base)
|
| 24 |
|
| 25 |
## Uses
|
|
|
|
| 57 |
|
| 58 |
## Getting started
|
| 59 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 60 |
```
|
| 61 |
+
from transformers import pipeline
|
| 62 |
+
pipe = pipeline(task="image-feature-extraction", model="microsoft/rad-dino-maira-2", pool=False)
|
| 63 |
+
patch_features = pipe("https://www.bhf.org.uk/-/media/images/information-support/tests/chest-x-ray/normal-chest-x-ray-620x400.jpg")
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 64 |
```
|
| 65 |
|
| 66 |
+
Refer to [RAD-DINO](https://huggingface.co/microsoft/rad-dino) for a more detailed example.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 67 |
|
| 68 |
## Training details
|
| 69 |
|