Update README.md

README.md

@@ -1,40 +1,199 @@
 ---
-base_model: []
+license: cc-by-nc-4.0
+base_model:
+- Qwen/Qwen3-14B
+- google/siglip2-so400m-patch16-384
 library_name: transformers
 tags:
-- mergekit
-- merge
-
+- multimodal
+- conversational
+- ncsoft
+- ncai
+- varco
+pipeline_tag: image-text-to-text
+language:
+- en
+- ko
 ---
-# vv21_llava_qwen3_linear_250711_15
 
-This is a merge of pre-trained language models created using [mergekit](https://github.com/cg123/mergekit).
+# VARCO-VISION-2.0-14B
+
+## Introduction
+**VARCO-VISION-2.0** is a multimodal AI model capable of understanding both images and text to answer user queries. It supports multi-image inputs, enabling effective processing of complex content such as documents, tables, and charts. The model demonstrates strong comprehension in both Korean and English, with significantly improved text generation capabilities and a deeper understanding of Korean cultural context. Compared to its predecessor, performance has been notably enhanced across various benchmarks, and its usability in real-world scenarios—such as everyday Q&A and information summarization—has also improved.
+
+In addition to the 14B full-scale model, a lightweight 1.7B version is available for on-device use, making it accessible on personal devices such as smartphones and PCs. VARCO-VISION-2.0 is a powerful open-source AI model built for Korean users and is freely available for a wide range of applications.
+
+## 🚨News🎙️
+- 👀 We are going to release VARCO-VISION-2.0-1.7B-OCR soon!
+- 👀 We are going to release VARCO-VISION-2.0-1.7B soon!
+- 📰 2025-07-16: We released VARCO-VISION-2.0-14B at [link](https://huggingface.co/NCSOFT/VARCO-VISION-2.0-14B)
+- 📰 2025-07-16: We released GME-VARCO-VISION-Embedding at [link](https://huggingface.co/NCSOFT/GME-VARCO-VISION-Embedding)
+
+## Key Features
+- **Multi-image Understanding**: Newly added support for multi-image inputs enables the model to analyze multiple images simultaneously and make more holistic and context-aware decisions.
+- **Korean Language Specialization**: The model is further specialized for Korean, with a deeper understanding of Korean language, context, and culture. Korean text generation has been significantly improved, resulting in more natural, fluent, and accurate responses.
+- **OCR with Text Localization**: Unlike typical models that only recognize and generate text from images, VARCO-VISION-2.0 can also identify the position of the text and provide bounding boxes around it. This makes it especially useful for document understanding, signage interpretation, and structured visual data.
+- **Enhanced Safety**: Improved robustness and filtering to ensure safer handling of harmful or sexually explicit content.
+
+<div align="center">
+    <img src="./Gimbap_Example-1-20250709-032708.png" width="100%" />
+</div>
+
+## VARCO-VISION-2.0 Family
+| Model Name                 | Base Models (Vision / Language)                                                                                                               | HF Link                                                          |
+| :------------------------: | :-------------------------------------------------------------------------------------------------------------------------------------------: | :--------------------------------------------------------------: |
+| VARCO-VISION-2.0-1.7B      | [siglip2-so400m-patch16-384](https://huggingface.co/google/siglip2-so400m-patch16-384) / [Qwen3-1.7B](https://huggingface.co/Qwen/Qwen3-1.7B) | [link](https://huggingface.co/NCSOFT/VARCO-VISION-2.0-1.7B)      |
+| VARCO-VISION-2.0-14B       | [siglip2-so400m-patch16-384](https://huggingface.co/google/siglip2-so400m-patch16-384) / [Qwen3-14B ](https://huggingface.co/Qwen/Qwen3-14B)  | [link](https://huggingface.co/NCSOFT/VARCO-VISION-2.0-14B)       |
+| VARCO-VISION-2.0-1.7B-OCR  | [siglip2-so400m-patch16-384](https://huggingface.co/google/siglip2-so400m-patch16-384) / [Qwen3-1.7B](https://huggingface.co/Qwen/Qwen3-1.7B) | [link](https://huggingface.co/NCSOFT/VARCO-VISION-2.0-1.7B-OCR)  |
+| GME-VARCO-VISION-Embedding | [Qwen2-VL-7B-Instruct](https://huggingface.co/Qwen/Qwen2-VL-7B-Instruct)                                                                      | [link](https://huggingface.co/NCSOFT/GME-VARCO-VISION-Embedding) |
+
+## Model Architecture
+VARCO-VISION-2.0 follows the architecture of [LLaVA-OneVision](https://arxiv.org/abs/2408.03326).
 
-## Merge Details
-### Merge Method
+## Evaluation
+We adopted benchmark scores directly from [OpenVLM Leaderboard](https://huggingface.co/spaces/opencompass/open_vlm_leaderboard) where available, and conducted our own evaluations for benchmarks not included in OpenVLM Leaderboard, comparing results against various open-source models to provide a fair and comprehensive evaluation.
+Please note that for certain benchmarks involving LLM-based evaluation (e.g., LLaVABench), results may not be exactly reproducible due to variations in the underlying LLM behavior.
 
-This model was merged using the [Linear](https://arxiv.org/abs/2203.05482) merge method.
+### English Benchmark
+| Benchmark     | InternVL3-14B | Ovis2-16B | Qwen2.5-VL-7B |VARCO-VISION-2.0-14B |
+| :-----------: | :-----------: | :-------: | :-----------: |:------------------: |
+| MMStar        | **68.9**      | *67.2*    | 64.1          | 64.8                |
+| SEEDBench_IMG | 77.5          | *77.7*    | 77.0          | **78.3**            |
+| LLaVABench    | 84.4          | **93.0**  | *91.0*        | 90.0                |
+| OCRBench      | 877           | *879*     | **888**       | 863                 |
 
-### Models Merged
+### Korean Benchmark
+| Benchmark    | InternVL3-14B | Ovis2-16B | Qwen2.5-VL-7B | VARCO-VISION-2.0-14B |
+| :----------: | :-----------: | :-------: | :-----------: | :------------------: |
+| K-MMStar     | **64.9**      | 29.7      | 49.3          | *63.3*               |
+| K-SEED       | **78.2**      | 73.2      | 75.7          | *77.4*               |
+| K-LLaVABench | 80.9          | 86.3      | *94.1*        | **95.1**             |
+| K-DTCBench   | **87.9**      | 81.7      | *82.1*        | 79.6                 |
 
-The following models were included in the merge:
-* /home/work/.varco_mllm/checkpoints-v2d1/training/vv2d1-llava-qwen3-14b-st4-250708/checkpoint-1400_hf
-* /home/work/.varco_mllm/checkpoints-v2d1/training/vv2d1-llava-qwen3-14b-st4-250708/checkpoint-1548_hf
+### Korean Cultural Benchmark
+| Benchmark        | InternVL3-14B | Ovis2-16B | Qwen2.5-VL-7B | VARCO-VISION-2.0-14B |
+| :--------------: | :-----------: | :-------: | :-----------: | :------------------: |
+| K-Viscuit        | 71.7          | **77.0**  | 70.9          | *72.9*               |
+| PangeaBench (ko) | **77.2**      | *76.9*    | 76.6          | 75.2                 |
 
-### Configuration
+### Text-only Benchmark
+| Benchmark  | InternVL3-14B | Ovis2-16B | Qwen2.5-VL-7B | VARCO-VISION-2.0-14B |
+| :--------: | :-----------: | :-------: | :-----------: | :------------------: |
+| MMLU       | **78.5**      | *78.4*    |  4.6          | 77.7                 |
+| MT-Bench   | **8.93**      | 8.59      | 8.07          | *8.88*               |
+| KMMLU      | *51.4*        | 49.3      | 39.6          | **57.4**             |
+| KoMT-Bench | 7.01          | *7.91*    | 6.84          | **7.95**             |
+| LogicKor   | 7.00          | **7.94**  | 6.55          | *7.86*               |
 
-The following YAML configuration was used to produce this model:
+**Note**: Some models show unusually low performance on the MMLU benchmark. This is primarily due to their failure to correctly follow the expected output format when only few-shot exemplars are provided in the prompts. Please take this into consideration when interpreting the results.
 
-```yaml
-models:
-  - model: /home/work/.varco_mllm/checkpoints-v2d1/training/vv2d1-llava-qwen3-14b-st4-250708/checkpoint-1400_hf
-    parameters:
-      weight: 1.0
-  - model: /home/work/.varco_mllm/checkpoints-v2d1/training/vv2d1-llava-qwen3-14b-st4-250708/checkpoint-1548_hf
-    parameters:
-      weight: 4.0
-merge_method: linear
-dtype: float16
+### OCR Benchmark
+| Benchmark | PaddleOCR | VARCO-VISION-2.0-14B |
+| :-------: | :-------: | :------------------: |
+| CORD      | *91.4*    | **93.3**             |
+| ICDAR2013 | *92.0*    | **93.2**             |
+| ICDAR2015 | *73.7*    | **82.7**             |
 
+## Usage
+To use this model, we recommend installing `transformers` version **4.53.1 or higher**. While it may work with earlier versions, using **4.53.1 or above is strongly recommended**, especially to ensure optimal performance for the **multi-image feature**.
+
+The basic usage is **identical to** [LLaVA-OneVision](https://huggingface.co/docs/transformers/main/en/model_doc/llava_onevision#usage-example):
+
+```python
+import requests
+from PIL import Image
+import torch
+from transformers import AutoProcessor, LlavaOnevisionForConditionalGeneration
+
+model_name = "NCSOFT/VARCO-VISION-2.0-14B"
+model = LlavaOnevisionForConditionalGeneration.from_pretrained(
+    model_name,
+    torch_dtype=torch.float16,
+    attn_implementation="sdpa",
+    device_map="auto",
+)
+processor = AutoProcessor.from_pretrained(model_name)
+
+conversation_1 = [
+    {
+        "role": "user",
+        "content": [
+            {"type": "image", "url": "https://www.ilankelman.org/stopsigns/australia.jpg"},
+            {"type": "text", "text": "What is shown in this image?"},
+            ],
+    },
+    {
+        "role": "assistant",
+        "content": [
+            {"type": "text", "text": "There is a red stop sign in the image."},
+            ],
+    },
+    {
+        "role": "user",
+        "content": [
+            {"type": "image", "url": "http://images.cocodataset.org/val2017/000000039769.jpg"},
+            {"type": "text", "text": "What about this image? How many cats do you see?"},
+            ],
+    },
+]
+conversation_2 = [
+    {
+        "role": "user",
+        "content": [
+            {"type": "image", "url": "https://huggingface.co/microsoft/kosmos-2-patch14-224/resolve/main/snowman.jpg"},
+            {"type": "text", "text": "이 이미���에는 무엇이 보이나요?"},
+            ],
+    },
+]
+
+inputs = processor.apply_chat_template(
+    [conversation_1, conversation_2],
+    add_generation_prompt=True,
+    tokenize=True,
+    return_dict=True,
+    padding=True,
+    return_tensors="pt"
+).to(model.device, torch.float16)
+
+generate_ids = model.generate(**inputs, max_new_tokens=1024, do_sample=False)
+outputs = processor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)
+print(outputs)
+```
+The following shows the input required for using OCR with text localization, along with the corresponding output:
+```python
+# INPUT
+image_file = "./assets/ocr.jpg"
+raw_image = Image.open(image_file)
+conversation = [
+    {
+        "role": "user",
+        "content": [
+            {"type": "text", "text": "<ocr>"},
+            {"type": "image"},
+        ],
+    },
+]
 
+# OUTPUT
+"""
+<char>백범로</char><bbox>0.172, 0.266, 0.328, 0.341</bbox>
+<char>124번길</char><bbox>0.347, 0.266, 0.512, 0.341</bbox>
+<char>Baekbeom-ro</char><bbox>0.171, 0.337, 0.433, 0.392</bbox>
+<char>124</char><bbox>0.444, 0.341, 0.508, 0.392</bbox>
+<char>만수주공아파트</char><bbox>0.109, 0.531, 0.335, 0.601</bbox>
+<char>시흥</char><bbox>0.443, 0.518, 0.522, 0.581</bbox>
+<char>시청</char><bbox>0.711, 0.521, 0.811, 0.594</bbox>
+<char>Mansu</char><bbox>0.102, 0.601, 0.181, 0.648</bbox>
+<char>Jugong</char><bbox>0.186, 0.601, 0.273, 0.658</bbox>
+<char>Apt</char><bbox>0.28, 0.601, 0.327, 0.651</bbox>
+<char>42</char><bbox>0.377, 0.601, 0.416, 0.648</bbox>
+<char>Shieung</char><bbox>0.445, 0.578, 0.53, 0.625</bbox>
+<char>인천대공원</char><bbox>0.43, 0.621, 0.609, 0.684</bbox>
+<char>모래내시장역</char><bbox>0.651, 0.59, 0.873, 0.665</bbox>
+<char>IncheonGrand</char><bbox>0.432, 0.681, 0.561, 0.723</bbox>
+<char>Park</char><bbox>0.564, 0.681, 0.611, 0.723</bbox>
+"""
 ```
+<div align="center">
+    <img src="./ocr.jpg" width="100%" />
+</div>