ViDoRe V3: a comprehensive evaluation of retrieval for enterprise use-cases

TL;DR

ILLUIN Technology is proud to release the ViDoRe V3 benchmark, designed and developed with contributions from NVIDIA. ViDoRe V3 is our latest benchmark, engineered to set a new industry gold standard for multi-modal, enterprise document retrieval evaluation. It addresses a critical challenge in production RAG systems: retrieving accurate information from complex, visually-rich documents.

ViDoRe V3 improves on existing RAG benchmarks by prioritizing enterprise relevance and rigorous data quality. Instead of relying on clean academic texts, the benchmark draws from 10 challenging, real-world datasets spanning diverse industrial domains, with 8 publicly released and 2 kept private. In addition, while previous benchmarks often rely on synthetically generated data, ViDoRe V3 features human-created and human-verified annotations.

This benchmark contains 26,000 pages and 3,099 queries translated into 6 languages. Each query is linked to retrieval ground truth data created and verified by human annotators: relevant pages, precise bounding box annotations for key elements, and a comprehensive reference answer.

Why we built ViDoRe V3

The document retrieval landscape is ever more diverse. New pipelines based on Visual Language Models are challenging traditional systems based on text-retrieval/generation models. With ViDoRe V1 and V2, we showed the first steps towards better evaluating VLM retrievers:

• ViDoRe V1 focused on extractive queries based on single pages,
• ViDoRe V2 extended the benchmark to more open-ended queries.

But corpora were still small compared to real world use cases and both relied heavily on synthetic generation. While these were steps in the right direction, previous benchmarks still left us with a fragmented picture. Corpora needed to be larger and more representative of enterprise data, queries more diverse, end-to-end evaluation was difficult, and human verification was missing.

Our core contributions

To address limitations of previous benchmarks, we focused on 3 main improvement pillars:

1. Enterprise-relevant Corpora: We assembled 10 diverse corpora, each focusing on a distinct, enterprise-relevant domain or task. For each domain, we curated 1,000+ pages from diverse, permissively licensed multimodal documents that mirror real-world enterprise retrieval challenges and complexity. 8 datasets are publicly released and 2 are private to prevent overfitting.
2. Human Verification of annotations: For each query, we provide human-annotated page-relevancy rankings, bounding boxes, and written answers to enable comprehensive retrieval/RAG evaluation.
3. Diverse queries: To systematically identify failure modes, queries span 7 types (e.g., multi-hop, numerical) and 3 formats (question, instruction, keyword). To assess cross-language capabilities, all queries are provided in 6 languages: English, French, Spanish, German, Italian, and Portuguese.

Public datasets

Private datasets

Two datasets will remain private and will be managed by the MTEB team (big thanks to them!) to ensure benchmark integrity and mitigate overfitting. This evaluation approach should provide a less biased method for assessing visual retriever models, leading to a more representative understanding of their true capabilities. To avoid revealing too much detail about these datasets, we are only disclosing the domain and the language of the documents.

The two private datasets cover:

1. Energy-related regulatory documents (English)
2. Telecom-related technical standard documents (English)

Query categories

We designed the ViDoRe V3 queries to reflect the diversity and complexity of real-world retrieval tasks. Each query is formatted as a question, instruction, or keyword and tagged with one or more of 7 query types.

To illustrate which combinations of query types tend to occur together and how common each one is, we visualize the distribution and cardinality of all combinations. Single-category queries are most common, but many queries blend multiple types, such as extractive questions requiring numerical comparisons.

We paid particular attention to ensuring queries are challenging for current retrieval systems across all domains. Most queries require information spread across multiple pages, forcing models to extract and synthesize content from the entire documents rather than relying on single-page matches.

A hybrid generation process: how to build a challenging benchmark

To build a benchmark that is robust, difficult, and high-quality, we developed a sophisticated hybrid process that balances human expertise with LLM-driven scaling. Our goal was to create realistic queries, so we began with a page-agnostic approach. Similarly to ViDoRe V2, instead of using a single page, queries were generated from high-level summaries of document sections. This prevents the tasks from being overly simple and ensures they mimic real-world user intent. This generation was achieved through both synthetic pipelines (including NVIDIA NeMo Data Designer with Qwen3-235B) for scale and expert human annotators for nuance and complexity.

With thousands of generated queries for thousands of corpus pages, finding the correct ground-truth answers required a massive annotation effort. We implemented a multi-stage funnel to scale this process. First, a VLM (Qwen2.5-32B) performed a high-recall loose filter to rapidly discard clearly irrelevant pages, limiting false negatives and focusing the annotators' effort. Following this pre-filtering, trained human annotators performed the critical work. They identified the true relevant pages and generated the final, detailed annotations, including page-level relevancy rankings, precise written answers, and ground-truth bounding boxes.

While perfect ground truth is an elusive goal for any dataset at this scale, we invested heavily in enforcing a multi-layered quality control framework. Our annotators had native-level language proficiency and all passed pre-production validation and pilot gates. Key tasks were completed by multiple annotators to ensure consensus, and data underwent both quality control and audit checks by experienced senior annotators. This layered approach was designed to make the ground truth and the benchmark tasks as reliable and realistic as possible.

As a final quality assurance step, we rigorously filtered the annotations. This involved checking for annotator consensus, performing manual review, and using Qwen2.5-VL-32B to confirm the presence of relevant information across the annotated pages. We then leveraged Qwen2.5-VL-32B one last time to merge the remaining outputs into one single golden answer.

A hard benchmark for current retrieval models

We evaluate a wide range of modern visual retrieval models on our benchmark using the MTEB framework. The results confirm that the benchmark is exceptionally challenging for current methods.

The best-performing models reach a score of 65% NDCG@10 on English datasets. When introducing multilingual documents and translated queries, performance degrades significantly, with the average score failing to reach 60% NDCG@10.

A deeper analysis of the results reveals several key patterns:

• Challenges with technical documents: Models struggle significantly when faced with the highly technical documents in our Industrial subset and our private Energy-EN set, particularly with interpreting dense schematics and complex graphs.
• Persistent multilingual challenges: Performance drops considerably on our French documents. For the Physics and Finance-FR splits, no model was able to reach 50% NDCG@10.
• Relative strength in computer science: Models demonstrate higher performance on the Computer Science split. We hypothesize this is a spill-over effect from the massive amount of coding data used to train modern VLMs, making them more knowledgeable about that domain.

The full, detailed evaluation results and a deeper analysis of the dataset's difficulty are available below. All the metrics reported, unless stated otherwise, are NDCG@10.

English evaluation results

Multilingual results

Query type difficulty analysis

We break down the score distribution by query type and task for nemo-retriever-colembed-3b. Model performance aligns well with the expected difficulty of each query type: open-ended (NDCG@10 = 0.438) and multi-hop queries (0.515) are the hardest to retrieve, while extractive (0.668) and boolean (0.657) are the easiest.

Benchmark scope

Coverage of enterprise documents: A key challenge in developing this benchmark was the limited availability of in-domain multi-modal documents. While significant effort was made to curate relevant documents, the corpora may not fully represent proprietary enterprise data in all contexts.

Language coverage: The benchmark is currently limited to French and English documents. While we attempted to curate relevant documents in additional languages, resource constraints prevented broader language coverage. To mitigate this limitation, queries were translated into multiple languages to enable evaluation of cross-lingual tasks.

Annotation quality: Achieving perfect annotation quality is challenging at this scale and task complexity/scope. We implemented a multi-layered quality-control framework incorporating both advanced LLMs/VLMs and senior human annotators throughout the pipeline to validate quality and minimize type 1/2 errors. Despite this rigorous validation process, some annotation errors may remain in the benchmark.

Usage

Evaluation

Here is a quick script on how to evaluate colqwen2.5-v0.2 on the new benchmark using MTEB (as of right now, not merged yet in the main branch) :

import mteb

benchmark = mteb.get_benchmark("ViDoRe(v3)")
model = mteb.get_model("vidore/colqwen2.5-v0.2")

results = mteb.evaluate(model=model, tasks=benchmark)

Sample visualization

Here is a simple script to visualize a query/answer pair, with bounding boxes plotted on the relevant pages.

from datasets import load_dataset

dataset_name = "vidore/vidore_v3_industrial"

dataset = {
    "queries": load_dataset(dataset_name, data_dir="queries", split="test"),
    "qrels": load_dataset(dataset_name, data_dir="qrels", split="test"),
    "corpus": load_dataset(dataset_name, data_dir="corpus", split="test")
}

query_sample = dataset["queries"][8]
print('Query:', query_sample['query'])
print("Answer:", query_sample['answer'])

> Query: What type of airflow is required to maintain ultra-clean environments in aerospace operations?
> Answer: Laminar airflow is required to maintain ultra-clean environments in aerospace operations.

related_qrels = dataset["qrels"].filter(lambda x: x['query_id'] == query_sample['query_id'])

import matplotlib.pyplot as plt
import matplotlib.patches as patches

def plot_bbox(image, bboxes):
    _, ax = plt.subplots(figsize=(18, 12))
    ax.imshow(image), ax.axis('off')
    for bbox in bboxes:
        rect = patches.Rectangle((bbox['x1'], bbox['y1']), bbox['x2'] - bbox['x1'], bbox['y2'] - bbox['y1'], linewidth=2, edgecolor='r', facecolor='none')
        ax.add_patch(rect)
    plt.show()

for qrel in related_qrels:
    plot_bbox(dataset["corpus"][qrel['corpus_id']]['image'], qrel['bounding_boxes'])

Acknowledgements

This work was granted access to the HPC resources of IDRIS (Jean Zay cluster)under the allocation AD011016393 made by GENCI. This project would not have been realized without the commitment of everyone involved—a debt we owe to all our annotators and colleagues.

Thanks also to the MTEB team for having collaborated with the private datasets.

Finally, thank you to those at NVIDIA who contributed to the design and development of this benchmark : Tom Balough, Gabriel Moreira, Bo Liu, Eric Tramel, Mengyao Xu, Radek Osmulski, Erin Potter, Hannah Brandon for their invaluable help and advice.

Links

• Dataset Collection: https://hf.co/collections/vidore/vidore-benchmark-v3
• HF Org 🤗: https://huggingface.co/vidore
• Paper 📄: Coming soon
• Codebase 💻: Coming soon