Question:
On which tasks did Visual-RFT appear fully trained in the experimental section?
A. Reasoning Grounding.
B. Image Classification.
C. Object Detection.
D. Open Vocabulary Object Detection.

Answer: AD

We validate the effectiveness of Visual-RFT on the following tasks.
In fine-grained image classification, the
model utilizes its advanced reasoning capabilities to ana-
lyze fine-grained categories with high precision. In the one-
shot setting with extremely limited data (e.g., around 100
samples), Visual-RFT boosts the accuracy with 24.3% over
the baseline, while SFT dropped by 4.3%. In few-shot ex-
periments, Visual-RFT also demonstrates exceptional per-
2
formance with minimal training data, showcasing supe-
rior few-shot learning capabilities compared to SFT. In
reasoning grounding, Visual-RFT excels in the LISA [11]
dataset, which heavily relies on reasoning, outperforming
specialized models like GroundedSAM [18]. Furthermore,
in open vocabulary object detection, Visual-RFT quickly
transfers recognition capabilities to new categories, includ-
ing rare categories in LVIS [5], showing strong generaliza-
tion. Specifically, the 2B model achieves mAP improve-
ments from 9.8 to 31.3 on new classes of COCO [15] and
from 2.7 to 20.7 on selected rare classes of LVIS [5]. These
diverse visual perception tasks not only highlight Visual-
RFT’s robust generalization capabilities in visual recogni-
tion but also underscore the crucial role of reinforcement
learning in enhancing visual perception and reasoning.

Visual-RFT: Visual Reinforcement Fine-Tuning
Ziyu Liu1,2*
Zeyi Sun1,2*
Yuhang Zang2B
Xiaoyi Dong2,3
Yuhang Cao2
Haodong Duan2
Dahua Lin2,3
Jiaqi Wang2B
1Shanghai Jiaotong University
2Shanghai Artificial Intelligence Laboratory
3The Chinese University of Hong Kong
{liuziyu77, szy2023}@sjtu.edu.cn, {zangyuhang, wangjiaqi}@pjlab.org.cn
https://github.com/Liuziyu77/Visual-RFT
Abstract
Reinforcement Fine-Tuning (RFT) in Large Reasoning
Models like OpenAI o1 learns from feedback on its answers,
which is especially useful in applications when fine-tuning
data is scarce. Recent open-source work like DeepSeek-
R1 demonstrates that reinforcement learning with verifi-
able reward is one key direction in reproducing o1. While
the R1-style model has demonstrated success in language
models, its application in multi-modal domains remains
under-explored. This work introduces Visual Reinforcement
Fine-Tuning (Visual-RFT), which further extends the appli-
cation areas of RFT on visual tasks. Specifically, Visual-
RFT first uses Large Vision-Language Models (LVLMs) to
generate multiple responses containing reasoning tokens
and final answers for each input, and then uses our pro-
posed visual perception verifiable reward functions to up-
date the model via the policy optimization algorithm such
as Group Relative Policy Optimization (GRPO). We design
different verifiable reward functions for different percep-
tion tasks, such as the Intersection over Union (IoU) re-
ward for object detection.
Experimental results on fine-
grained image classification, few-shot object detection, rea-
soning grounding, as well as open-vocabulary object de-
tection benchmarks show the competitive performance and
advanced generalization ability of Visual-RFT compared
with Supervised Fine-tuning (SFT). For example, Visual-
RFT improves accuracy by 24.3% over the baseline in one-
shot fine-grained image classification with around 100 sam-
ples.
In few-shot object detection, Visual-RFT also ex-
ceeds the baseline by 21.9 on COCO’s two-shot setting
and 15.4 on LVIS. Our Visual-RFT represents a paradigm
shift in fine-tuning LVLMs, offering a data-efficient, reward-
driven approach that enhances reasoning and adaptability
for domain-specific tasks.
∗Equal contribution. B Corresponding author.
mAP
mouse
fork
hot
dog
cat
airplane
suitcase
parking
meter
sandwich
train
hair drier
toilet
toaster
snowboard
frisbee
bear
(a) OV Results on COCO
mAP
casserole
die
egg role
futon
garbage
handsaw
hippopotamus
kitchen
table
mallet
omelet
shot glass
stepladder
suger bowl
(b) OV Results on LVIS
mAP
bus
train
fire hygrant
stop sign
cat
dog
bed
toilet
(c) Few-shot Results on COCO
(d) Reasoning Grounding on LISA
(e) CLS on Fine-Grained Datasets
20.0
22.5
25.0
27.5
30.0
32.5
35.0
37.5
40.0
mIoU_test
mIoU_val
gIoU_test
Reasoning Grounding on LISA
30
40
50
60
70
80
90
100
4-shot Fine-Grained Classification
Qwen2-VL-2B
Qwen2-VL-2B + SFT
Qwen2-VL-2B + Visual-RFT
Qwen2-VL-2B
Qwen2-VL-2B + SFT
Qwen2-VL-2B + Visual-RFT
Qwen2-VL-2B
Qwen2-VL-2B + SFT
Qwen2-VL-2B + Visual-RFT
Metrics
Average Flower102
Pets37
Aircraft
Cars
Categories
+10.7
+4.2
+9.1
+25.9
+16.6
+19.7
+28.9
+38.5
Accuracy
Scores
Figure 1. Our Visual Reinforcement Fine-Tuning (Visual-RFT)
performs better than previous Supervised Fine-Tuning (SFT) on a
variety of tasks, such as Open Vocabulary(OV)/Few-shot Detec-
tion, Reasoning Grounding, and Fine-grained Classification.
1. Introduction
Large Reasoning Models (LRMs) such as OpenAI o1 [7]
represent frontier AI models designed to spend more time
“thinking” before answering, and achieving excellent rea-
soning abilities. One impressive capability of OpenAI o1
is Reinforcement Fine-Tuning (RFT) 1, which efficiently
fine-tune the model with merely dozens to thousands of
samples to excel at domain-specific tasks. Although the
implementation details of o1 are not publicly available, re-
cent open-source studies like DeepSeek R1 [4] reveal one
promising direction in reproducing o1 is Verifiable Re-
wards [4, 12, 37]: the reward score in reinforcement learn-
ing is directly determined by pre-defined rules, rather than
predicted by the separate reward model [17, 26, 45] trained
on preference data.
A primary distinction between the RFT and Previous Su-
pervised Fine-Tuning (SFT) lies in data efficiency. Previous
1https://openai.com/form/rft-research-program
1
arXiv:2503.01785v1  [cs.CV]  3 Mar 2025
Data-driven Fine-tuning
Question：
This is an image containing
a plant. Please identify the
species of the plant based
on the image.
Verifiable Reward
�=  �,  �� �����= ������
�,  ���������
Completions
Policy Optimization
Reinforcement
Finetuning
Prompts
Detection
Classification
Open Vocabulary Detection
Few-shot Detection
Reasoning Grounding
Fine-Grained Few-shot CLS
 its bright yellow flowers and small, greenish-brown
buds.The petals are often orange or red at the
base, and the flowers are often clustered together.
The leaves are green and have a slightly wavy
edge.</think><answer>wallflower</answer>
Reasoning:<think>The image appears to be
of a wallflower, which is characterized by
Question：Output the bounding box in the image corresponding
to the instruction: the pokeymon that can perform Thunderbolt.
Output the thinking process in <think> </think> and your grouding
b o x .  F o l l o w i n g  " < t h i n k >  t h i n k i n g  p r o c e s s  < / t h i n k >
<answer>(x1,y1),(x2,y2)</answer>)" format.
Thunderbolt is a move that can be used by several Pokémon. In
the image, there are several Pokémon, including Pikachu, which is
known for its Thunderbolt move. Pikachu is located in the bottom
right corner of the image.</think>
<answer>(696,300),(999,999)</answer>
Reasoning:<think>The instruction is to find the bounding
box of the Pokémon that can perform Thunderbolt.
(c) Applications of Visual-RFT
(b)     Visual Reinforcement FineTuning (Visual-RFT)
(a) Visual Instruction Tuning
LVLMs
Limited Data (10-1000)
Large-scale dataset
CLS Reward
IoU Reward
�=  �(���),  �� �����
�,  ���������
Wild Data:
Domain-specific Data:
Reasoning
Input
Output
Input
Output
LVLMs
Prompts
Limited Data (10-1000)
General Data:
Grounding
Figure 2. Overview of Visual-RFT. Compared to the (a) Visual Instruction Tuning that is data-hungry, (b) our Visual Reinforcement
Fine-Tuning (Visual-RFT) is more data efficient with limited data. (c) We successfully empower Large Vision-Language Models (LVLMs)
with RFT on a series of multi-modal tasks, and present examples of the model’s reasoning process at the bottom.
SFT paradigm (see Fig. 2 (a)) directly imitates the “ground-
truth” answers provided in the high-quality, curated data,
thus relying on a large amount of training data. By contrast,
RFT evaluates the model’s responses and adjusts based on
whether they’re correct, helping it learn through trial and er-
ror. Thus, RFT is particularly useful in domains where data
is scarce [7, 24]. However, a previous common sense is that
RFT is applied merely in tasks like scientific (e.g., mathe-
matics) and code generation. That’s because math and cod-
ing exhibit clear and objective final answers or test cases,
making their rewards relatively straightforward to verify. In
this paper, we demonstrate that RFT can be applied beyond
math and code domains to visual perception tasks. Specif-
ically, we introduce Visual Reinforcement Fine-Tuning
(Visual-RFT), which successfully extends RFT to em-
power Large Vision-Language Models (LVLMs) in various
multi-modal tasks (see Fig. 1), such as few-shot classifica-
tion and open-vocabulary object detection.
To extend RFT on visual tasks, we present the implemen-
tation details of Visual-RFT in Fig. 2 (b). For each input,
Visual-RFT uses Large Vision-Language Models (LVLMs)
to generate multiple responses (trajectories) that contain
the reasoning tokens and final answers. Crucially, we de-
fine task-specific, rule-based verifiable reward functions to
guide policy optimization, such as GRPO [31], in updat-
ing the model. For instance, we propose the Intersection
over Union (IoU) reward for the object detection task. Our
Visual-RFT contrasts with SFT, which relies on memoriz-
ing correct answers. Instead, our approach explores dif-
ferent possible solutions and learns to optimize for a de-
sired outcome defined by our verified reward function. It’s
about discovering what works best, not just mimicking pre-
defined answers. Our approach shifts the training paradigm
from data scaling in SFT to the strategic design of variable
reward functions tailored to specific multi-modal tasks. As
shown in Fig. 2 (c), the synergistic combination of verifi-
able rewards and visual perception abilities (e.g., detec-
tion, grounding, classification) allows our model to achieve
rapid and data-efficient mastery of new concepts, facilitated
by a detailed reasoning process.
We validate the effectiveness of Visual-RFT on the fol-
lowing tasks.
In fine-grained image classification, the
model utilizes its advanced reasoning capabilities to ana-
lyze fine-grained categories with high precision. In the one-
shot setting with extremely limited data (e.g., around 100
samples), Visual-RFT boosts the accuracy with 24.3% over
the baseline, while SFT dropped by 4.3%. In few-shot ex-
periments, Visual-RFT also demonstrates exceptional per-
2
Question: the part
of the vehicle that
can be opened?
Reference
Model
Policy Model
···
Reasoning:
<think> The vehicle has a door that can be opened. The door is located on the right
side of the vehicle, near the top.</think>
Policy Gradient
Optimization
Question: What
spices is the flower
in the image?
Reasoning
<think> The image appears to be of a columbine flower. The flower has five light yello
w petals and a yellow center, which are characteristic features of this species. The flo
wer also has a purple or lavender color on the petals, which is typical for this plant. Th
e presence of a small insect on the flower is also a common feature of columbine flower
s.</think> <answer> columbine </answer>
Verifiable
Reward
����=  �,  �� �����= ������
�,  ���������
IoU Reward & CLS Reward
����=  �(���),  �� �����
�,  ���������
Figure 3. Framework of Visual-RFT. Given the question and visual image inputs, the policy model generates multiple responses contain-
ing reasoning steps. Then the verifiable reward such as IoU reward and CLS reward is used with the policy gradient optimization algorithm
to update the policy model.
formance with minimal training data, showcasing supe-
rior few-shot learning capabilities compared to SFT. In
reasoning grounding, Visual-RFT excels in the LISA [11]
dataset, which heavily relies on reasoning, outperforming
specialized models like GroundedSAM [18]. Furthermore,
in open vocabulary object detection, Visual-RFT quickly
transfers recognition capabilities to new categories, includ-
ing rare categories in LVIS [5], showing strong generaliza-
tion. Specifically, the 2B model achieves mAP improve-
ments from 9.8 to 31.3 on new classes of COCO [15] and
from 2.7 to 20.7 on selected rare classes of LVIS [5]. These
diverse visual perception tasks not only highlight Visual-
RFT’s robust generalization capabilities in visual recogni-
tion but also underscore the crucial role of reinforcement
learning in enhancing visual perception and reasoning.
In summary, our key contributions are as follows:
(1)
We
introduce
Visual
Reinforcement
Fine-tuning
(Visual-RFT), which extends reinforcement learning with
verifiable rewards on visual perception tasks that are effec-
tive with limited data for fine-tuning.
(2) We design different verifiable rewards for different vi-
sual tasks that enable efficient, high-quality reward compu-
tation at a negligible cost. This allows the seamless transfer
of DeepSeek R1’s style reinforcement learning to LVLMs.
(3) We conduct extensive experiments on various visual per-
ception tasks, including fine-grained image classification,
few-shot object detection, reasoning grounding, and open
vocabulary object detection. On all the settings, Visual-
RFT achieves remarkable performance improvements, sig-
nificantly surpassing the supervised fine-tuning baselines.
(4) We fully open-source the training code, training data,
and evaluation scripts on Github to facilitate further re-
search.
2. Related Work
Large Vision Language Models (LVLMs) like GPT-
4o [23] achieves excellent visual understanding ability by
integrating both visual and textual data.
This integra-
tion enhances the models’ ability to understand complex
multi-modal inputs and enables more advanced AI sys-
tems [13, 16, 38, 47] capable of processing and respond-
ing to both images and text.
Generally, the training of
LVLMs involves two steps: (a) pre-training and (b) post-
training which contains supervised fine-tuning and rein-
forcement learning. Post-training is crucial in improving
the model’s response quality, instruction following, and rea-
soning abilities. While there has been significant research
on using reinforcement learning to enhance LLMs during
post-training [1, 3, 25, 28, 32, 33, 36, 40, 44, 52, 53], the
progress for LVLMs has been slower. In this paper, we pro-
pose Visual-RFT, which used GRPO-based reinforcement
algorithms and verifiable reward during the post-training
phase to enhance the model’s visual perception and reason-
ing capabilities.
Reinforcement Learning Recently, with the emergence of
reasoning models like OpenAI’s o1 [7], the research fo-
cus in Large Language Models (LLMs) has increasingly
shifted towards enhancing the models’ reasoning capabil-
ities through reinforcement learning (RL) techniques. Stud-
3
Table 1. Prompts used to construct the dataset. We have listed
the detection prompt and classification prompt separately.
Detection Prompt: Detect all objects belonging to the category
’{category}’ in the image, and provide the bounding boxes (be-
tween 0 and 1000, integer) and confidence (between 0 and 1,
with two decimal places). If no object belonging to the cate-
gory ’{category}’ in the image, return ’No Objects’. Output
the thinking process in <think> </think> and final answer in
<answer> </answer> tags. The output answer format should
be as follows: <think> ... </think><answer>[’Position’: [x1,
y1, x2, y2], ’Confidence’: number, ...]</answer> Please strictly
follow the format.
Classification Prompt: This is an image containing a plant.
Please identify the species of the plant based on the image. Out-
put the thinking process in <think> </think> and final an-
swer in </think> </answer> tags. The output answer format
should be as follows: <think> ... </think> </think>species
name</answer> Please strictly follow the format.
ies have explored improving LLMs’ performance in reason-
ing tasks such as solving mathematical problems [2, 20, 31,
39, 41] and coding [6, 8, 46, 48]. A notable breakthrough
in this area is Deepseek-R1-Zero [4], which introduced a
new approach to achieving robust reasoning capabilities us-
ing RL merely, eliminating the supervised fine-tuning (SFT)
stage. However, current research on RL-based reasoning
has largely been confined to the language domain, with lim-
ited exploration of its application in multi-modal settings.
For LVLMs, RL has primarily been used for tasks like miti-
gating hallucinations and aligning models with human pref-
erence [19, 34, 35, 42, 43, 49–51], but there remains a sig-
nificant gap in research focusing on enhancing reasoning
and visual perception of Large Vision Language Models.
To address this gap, our work introduces a novel reinforce-
ment fine-tuning strategy Visual-RFT, applying verifiable
rewards with GRPO-based [31] RL to a broad range of vi-
sual perception tasks. Our approach aims to improve the
performance of LVLMs in processing various visual tasks,
especially when the fine-tuning data is limited.
3. Methodology
3.1. Preliminary
Reinforcement Learning with Verifiable Rewards.
Re-
inforcement Learning with Verifiable Rewards (RLVR)
[4, 12, 37] is a novel training approach designed to en-
hance language models in tasks with objectively verifi-
able outcomes, such as math and coding. Unlike previous
Reinforcement Learning from Human Feedback (RLHF)
[17, 26, 45], which relies on a trained reward model, RLVR
instead uses a direct verification function to assess correct-
ness. This approach simplifies the reward mechanism while
maintaining strong alignment with the task’s inherent cor-
rectness criteria.
Given the input question q, the policy
model πθ generates responses o and receives the verifiable
reward. More specifically, RLVR optimizes the following
objective:
max
πθ Eo∼πθ(q) [RRLVR(q, o)]
(1)
= [R(q, o) −βKL[πθ(o|q) ∥πref(o|q)]] ,
(2)
where πref is the reference model before optimization,
R is the verifiable reward function, and β is the hyper-
parameters to control the KL-divergence. The verifiable re-
ward function R takes the question and output pair (q, o) as
inputs, and checks if the ground-truth answer remains the
same as the prediction o:
R(q, o) =
(
1,
if o = ground truth,
0,
otherwise.
(3)
DeepSeek R1-Zero and GRPO.
The DeepSeek R1-Zero
algorithm eliminates dependence on supervised fine-tuning
(SFT) by employing reinforcement learning for training,
specifically through its Group Relative Policy Optimization
(GRPO) framework. Different from reinforcement learn-
ing algorithms such as PPO [30] that require a critic model
to evaluate policy performance, GRPO compares groups
of candidate responses directly, eliminating the need for
an additional critic model. For a given question q, GRPO
first generates G distinct responses {o1, o2, ..., oG} from
the current policy πθold. Then GRPO takes actions based
on these responses and denotes the obtained rewards as
{r1, r2, ..., rG}.
By computing their mean and standard
deviation for normalization, GRPO determines the relative
quality of these responses:
Ai = ri −mean({r1, . . . , rG})
std({r1, . . . , rG})
,
(4)
where Ai represents the relative quality of the i-th answer.
GRPO encourages the model to favor better answers with a
high reward value within the group.
3.2. Visual-RFT
The framework of Visual-RFT is shown in Fig. 3.
The
multi-modal input data from the user consists of images and
questions. The policy model πθ outputs a reasoning process
and generates a group of responses based on the input. Each
response is passed through a verifiable reward function to
compute the reward. After group computation of the re-
wards for each output, the quality of each response is eval-
uated and used to update the policy model. To ensure the
stability of the policy model training, Visual-RFT uses KL
divergence to limit the difference between the policy model
and the reference model. We will further discuss how to de-
sign the verifiable reward for visual tasks in Sec. 3.2.1, and
the data preparation steps in Sec. 3.2.2
4
SFT
<answer> globe-flower </answer>
<think> The image shows a yellow buttercup
flower with a green stem and green leaves in
the background. Buttercups are a type of fl
owering plant in the buttercup family. They
are known for their bright yellow flowers an
d are commonly found in meadows and grassl
ands. </think>
<answer>buttercup</answer>
SFT
<answer> hedgehog </answer>
<think>The dog in the image appears to be a J
apanese Chin, also known as a Japanese Spaniel.
This breed is known for its small size, fluffy co
at, and distinctive black and white coloration. T
he Japanese Chin is a popular pet and is known
for its friendly and affectionate nature.
</think>
<answer>
Japanese Chin (or Japanese Spaniel)
</answer>
This is an image
containing a pet.
Please identify the
species of the pet
based on the image
T h i s  i s  a n  i m a g e
containing a flower.
Please identify the
species of the flower
based on the image
Visual
RFT
Visual
RFT
Figure 4. Qualitative results of Fine-Grained Image Classification. The thinking process significantly improves the reasoning ability of
LVLMs, leading to higher image classification performance.
3.2.1. Verifiable Reward in Visual Perception
The reward model is a key step in reinforcement learning
(RL) that aligns models with preference alignment algo-
rithms, which can be as straightforward as a verification
function that checks for exact matches between predictions
and ground-truth answers. The RL training process in the
recent DeepSeek-R1 [4] model achieves a significant im-
provement in the model’s reasoning ability through the ver-
ifiable reward design. To transfer this strategy to the vi-
sual domain, we design different rule-based verifiable re-
ward functions for various visual perception tasks.
IoU Reward in Detection Tasks.
For the detection task,
the model’s output consists of bounding boxes (bbox) and
corresponding confidences.
The reward function for the
detection task should adequately consider the Intersection-
over-Union (IoU) metric, which is used to compute the
mean Average Precision (mAP) in evaluation. Therefore,
we design an IoU and confidence-based reward function
Rd.
First, for the model’s output box and confidence,
we sort these boxes based on their confidence, denoted as
{b1, b2, ..., bn}. We then match each bi with the ground
truth bbox,{bg
1, bg
2, ..., bg
m}, and calculate the IoU, while set-
ting an IoU threshold τ. Bounding boxes with an IoU be-
low this threshold τ are considered invalid, and unmatched
bboxes have an IoU of 0. After matching, we obtain the IoU
and confidence for each box from the initial set, denoted as
{iou1 : c1, iou2 : c2, ..., ioun : cn}.
We then use these IoU results and confidence to con-
struct our reward Rd. Our reward Rd consists of three parts,
including the IoU reward, Confidence reward, and Format
reward:
Rd = RIoU + Rconf + Rformat.
(5)
The IoU reward RIoU is the average IoU of all the bounding
boxes in the model’s output,
RIoU = iou1 + iou2 + ... + ioun
n
.
(6)
The confidence reward Rconf is related to IoU. For each
bounding box, if the ioui is non-zero, indicating a success-
ful match, the confidence reward for this single box rc as
the predicted confidence in computed as:
rci =
 ci
, if
ioui ̸= 0,
1 −ci
, if
ioui = 0.
(7)
This means that for successfully matched boxes, the higher
the confidence, the better. If the ioui is zero, indicating a
failed match, the lower the confidence reward rc for this
box, the better. The overall confidence reward Rconf for the
model’s output is the average of the confidence rewards of
all the bounding boxes in that output,
Rconf =
Pn
i=1 rci
n
.
(8)
The format reward Rformat is used to force the model predic-
tion to meet the required HTML tag format of <think> and
<answer> (will detailed in Sec. 3.2.2).
CLS Reward in Classification Tasks.
In classification
tasks, the reward function we use consists of two parts: ac-
curacy reward Racc and format reward Rformat. The accuracy
5
SFT
<think>
The image shows a athlete in a pool
expressing a strong emotion of
surprise or excitement. The athlete is
wearing goggles over her eyes. To
maintain clear vision underwater, she
should wear goggles over her eyes.
</think>
<answer>(350,187),(574,612)</answer>
SFT
Visual
RFT
Birds often need a stable surface to land on and rest.
What in the picture could provide a suitable landing spot
for a bird?
<think> The bird is perched on a
concrete ledge, which provides a
stable surface for the bird to land
on and rest. The ledge is also
visible in the image. </think>
SFT
SFT
Detect the object that both teams are fighting for in
a game of American football.
<think> The object that
both teams are fighting for
i n  a  g a m e  o f  A m e r i c a n
football is the ball. </think>
Output the bounding box in the image corresponding to
the instruction: What object in the room is mainly used as
a sitting tool for writing or using an electronic device?
SFT
Detect the engines for this plane.
<think> The image shows an Avro RJX airplane with two engines.
To determine the bounding box for the engines, I need to
identify the areas where the engines are visible on the airplane.
The engines are located on the wings of the airplane </think>
<think> The chair in the room
is mainly used as a sitting tool
f o r  w r i t i n g  o r  u s i n g  a n
electronic device.</think>
The athlete in the picture
seems to be very excited
about the competition
results. If she needs to
maintain clear vision
underwater, what item
should she wear over her
eyes?
Visual
RFT
Visual
RFT
Visual
RFT
Visual
RFT
Figure 5. Qualitative results of reasoning grounding on LISA [11]. Thinking process significantly improves reasoning grounding ability
with Visual-RFT.
reward is determined by comparing the model’s output class
with the ground truth class, yielding a value of 1 for correct
classification and 0 for incorrect classification:
Rcls = Racc + Rformat.
(9)
3.2.2. Data Preparation
To train the Visual-RFT on various visual perception tasks,
we need to construct the multi-modal training dataset. Sim-
ilar to DeepSeek-R1, to enhance the model’s reasoning
ability and apply this ability to improve visual perception,
Visual-RFT designed a prompt format to guide the model
to output its reasoning process before providing the final
answer. The prompts used for detection and classification
tasks are shown in Tab 1.
During the training process, we use the format reward
to guide the model to output its reasoning process and the
final answer in a structured format. The reasoning process
is key to the model’s self-learning and improvement during
reinforcement fine-tuning, while the reward determined by
the answer directs the model’s optimization.
4. Experiments
4.1. Experimental Setup
Implementation Details
Our method is adaptable to var-
ious visual perception tasks. We employ a few-shot learn-
ing approach, providing the model with a minimal num-
ber of samples for training. For the image classification
and object detection task, we adopt a few-shot setting to
evaluate the model’s fine-grained discriminative and recog-
nition capability, applying reinforcement learning on lim-
ited data. Then, for the LISA [11] dataset focusing on rea-
soning grounding, which demands strong reasoning abili-
6
Table 2.
Few-shot results on Fine-grained Classification
dataset.
We evaluated four fine-grained image classification
datasets.
Models
Average
Flower102
Pets37
FGVC
Cars196
Qwen2-VL-2B
56.0
54.8
66.4
45.9
56.8
one-shot
+ SFT
51.7
56.6
54.7
65.3
30.0
+ Visual-RFT
80.3
70.8
84.1
72.5
93.8
∆
+24.3
+16.0 +17.7 +26.6
+37.0
2-shot
+ SFT
58.8
60.3
65.6
68.9
40.2
+ Visual-RFT
83.5
75.8
87.5
75.3
95.4
∆
+27.5
+21.0 +21.1 +29.4
+38.6
4-shot
+ SFT
55.6
58.5
55.5
67.9
40.5
+ Visual-RFT
81.9
71.4
86.1
74.8
95.3
∆
+25.9
+16.6 +19.7 +28.9
+38.5
8-shot
+ SFT
60.3
59.6
71.4
69.2
40.9
+ Visual-RFT
85.1
77.7
90.2
75.9
96.5
∆
+29.1
+22.9 +23.8 +30.0
+39.7
16-shot
+ SFT
64.0
66.8
71.6
76.1
41.5
+ Visual-RFT
85.3
79.2
87.1
79.4
95.3
∆
+29.3
+24.4 +20.7 +33.5
+38.5
ties, we train the model using Visual-RFT and assess its
reasoning and perception performance. Lastly, for open-
vocabulary object detection, we evaluate the model’s gen-
eralization capability by training the Qwen2-VL-2/7B [38]
using Visual-RFT on a subdivided COCO dataset contain-
ing 65 base classes. We then test it on 15 novel classes from
COCO and 13 rare classes from LVIS [5]. The model’s vi-
sual perception and reasoning abilities are assessed in an
open-vocabulary detection setting. In our detection experi-
ments, we first prompt the model to check whether the cat-
egory is present in the image, then predict bound boxes for
categories that exist in the images.
4.2. Few-Shot Classification
To demonstrate the extensive generalization ability of
Visual-RFT in the visual domain, we conduct few-shot
experiments on fine-grained image classification. We se-
lected four datasets: Flower102 [22], Pets37 [27], FGVC-
Aircraft [21], and Car196 [10], which contain dozens to
hundreds of similar categories, adding significant difficulty
Table 3. Few-Shot results on COCO dataset of 8 categories.
We conducted one-shot, 2-shot, 4-shot, 8-shot, and 16-shot exper-
iments on 8 categories from the COCO dataset.
Models
mAP
bus
train
fire hydrant
stop sign
cat
dog
bed
toilet
Qwen2-VL-2B
Baseline
19.6
19.0
15.8
25.8
18.4
29.9
23.2
14.6
9.8
1-shot
+ SFT
19.5
18.3
17.4
23.1
18.2
28.0
23.4
17.3
10.4
+ Visual-RFT 33.6
23.4
35.7
39.1
23.8
54.3
42.5
19.5
30.8
∆
+14.0 +4.4 +19.9 +13.3 +5.4 +24.4 +19.3 +4.9 +21.0
2-shot
+ SFT
21.0
22.1
15.8
29.8
19.0
28.9
26.5
15.5
10.2
+ Visual-RFT 41.5
28.8
39.6
38.2
48.0
63.8
52.7
25.9
34.9
∆
+21.9 +9.8 +23.8 +12.4 +29.6 +33.9 +29.5 +11.3 +25.1
4-shot
+ SFT
25.2
25.4
23.6
26.6
21.5
35.6
30.6
18.4
19.9
+ Visual-RFT 40.6
30.0
40.6
45.7
35.0
60.9
44.9
24.6
43.1
∆
+21.0 +11.0 +24.8 +19.9 +16.6 +31.0 +21.7 +10.0 +33.3
8-shot
+ SFT
30.2
25.8
35.1
29.4
21.9
44.5
39.0
22.6
23.5
+ Visual-RFT 47.4
36.2
47.9
50.4
47.7
65.2
57.0
30.4
44.0
∆
+27.8 +17.2 +32.1 +24.6 +29.3 +35.3 +33.8 +15.8 +34.2
16-shot
+ SFT
31.3
24.0
35.9
32.0
23.6
39.8
40.6
27.5
26.8
+ Visual-RFT 46.8
36.2
44.4
51.4
48.5
66.6
56.2
27.6
43.4
∆
+27.2 +17.2 +28.6 +25.6 +30.1 +36.7 +33.0 +13.0 +33.6
Qwen2-VL-7B
Baseline
43.0
35.0
43.3
37.1
36.7
57.3
50.3
37.4
47.1
4-shot
+ SFT
44.1
41.4
51.7
35.6
30.8
60.5
52.7
38.5
41.5
+ Visual-RFT 54.3
44.3
59.8
52.0
46.0
72.7
62.8
41.9
55.0
∆
+11.3 +9.3 +16.5 +14.9 +9.3 +15.4 +12.5 +4.5
+7.9
to the classification task.
As shown in Tab. 2, with just one-shot of data, Visual-
RFT already delivers a significant performance boost
(+24.3%).
In contrast, SFT shows a noticeable decline
(-4.3%) with the same minimal amount of data.
Under
the 4-shot setting, the performance of SFT is still slightly
lower than the baseline, while the reinforcement fine-tuned
model with Visual-RFT achieves an average performance
improvement of 25.9. Under the 8-shot and 16-shot set-
tings, as the amount of data increases, SFT’s performance
slightly exceeds the baseline. However, SFT still lags sig-
nificantly behind the performance of the Visual-RFT. In
Fig.4, we present some inference cases of the model af-
ter reinforcement fine-tuning when handling fine-grained
classification tasks. These results not only demonstrate the
strong generalization ability of Visual-RFT and its capacity
to learn from limited data but also confirm that reinforce-
ment fine-tuning, compared to SFT, leads to a genuine un-
7
Table 4. Few-shot results on LVIS dataset of 6 rare categories.
We conducted 10-shot experiments on 6 rare categories from the
LVIS dataset.
Models
mAP
horse buggy
die
kitchen table
omelet
papaya
stepladder
Qwen2-VL-2B 4.0
2.9
1.2
13.4
4.7
1.5
0.0
+ SFT
10.0
7.0
7.6
34.1
4.7
6.3
0.0
+ Visual-RFT 19.4
9.1
19.6
42.2
20.4
14.5
10.9
∆
+15.4 +6.2 +18.4 +29.2 +15.7 +13.0 +10.9
Qwen2-VL-7B 15.4
19.7 21.9
14.5
18.1
18.5
0.0
+ SFT
27.6
26.9 21.9
49.7
29.2
25.2
12.7
+ Visual-RFT 33.8
26.2 27.8
70.6
23.5
21.2
29.3
∆
+18.4 +6.5 +5.9 +56.1 +5.4
+2.7 +29.3
Table 5. Few-shot results on MG dataset of 5 categories. By in-
troducing out-of-domain data, we increased the difficulty of model
recognition and reasoning, further demonstrating the strong gener-
alization ability of reinforcement fine-tuning in visual perception
tasks.
Models
mAP bird feline-or-canid serpent slime wyvern
Qwen2-VL-2B 20.6
12.9
19.8
25.5
18.4
26.4
4-shot
+ SFT
26.8
19.5
22.4
26.8
33.5
31.8
+ Visual-RFT 61.8
63.9
53.2
70.2
64.5
57.5
∆
+41.2 +51.0
+33.4
+44.7
+46.1
+31.1
16-shot
+ SFT
51.3
42.7
44.4
56.4
61.1
52.0
+ Visual-RFT 63.4
59.9
50.8
76.3
71.7
58.1
∆
+42.8 +47.0
+56.4
+50.8
+53.3
+31.7
derstanding of the task and deeper learning from reasoning.
4.3. Few-Shot Object Detection
Few-shot learning has always been one of the core chal-
lenges faced by traditional visual models and large-scale
vision-language models (LVLMs).
Reinforcement fine-
tuning provides a new solution to this problem by enabling
the model to quickly learn and understand with a small
amount of data.
We selected eight categories from the
COCO dataset, with 1, 2, 4, 8, and 16 images per cate-
gory, to construct training sets with limited data. For the
LVIS dataset, we select 6 rare categories. Since the training
images for these rare categories are very sparse, with each
category having between 1 and 10 images, we approximated
this as a 10-shot setting. We then train the Qwen2-VL-2/7B
model for 200 steps using both reinforcement fine-tuning
and SFT, to evaluate the model’s learning ability with lim-
Table 6. Reasoning Grounding Results on LISA [11]. Visual-
RFT surpasses SFT in reasoning grounding with 239 training im-
ages.
Model
mIoUtest
mIoUval
gIoUtest
OV-Seg [14]
28.4
30.5
26.1
X-Decoder [54]
28.5
29.1
24.3
GroundedSAM [18]
26.2
28.6
21.3
Qwen2-VL-2B
26.9
30.1
25.3
+ SFT
28.3
29.7
25.3
+ Visual-RFT
37.6
34.4
34.4
∆
+10.7
+4.3
+9.1
Qwen2-VL-7B
40.4
45.2
38.0
+ SFT
39.1
43.9
37.2
+ Visual-RFT
43.9
47.1
43.7
∆
+3.5
+1.9
+5.6
Table 7. Open Vocabulary Object Detection Results on COCO
dataset. We trained on 65 base categories and tested on 15 novel
categories.
Models
mAPn
mAPb
mAPall
Qwen2-VL-2B
9.8
6.0
6.7
+ SFT
13.6
7.8
8.9
+ Visual-RFT
31.3
20.6
22.6
∆
+21.5
+14.6
+15.9
Qwen2-VL-7B
26.3
17.5
19.2
+ SFT
25.7
17.5
19.0
+ Visual-RFT
35.8
25.4
27.4
∆
+9.5
+7.9
+8.2
ited data.
As shown in Tab. 3 and Tab. 4, although both SFT and
reinforcement fine-tuning can improve the model’s recog-
nition accuracy under the few-shot setting, the model after
reinforcement fine-tuning consistently outperforms the SFT
model by a large margin, maintaining a significant lead. On
the COCO [15] categories, as the training data increases, the
SFT model reaches an average mAP of approximately 31,
while the reinforcement fine-tuned model approaches 47. In
the LVIS [5] few-shot experimental results shown in Tab. 4,
for the six more challenging rare categories in LVIS, rein-
forcement fine-tuning still outperforms SFT. The results in
Tab. 3 and Tab. 4 clearly demonstrate the exceptional perfor-
mance of reinforcement fine-tuning in the few-shot setting,
where the model achieves a significant improvement in vi-
sual perception capabilities through reinforcement learning
with only a small amount of data.
8
Table 8. Open Vocabulary Object Detection Results on LVIS dataset. We trained on the 65 base categories of the COCO dataset and
tested on the 13 rare categories of the LVIS dataset.
Models
mAP
casserole
die
egg roll
futon
garbage
handsaw
hippopotamus
kitchen table
mallet
omelet
shot glass
stepladder
sugar bowl
GroudingDINO-B [18]
23.9
17.1
0.0
2.4
47.5
27.7
13.4
15.2
92.5
0.0
26.6
16.0
41.0
10.7
Qwen2-VL-2B
2.7
1.6
1.2
0.0
2.4
0.0
10.0
0.0
13.4
0.2
4.7
2.1
0.0
0.0
+ SFT
7.6
3.9
21.2
0.0
0.0
10.7
9.0
11.6
19.4
0.0
11.7
6.3
0.0
5.2
+ Visual-RFT
20.7
24.5
23.4
2.0
16.0
27.7
20.2
14.4
45.8
11.1
22.7
6.0
6.0
40.2
∆
+18.0
+22.9 +22.2
+2.0 +13.6 +27.7
+10.2
+14.4
+32.4
+10.9
+18.0 +3.9
+6.0
+40.2
Qwen2-VL-7B
15.7
3.7
21.9
0.7
24.5
15.3
19.2
13.1
14.5
11.9
18.1
27.9
0.0
33.8
+ SFT
24.0
20.8
25.4
0.6
41.8
12.2
19.2
18.8
42.5
11.9
15.3
27.9
28.1
47.8
+ Visual-RFT
30.4
19.7
27.8
4.3
41.8
17.4
35.1
20.0
70.6
16.7
23.5
29.8
29.3
59.8
∆
+14.7
+16.0 +5.9
+3.6 +17.3 +2.1
+15.9
+6.9
+56.1
+4.8
+5.4
+1.9
+29.3
+26.0
We further test on some abstract out-of-domain datasets.
We selected the MG (Monster Girls) dataset, which con-
tains different types of anime-style monster girls. By us-
ing out-of-domain data, we increased the difficulty of both
model recognition and reasoning, and conducted experi-
ments under 4-shot and 16-shot settings. The results, shown
in Tab. 5, indicate that reinforcement fine-tuning achieved
a significant performance improvement, surpassing super-
vised fine-tuning (SFT).
4.4. Reasoning Grounding
Another crucial aspect of vision-language intelligence is
grounding the exact object according to user needs. Pre-
vious specialized detection systems lack reasoning abilities
and fail to fully understand the user’s intentions. Pioneered
by LISA [11], there have been works done to enable large
language models (LLMs) to output control tokens for other
models (such as SAM [9]) or directly predict bounding box
coordinates [29, 38] through supervised fine-tuning. In our
work, we explore the use of Visual-RFT in this task and
find that reinforcement learning (RL) leads to significant
improvements over supervised fine-tuning.
We finetune Qwen2-VL 2B/7B model [38] using Visual-
RFT and supervised fine-tuning (SFT) on the LISA training
set, which consists of 239 images with reasoning ground-
ing objects. We follow the same test setting with LISA and
compare the results of SFT and our method, both with 500
fine-tuning steps. As shown in Tab. 6, Visual-RFT signifi-
cantly improves the final results in terms of bounding box
IoU compared to SFT. Additionally, we prompt SAM [9]
with the Qwen2-VL predicted bounding box to generate
the segmentation mask (evaluated using gIoU). Visual-RFT
significantly enhances grounding ability and outperforms
previous specialized detection systems. Qualitative results
are visualized in Fig. 5, where the thinking process signif-
icantly improves the ability to reason and grounding accu-
racy. Through Visual-RFT, Qwen2-VL learns to think crit-
ically and carefully examine the image to produce accurate
grounding results.
4.5. Open Vocabulary Object Detection
The advantage of Visual-RFT over SFT arises from the
former’s true deep understanding of the task, rather than
merely memorizing the data.
To further demonstrate
the powerful generalization ability of reinforcement fine-
tuning, we conduct open vocabulary object detection ex-
periments.
We first randomly sampled 6K annotations
from the COCO dataset, which included 65 base categories.
We perform Visual-RFT and SFT on the Qwen2-VL-2/7B
model [38] using this data, and test the model on 15 new
categories it has never seen before. To increase the diffi-
culty, we further test 13 rare categories from the LVIS [5]
dataset.
As shown in Tab. 7 and Tab. 8, after reinforcement fine-
tuning, the Qwen2-VL-2/7B model achieves an average
mAP increase of 21.5 and 9.5 on 15 new categories from the
COCO dataset. On the more challenging rare categories of
the LVIS [5] dataset, mAP increased by 18.0 and 14.7. The
Visual-RFT not only transfers its detection capabilities from
the COCO base categories to the new COCO categories but
also achieves significant improvements on the more chal-
lenging rare categories of LVIS. Notably, for some rare
LVIS categories in Tab. 8, the original or SFT-trained mod-
els cannot recognize these categories, resulting in 0 AP.
However, after reinforcement fine-tuning, the model shows
a qualitative leap from 0 to 1 in recognizing these previ-
ously unidentifiable categories (such as egg roll and futon).
This demonstrates that Visual-RFT has a significant impact
9
on improving the performance and generalization ability in
visual recognition for LVLMs.
5. Conclusion
In this paper, we introduce Visual Reinforcement Fine-
tuning (Visual-RFT), the first approach to adapt the GRPO-
based reinforcement learning strategy for enhancing the vi-
sual perception and grounding ability of LVLMs. By using
a rule-based verifiable reward system, Visual-RFT reduces
the need for manual labeling and simplifies reward com-
putation, achieving significant improvements across vari-
ous visual perception tasks. Extensive experiments show
that Visual-RFT excels in fine-grained classification, open
vocabulary detection, reasoning grounding and few-shot
learning tasks. It outperforms supervised fine-tuning (SFT)
with minimal data and shows strong generalization. This
work demonstrates the potential of reinforcement learning
to enhance the capabilities of LVLMs, making them more
efficient and effective in visual perception tasks.
References
[1] Marwa Abdulhai, Isadora White, Charlie Snell, Charles Sun,
Joey Hong, Yuexiang Zhai, Kelvin Xu, and Sergey Levine.
Lmrl gym: Benchmarks for multi-turn reinforcement learn-
ing with language models. arXiv preprint arXiv:2311.18232,
2023. 3
[2] Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu
Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei
Chu, et al.
Internlm2 technical report.
arXiv preprint
arXiv:2403.17297, 2024. 4
[3] Thomas Carta, Cl´ement Romac, Thomas Wolf, Sylvain Lam-
prier, Olivier Sigaud, and Pierre-Yves Oudeyer. Grounding
large language models in interactive environments with on-
line reinforcement learning. In ICLR, 2023. 3
[4] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song,
Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi
Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning
capability in llms via reinforcement learning. arXiv preprint
arXiv:2501.12948, 2025. 1, 4, 5
[5] Agrim Gupta, Piotr Dollar, and Ross Girshick.
Lvis: A
dataset for large vocabulary instance segmentation. In Pro-
ceedings of the IEEE/CVF conference on computer vision
and pattern recognition, pages 5356–5364, 2019. 3, 7, 8, 9
[6] Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng
Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Kem-
ing Lu, et al. Qwen2. 5-coder technical report. arXiv preprint
arXiv:2409.12186, 2024. 4
[7] Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richard-
son, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander
Madry, Alex Beutel, Alex Carney, et al. Openai o1 system
card. arXiv:2412.16720, 2024. 1, 2, 3
[8] Fangkai Jiao, Geyang Guo, Xingxing Zhang, Nancy F
Chen, Shafiq Joty, and Furu Wei.
Preference optimiza-
tion for reasoning with pseudo feedback.
arXiv preprint
arXiv:2411.16345, 2024. 4
[9] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao,
Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White-
head, Alexander C Berg, Wan-Yen Lo, et al. Segment any-
thing. In Proceedings of the IEEE/CVF international confer-
ence on computer vision, pages 4015–4026, 2023. 9
[10] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei.
3d object representations for fine-grained categorization. In
ICCV workshops, 2013. 7
[11] Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui
Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation
via large language model. In Proceedings of the IEEE/CVF
Conference on Computer Vision and Pattern Recognition,
pages 9579–9589, 2024. 3, 6, 8, 9
[12] Nathan Lambert,
Jacob Morrison,
Valentina Pyatkin,
Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester
James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al.
T\” ulu 3: Pushing frontiers in open language model post-
training. arXiv preprint arXiv:2411.15124, 2024. 1, 4
[13] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li,
Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Zi-
wei Liu, et al. Llava-onevision: Easy visual task transfer.
arXiv preprint arXiv:2408.03326, 2024. 3
10
[14] Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan
Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana
Marculescu. Open-vocabulary semantic segmentation with
mask-adapted clip. In Proceedings of the IEEE/CVF con-
ference on computer vision and pattern recognition, pages
7061–7070, 2023. 8
[15] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays,
Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence
Zitnick. Microsoft coco: Common objects in context. In
Computer Vision–ECCV 2014: 13th European Conference,
Zurich, Switzerland, September 6-12, 2014, Proceedings,
Part V 13, pages 740–755. Springer, 2014. 3, 8
[16] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao
Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu
Zhang, Chong Ruan, et al. Deepseek-v3 technical report.
arXiv preprint arXiv:2412.19437, 2024. 3
[17] Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He,
Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou.
Skywork-Reward: Bag of tricks for reward modeling in llms.
arXiv preprint arXiv:2410.18451, 2024. 1, 4
[18] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao
Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang,
Hang Su, et al.
Grounding dino:
Marrying dino with
grounded pre-training for open-set object detection.
In
European Conference on Computer Vision, pages 38–55.
Springer, 2024. 3, 8, 9
[19] Ziyu Liu, Yuhang Zang, Xiaoyi Dong, Pan Zhang, Yuhang
Cao, Haodong Duan, Conghui He, Yuanjun Xiong, Dahua
Lin, and Jiaqi Wang. Mia-dpo: Multi-image augmented di-
rect preference optimization for large vision-language mod-
els. arXiv preprint arXiv:2410.17637, 2024. 4
[20] Trung Quoc Luong, Xinbo Zhang, Zhanming Jie, Peng Sun,
Xiaoran Jin, and Hang Li. Reft: Reasoning with reinforced
fine-tuning, 2024. 4
[21] Subhransu Maji,
Esa Rahtu,
Juho Kannala,
Matthew
Blaschko, and Andrea Vedaldi. Fine-grained visual classi-
fication of aircraft. arXiv preprint arXiv:1306.5151, 2013.
7
[22] Maria-Elena Nilsback and Andrew Zisserman. Automated
flower classification over a large number of classes.
In
ICVGIP, 2008. 7
[23] OpenAI. Hello gpt-4o, 2024. 3
[24] OpenAI. Openai o3-mini system card, 2025. 2
[25] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Car-
roll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini
Agarwal, Katarina Slama, Alex Ray, et al.
Training lan-
guage models to follow instructions with human feedback.
In NeurIPS, 2022. 3
[26] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Car-
roll L. Wainwright, Pamela Mishkin, Chong Zhang, Sand-
hini Agarwal, Katarina Slama, Alex Ray, John Schulman,
Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens,
Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan
Leike, and Ryan J. Lowe. Training language models to fol-
low instructions with human feedback. In NeurIPS, 2022. 1,
4
[27] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and
CV Jawahar. Cats and dogs. In CVPR, 2012. 7
[28] Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kiant´e
Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Han-
naneh Hajishirzi, and Yejin Choi. Is reinforcement learning
(not) for natural language processing: Benchmarks, base-
lines, and building blocks for natural language policy opti-
mization. In ICLR, 2023. 3
[29] Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdel-
rahman Shaker, Salman Khan, Hisham Cholakkal, Rao M
Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S Khan.
Glamm: Pixel grounding large multimodal model. In Pro-
ceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition, pages 13009–13018, 2024. 9
[30] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Rad-
ford, and Oleg Klimov. Proximal policy optimization algo-
rithms. arXiv:1707.06347, 2017. 4
[31] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao
Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li,
Y Wu, et al. Deepseekmath: Pushing the limits of mathe-
matical reasoning in open language models. arXiv preprint
arXiv:2402.03300, 2024. 2, 4
[32] Charlie Victor Snell, Ilya Kostrikov, Yi Su, Sherry Yang, and
Sergey Levine. Offline RL for natural language generation
with implicit language q learning. In ICLR, 2023. 3
[33] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler,
Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and
Paul F Christiano. Learning to summarize with human feed-
back. In NeurIPS, 2022. 3
[34] Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu,
Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui,
Yu-Xiong Wang, Yiming Yang, et al. Aligning large multi-
modal models with factually augmented rlhf. arXiv preprint
arXiv:2309.14525, 2023. 4
[35] Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu,
Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui,
Yu-Xiong Wang, Yiming Yang, et al. Aligning large multi-
modal models with factually augmented rlhf. arXiv preprint
arXiv:2309.14525, 2023. 4
[36] Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu,
Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui,
Yu-Xiong Wang, Yiming Yang, et al. Aligning large multi-
modal models with factually augmented rlhf. In ACL, 2024.
3
[37] Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu
Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang
Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement
learning with llms. arXiv preprint arXiv:2501.12599, 2025.
1, 4
[38] Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan,
Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin
Ge, et al. Qwen2-vl: Enhancing vision-language model’s
perception of the world at any resolution.
arXiv preprint
arXiv:2409.12191, 2024. 3, 7, 9
[39] An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen
Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren
Zhou, Junyang Lin, et al. Qwen2. 5-math technical report:
Toward mathematical expert model via self-improvement.
arXiv preprint arXiv:2409.12122, 2024. 4
11
[40] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran,
Karthik R Narasimhan, and Yuan Cao. React: Synergizing
reasoning and acting in language models. In ICLR, 2023. 3
[41] Huaiyuan Ying, Shuo Zhang, Linyang Li, Zhejian Zhou,
Yunfan Shao, Zhaoye Fei, Yichuan Ma, Jiawei Hong, Kuikun
Liu, Ziyi Wang, et al. Internlm-math: Open math large lan-
guage models toward verifiable reasoning.
arXiv preprint
arXiv:2402.06332, 2024. 4
[42] Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng
Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng,
Maosong Sun, et al. RlHF-V: Towards trustworthy mllms
via behavior alignment from fine-grained correctional human
feedback. In CVPR, 2024. 4
[43] Tianyu Yu, Haoye Zhang, Yuan Yao, Yunkai Dang, Da Chen,
Xiaoman Lu, Ganqu Cui, Taiwen He, Zhiyuan Liu, Tat-Seng
Chua, et al. RLAIF-V: Aligning mllms through open-source
ai feedback for super gpt-4v trustworthiness. arXiv preprint
arXiv:2405.17220, 2024. 4
[44] Yuhang Zang, Wei Li, Jun Han, Kaiyang Zhou, and
Chen Change Loy. Contextual object detection with mul-
timodal large language models. IJCV, 2024. 3
[45] Yuhang Zang, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Ziyu
Liu, Shengyuan Ding, Shenxi Wu, Yubo Ma, Haodong Duan,
Wenwei Zhang, et al. A simple yet effective multi-modal
reward model. arXiv preprint arXiv:2501.12368, 2025. 1, 4
[46] Kechi Zhang, Ge Li, Yihong Dong, Jingjing Xu, Jun Zhang,
Jing Su, Yongfei Liu, and Zhi Jin. Codedpo: Aligning code
models with self generated and verified source code. arXiv
preprint arXiv:2410.05605, 2024. 4
[47] Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Rui
Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang,
Linke Ouyang, et al. Internlm-xcomposer-2.5: A versatile
large vision language model supporting long-contextual in-
put and output. arXiv preprint arXiv:2407.03320, 2024. 3
[48] Yuxiang Zhang, Shangxi Wu, Yuqi Yang, Jiangming Shu,
Jinlin Xiao, Chao Kong, and Jitao Sang. o1-coder: an o1
replication for coding.
arXiv preprint arXiv:2412.00154,
2024. 4
[49] Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Ji-
aqi Wang, and Conghui He. Beyond hallucinations: Enhanc-
ing lvlms through hallucination-aware direct preference op-
timization. arXiv preprint arXiv:2311.16839, 2023. 4
[50] Yiyang Zhou, Chenhang Cui, Rafael Rafailov, Chelsea Finn,
and Huaxiu Yao. Aligning modalities in vision large lan-
guage models via preference fine-tuning.
arXiv preprint
arXiv:2402.11411, 2024.
[51] Yiyang Zhou, Chenhang Cui, Rafael Rafailov, Chelsea Finn,
and Huaxiu Yao. Aligning modalities in vision large lan-
guage models via preference fine-tuning.
arXiv preprint
arXiv:2402.11411, 2024. 4
[52] Yifei Zhou, Andrea Zanette, Jiayi Pan, Sergey Levine, and
Aviral Kumar. Archer: Training language model agents via
hierarchical multi-turn rl. In ICML, 2024. 3
[53] Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B
Brown, Alec Radford, Dario Amodei, Paul Christiano, and
Geoffrey Irving. Fine-tuning language models from human
preferences. arXiv:1909.08593, 2019. 3
[54] Xueyan Zou, Zi-Yi Dou, Jianwei Yang, Zhe Gan, Linjie Li,
Chunyuan Li, Xiyang Dai, Harkirat Behl, Jianfeng Wang, Lu
Yuan, et al. Generalized decoding for pixel, image, and lan-
guage. In Proceedings of the IEEE/CVF conference on com-
puter vision and pattern recognition, pages 15116–15127,
2023. 8
12


Question:
This paper introduces speculative decoding into MLLMs for the first time. The authors adopt a language-only model as the draft model, rather than a small MLLM with the same architecture as the target model. This is because:
A. The language-only model are easier to train than the multimodal model.
B. There are suitable language-only models available, but no existing MLLMs that can be used directly.
C. Experiments conditioning on both types of models show that the language-only model is more effective.
D. When using the language-only model, the token pass rate is easier to calculate.

Answer: C

To achieve higher gain with speculative decoding, we need a draft model significantly smaller than and well-aligned with our target model (LLaVA-7B). The most common choice for draft models in prior works on LLMs is to use a small pre-trained model from the same family of models as the target model or train a smaller model which has the same architecture as the target model [15]. Since there is no publicly available smaller model in the LLaVA family, we need to train a draft model from scratch. A natural choice for draft model architecture is to follow LLaVA’s architecture where the draft model comprises an adapter and a language-model backbone with smaller number of parameters than the LLaVA 7B. In our approach, we use both, 1) a smaller LLaVA draft model which consists of a smaller image adapter and a draft language model, and 2) the language-only draft model which generates draft tokens by conditioning only on the input text tokens. Given an input image with image embeddings V1:m, token embeddings X1:n the draft model generates the draft tokens ˆXn+1:n+L where the draft token ˆXn+j ∼p(·|X1:n, ˆXn+1:n+j−1) is generated by conditioning only on the text tokens. The target LLaVA model verifies the draft tokens by computing the target distribution which is conditioned on both the image embeddings V1:m and the text token embeddings X1:n, i.e., draft token ˆXn+j is accepted with probability min(1, q( ˆXn+j|V1:m, X1:n, ˆXn+1:n+j−1) / p( ˆXn+j|X1:n, ˆXn+1:n+j−1) ). Using the language-model-only draft model is more efficient than a draft model with LLaVA architecture since 1) it does not need an additional adapter as it does not condition on the image embeddings for generating draft tokens, and 2) it does not require the training of the adapter.

On Speculative Decoding for Multimodal Large Language Models
Mukul Gagrani*
Raghavv Goel*
Wonseok Jeon
Junyoung Park
Mingu Lee
Christopher Lott
Qualcomm AI Research
Abstract
Inference with Multimodal Large Language Models
(MLLMs) is slow due to their large-language-model
backbone which suffers from memory bandwidth bottle-
neck and generates tokens auto-regressively. In this pa-
per, we explore the application of speculative decoding
to enhance the inference efficiency of MLLMs, specifi-
cally the LLaVA 7B model. We show that a language-
only model can serve as a good draft model for spec-
ulative decoding with LLaVA 7B, bypassing the need
for image tokens and their associated processing com-
ponents from the draft model. Our experiments across
three different tasks show that speculative decoding can
achieve a memory-bound speedup of up to 2.37× us-
ing a 115M parameter language model that we trained
from scratch.
Additionally, we introduce a compact
LLaVA draft model incorporating an image adapter,
which shows marginal performance gains in image cap-
tioning while maintaining comparable results in other
tasks.
1. Introduction
Large Language Models (LLMs) have become ubiqui-
tous across various domains due to their impressive per-
formance.
However, LLMs only take text queries as
input but real-world data comes in the form of multi-
ple modalities including visual data. Multi-modal Large
Language Models (MLLMs) [1, 13, 21, 22] provides the
LLMs with image understanding abilities, and the fu-
sion of visual and textual tokens enhances the model’s
interaction with users, leading to more informed re-
sponses. MLLMs comprise of an image encoder to pro-
cess the image information and an adapter which trans-
forms the image encodings to the language model em-
bedding space. In addition, MLLMs have a language-
model backbone in the form of a LLM and thus inherit
the auto-regressive generation and memory-bandwidth
*Equal contribution.
Correspondence to {mgagrani,raghgoel,mingul}@qti.qualcomm.com.
Qualcomm AI Research is an initiative of Qualcomm Technologies,
Inc.
bottleneck which lead to slow inference [19].
Speculative decoding [3, 7, 9, 15, 20] has been pro-
posed as a solution to accelerate the LLM inference
without loss in accuracy, where a smaller draft model
predicts multiple future tokens which are verified in a
single call of the LLM. Given that MLLMs have a LLM
backbone, speculative decoding can be used to make in-
ference with MLLMs more efficient. Many recent works
have studied the application of speculative decoding and
its variants [2, 5, 7, 8, 18, 20] for LLMs, but no such
work exists in the context of MLLMs to the best of our
knowledge.
In this paper, we apply speculative decoding to
LLaVA 7B model (with LLaMA 7B model as language-
model backbone) to make inference more efficient,
block diagram shown in Figure 1. Due to the lack of
publicly available models of LLaVA and LLaMA fam-
ilies smaller than 7B parameters, we train a language
model of size 115M from scratch for speculative decod-
ing. We show that language-only model which does not
consider the image tokens (and hence does not require
the image encoder and adapter) can serve as a good draft
model for LLaVA 7B. We conduct experiments on three
different tasks including image QA on LLaVA Instruct
150K dataset [13], image captioning on Coco dataset
[11] and ScienceQA dataset [14], using draft model
candidates which have gone through different stages of
training and fine-tuning. Our results show that we can
achieve memory-bound speedup of upto 2.37× using
only a language model as draft model. We also create
a small LLaVA draft model which consists of an image
adapter along with our trained language model and show
that it improves the performance slightly on COCO cap-
tioning task and ScienceQA task while performing sim-
ilar to language-model-only draft models on the other
tasks.
2. Background
2.1. Speculative Decoding
SPeculative Decoding (SPD) [3, 9] involves a smaller
draft model generating multiple tokens which are veri-
fied in parallel by the target LLM. Given an input con-
1
arXiv:2404.08856v1  [cs.CL]  13 Apr 2024
text X1:n := [X1, . . . , Xn], the draft model generates
a sequence of tokens ˆXn+1:n+L in an auto-regressive
fashion, ˆXn+j ∼p(·|X1:n, ˆXn+1:n+j−1). The draft to-
kens are then verified via a single call of the target LLM
(q) using rejection sampling criteria that guarantees the
same output token distribution as that of the target LLM.
Specifically, token ˆXn+j is accepted with probability
min
(
1, q( ˆXj|X1:n, ˆXn+1:n+j−1)
p( ˆXj|X1:n, ˆXn+1:n+j−1)
)
.
If a draft token ˆXn+j is rejected, then a new token
is sampled from the residual distribution defined as
pres(x) = max(0, q(x) −p(x)).
2.2. Multimodal Large Language Models
An image-based Multimodal Large Language Model
(MLLM) consists of 1) a vision encoder to encode the
input image, 2) an adapter to convert the image encod-
ings to language model embeddings, and 3) a language-
model backbone.
We describe the framework of the
LLaVA model in more detail as follows; given an in-
put image I and the text query Q, the image I is con-
verted into a sequence H1, H2, . . . , Hm of m image en-
codings, and the text query is converted to a sequence
of token embeddings X1, X2, . . . Xn. The image en-
codings are further transformed via an adapter gθ (a
small multi-layer perceptron) to get image embeddings,
Vi = gθ(Hi). This is done to convert the encodings
Hi to the language model embedding space. Tokens are
then generated by the language model conditioning on
the image embeddings and the token embeddings as fol-
lows:
Xn+1 ∼q(·|V1:m, X1:n)
(1)
3. SPD for MLLMs
To achieve higher gain with speculative decoding, we
need a draft model significantly smaller than and well-
aligned with our target model (LLaVA-7B). The most
common choice for draft models in prior works on
LLMs is to use a small pre-trained model from the same
family of models as the target model or train a smaller
model which has the same architecture as the target
model [15]. Since there is no publicly available smaller
model in the LLaVA family, we need to train a draft
model from scratch. A natural choice for draft model
architecture is to follow LLaVA’s architecture where the
draft model comprises an adapter and a language-model
backbone with smaller number of parameters than the
LLaVA 7B. In our approach, we use both, 1) a smaller
LLaVA draft model which consists of a smaller image
adapter and a draft language model, and 2) the language-
only draft model which generates draft tokens by con-
<image>
Vision Encoder
Image Projector
𝑉1
Target Language Model
Draft Language Model
<input text>
⋯
𝑉𝑚
𝐻1
⋯
𝐻𝑚
Text Tokenizer
𝑋1
⋯
𝑋𝑛
෠𝑋𝑛+1
෠𝑋𝑛+2
෠𝑋𝑛+3
෠𝑋𝑛+2
෠𝑋𝑛+3
Autoregressive
Decoding
𝑉1
⋯
𝑉𝑚
𝑋1
⋯
𝑋𝑛
෠𝑋𝑛+1
෠𝑋𝑛+2
෠𝑋𝑛+3
Parallel
Evaluation
෠𝑋𝑛+1
෠𝑋𝑛+1
෠𝑋𝑛+2
෠𝑋𝑛+1
෠𝑋𝑛+2
෠𝑋𝑛+3
Verification

✓
✓
Input
Processing
Figure 1. SPD with a MLLM as target having three compo-
nents: vision encoder, image projector, and target language
model, and the smaller language model as draft. The small
draft model generates draft tokens autoregressively for block-
size number of iterations followed by parallel evaluation by the
target language model which also uses image features.
ditioning only on the input text tokens. Given an in-
put image with image embeddings V1:m, token embed-
dings X1:n the draft model generates the draft tokens
ˆXn+1:n+L where the draft token
ˆXn+j ∼p(·|X1:n, ˆXn+1:n+j−1)
is generated by conditioning only on the text tokens. The
target LLaVA model verifies the draft tokens by comput-
ing the target distribution which is conditioned on both
the image embeddings V1:m and the text token embed-
dings X1:n, i.e., draft token ˆXn+j is accepted with prob-
ability
min
(
1, q( ˆXn+j|V1:m, X1:n, ˆXn+1:n+j−1)
p( ˆXn+j|X1:n, ˆXn+1:n+j−1)
)
.
Using the language-model-only draft model is more ef-
ficient than a draft model with LLaVA architecture since
1) it does not need an additional adapter as it does not
condition on the image embeddings for generating draft
tokens, and 2) it does not require the training of the
adapter. Figure 1 shows SPD with MLLM consisting
of the smaller draft language model doing autoregressive
generation followed by the large target model evaluating
the draft model predicted tokens in parallel while using
the image.
4. Experiments
We run experiments on three visual instruction tasks us-
ing SPD with LLaVA-7B [12] as our target model which
uses the LLaMA-7B model as the language-model back-
bone. We employ draft models that underwent different
stages of training with the size of the language part of
each draft model fixed to 115M.
2
Draft Model Candidates.
We train draft model of
size 115M which follow the LLaMA-2 architecture. We
follow the training pipeline of [6] to pre-train a draft
model from scratch and fine-tune the draft model on
instruction finetuning datasets using TVD++ loss [6].
We further fine-tune our draft model on a subset of
LLaVA Instruct 150K dataset [13].
For our experi-
ments, we consider the following four draft models after
each stage of training and finetuning: 1) base-LLaMA, a
draft LLaMA model after pre-training using next-token-
prediction loss on 600B English tokens, 2) chat-LLaMA,
an instruction fine-tuned draft LLaMA model follow-
ing [6] initialized with base-LLaMA draft model, and
3) fine-tuned-LLaVA (ft-llava), a fine-tuned LLaVA draft
model where the image adapter was initialized using
subcloning [17] of LLaVA-7B image adapter and the
language model was initialized from the chat-LLaMA
draft model (the model is then fine-tuned on LLaVA
dataset). We also include another draft model 4) fine-
tuned-LLaVA-text (ft-llava-text), which simply uses the
language model part of 3). Note that only the fine-tuned-
LLaVA draft model uses image information while all
other draft models only consume the text part of the
input prompt; when the draft model uses image infor-
mation, the vision encoder (CLIP-based [16]) is shared
with the target model to avoid re-computation of image
embeddings. The detailed parameters are given in Ap-
pendix A.1
Evaluation Tasks.
We focus on open-ended text gen-
eration and multiple choice question-answering with
reasoning to encourage a higher number of token gen-
eration, which is beneficial when using SPD. To this
end, we evaluate on 1) LLaVA Instruct 150K dataset
[13], 2) Image captioning task on images from COCO
dataset [11], and 3) Science QA (SQA) with chain-of-
thought (CoT) reasoning [14]. The system prompts set-
tings for all the tasks are described in Appendix A.2
Metrics.
The efficacy of SPD is evaluated with the
following metrics; 1) block efficiency (τ), the average
number of tokens generated per block (or target model
run), for a block of size γ and input x, the maximum
value of τ(x) can be γ + 1, block-size (γ) is also known
as draft length (DL) in some works; 2) memory-bound
speedup (MBSU), the hypothetical speedup achieved by
SPD for a given block efficiency τ(x) and a relative la-
tency c defined as ratio between number of parameters of
draft to target model, i.e., MBSU(x) = cτ(x)
cγ+1; 3) token
rate, the total number of tokens generated divided by
the total time of generation, giving an estimate of tokens
generated for per second. We measure these metrics on
various tasks using different block size γ in {3, 5}
DL=3
DL=5
1.8
2.0
2.1
2.3
2.4
MBSU
LLaVA-eval
DL=3
DL=5
1.5
1.7
1.8
2.0
2.1
COCO-Caption
DL=3
DL=5
2.0
2.1
2.3
2.4
2.5
SQA
DL=3
DL=5
1.9
2.1
2.3
2.4
2.6
Block Efficiency
DL=3
DL=5
1.6
1.8
2.0
2.1
2.3
DL=3
DL=5
2.1
2.3
2.4
2.6
2.8
DL=3
DL=5
1.0
1.1
1.2
1.4
1.5
Token Rate
DL=3
DL=5
0.8
1.0
1.1
1.3
1.4
DL=3
DL=5
1.1
1.2
1.3
1.4
1.5
base-llama
chat-llama
ft-llava-text
ft-llava
Figure 2. MBSU, block efficiency and token rate (relative to
auto-regressive generation) for SPD are depicted; We consider
LLaVA-eval, COCO-Caption and SQA datasets for evaluation;
For draft models, base-LLaMA, chat-LLaMA, ft-LLaVA-text,
ft-LLaVA are considered, we consider three text-only draft
models and a single text and image draft model; For draft
length (DL) (or block size of SPD), we consider either 3 or
5.
Decoding.
We use greedy decoding for all experi-
ments so that the SPD generation is identical to the target
model’s autoregressive generation. We leave it as future
work to explore sampling-based decoding (varying tem-
perature, varying top-p, top-k) in the context of SPD for
MLLMs.
Results.
Our results show that using SPD with LLaVA
7B target model gives considerable speedup in output
generation, and we emphasize that when using a draft
model without any image information, SPD can still give
considerable and competitive speedup to that of a draft
model using image information.
From Figure 2 (top and middle plots), we observe that
using SPD gives more than 2× gains in terms of block
efficiency and MBSU. The performance trend when in-
creasing the block size from 3 to 5 for each task is simi-
lar with the exception for SQA task where base-llama
draft model performs better than other text-only draft
models for block size = 5. For LLaVA-eval task on
both block sizes (either 3 or 5), the ft-llava-text draft
model performs the best closely followed by ft-llava.
For COCO-caption task, ft-llava performs best, followed
by ft-llava-text for both block sizes. Lastly, for the SQA
task, for block size=3, ft-llava draft model performs the
best followed by ft-llava-text while for block-size=5, ft-
3
Figure 3. Example 1
Input prompt:
〈s〉A
chat between a curi-
ous user and an ar-
tificial intelligence as-
sistant.
The assistant
gives helpful, detailed,
and polite answers to
the
user’s
questions.
USER: 〈image〉
Provide a detailed de-
scription of the given image ASSISTANT:
Output generation: The image features a white plate filled
with a variety of vegetables, including several carrots, green
beans, and broccoli. The carrots are scattered across the
plate, with some placed closer to the center and others to-
wards the edges. The green beans are also spread out, with
some near the center and others closer to the edges. The
broccoli is located towards the top left corner of the plate.
The arrangement of the vegetables creates a colorful and
appetizing display.
Figure 4. Example 2
Input
prompt:
〈s〉A chat between
a
curious
user
and
an
artificial
intelligence
assis-
tant.
The assistant
gives
helpful,
detailed, and polite
answers
to
the
user’s
questions.
USER: 〈image〉
Provide a detailed
description
of
the
given
image
ASSISTANT:
Output generation: The image depicts a kitchen counter
filled with various food items and kitchen utensils. There
are several bowls placed on the counter, with one large bowl
towards the left side and two smaller bowls closer to the
center. A few cups can be seen as well, with one near the
left side of the counter and another towards the right side.
In addition to the bowls and cups, there are multiple bottles
scattered across the counter, with some placed near the cen-
ter and others towards the right side. A spoon is also visible
on the counter, located near the center.
The kitchen counter is surrounded by various appliances, in-
cluding an oven on the right side, a sink in the background,
and a refrigerator on the left side. A microwave can be seen
above the counter, and a knife is placed near the right edge
of the counter.
Figure 5. SPD examples on COCO-caption task
llava draft model performs best followed by base-llama.
In addition, all our draft models show the improved to-
ken rate compared to auto-regressive generation in Fig-
ure 2 (bottom) with block size 3 giving better token rate
than block size 5, thus, SPD generates more token to-
kens per second than autoregressive decoding. The to-
ken rate shown corresponds to the ratio of the token rate
of SPD using a particular draft model to the token rate
of autoregressive generation using target model.
We further provide qualitative results on the COCO-
captioning task to show the tokens accepted during a
generation process when using the fine-tune-LLaVA-
text draft model so no image information used by draft
model in Figure 5.
Based on the output generations
in the figure, where tokens in blue and underlined are
the accepted tokens, we observe that the draft model
can predict common words and propositions, along with
halves of words. For example, the draft model can pre-
dict “tables” given “vege”. Similarly in the second ex-
ample, given the context and additional token “app”, the
draft model was able to predict “liances”. We believe
in general open-ended text generation has several tokens
comprising of common words, propositions, and word
completions which do not require knowledge of image
tokens, thus, even a draft model without using image
information gives competitive performance. Moreover,
draft model can also predict the repetition of certain to-
kens once they have been generated. For example, in the
second image the word “counter” and “bowls” can be
predicted by the draft model multiple times once it has
been generated by the target model. Lastly, performing
more rigorous training on a small multi-modal language
model is left as our future work.
5. Conclusion
In this paper, we present the first effort towards using
speculative decoding for accelerating inference when
using multi-modal large language models, specifically
for image-text domain. We show that using the text-
only draft model achieves performance competitive to
using a draft model utilizing image features. We per-
form various experiments on different visual question-
answering tasks focusing on generating higher number
output tokens: open-ended text generation and text gen-
eration with reasoning using different draft models (text-
only and image-text). We achieved significant speedup
of upto 2.37× for text-only draft model and marginal
better speedup for image-text draft model, empirically
showing the potential of using SPD for MLLMs.
Our work opens several future avenues owing to the
general framework presented. Our work can be extended
to other target models such as BLIP-2 [10], MiniGPT-
4 [22] and OpenFlamingo [1], and other modalities
such as audio [4] which are also bottlenecked by auto-
regressive generation. Furthermore, recent advancement
in SPD algorithm to tree-based decoding can also be
used following [2, 7, 15, 20] to further increase gener-
ation speed.
4
References
[1] Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel,
Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan
Bitton, Samir Gadre, Shiori Sagawa, et al.
Open-
flamingo: An open-source framework for training large
autoregressive vision-language models. arXiv preprint
arXiv:2308.01390, 2023. 1, 4
[2] Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu
Peng,
and
Tri
Dao.
Medusa:
Simple
frame-
work for
accelerating
llm
generation
with
multi-
ple decoding heads.
https://github.com/
FasterDecoding/Medusa, 2023. 1, 4
[3] Charlie Chen, Sebastian Borgeaud, Geoffrey Irving,
Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper.
Accelerating large language model decoding with specu-
lative sampling. arXiv preprint arXiv:2302.01318, 2023.
1
[4] Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shil-
iang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou.
Qwen-audio: Advancing universal audio understanding
via unified large-scale audio-language models.
arXiv
preprint arXiv:2311.07919, 2023. 4
[5] Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang.
Breaking the sequential dependency of LLM inference
using lookahead decoding, 2023. 1
[6] Raghavv Goel, Mukul Gagrani, Wonseok Jeon, Juny-
oung Park, Mingu Lee, and Christopher Lott.
Direct
alignment of draft model for speculative decoding with
chat-fine-tuned llms. arXiv preprint arXiv:2403.00858,
2024. 3
[7] Wonseok Jeon, Mukul Gagrani, Raghavv Goel, Jun-
young Park, Mingu Lee, and Christopher Lott.
Re-
cursive speculative decoding: Accelerating llm infer-
ence via sampling without replacement. arXiv preprint
arXiv:2402.14160, 2024. 1, 4
[8] Sehoon Kim,
Karttikeya Mangalam,
Jitendra Ma-
lik, Michael W Mahoney, Amir Gholami, and Kurt
Keutzer. Big little transformer decoder. arXiv preprint
arXiv:2302.07863, 2023. 1
[9] Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast
inference from transformers via speculative decoding. In
Proceedings of the 40th International Conference on Ma-
chine Learning (ICML), 2023. 1
[10] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.
Blip-2: Bootstrapping language-image pre-training with
frozen image encoders and large language models. In
International conference on machine learning, pages
19730–19742. PMLR, 2023. 4
[11] Tsung-Yi Lin, Michael Maire, Serge Belongie, James
Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and
C Lawrence Zitnick. Microsoft coco: Common objects
in context. In Computer Vision–ECCV 2014: 13th Euro-
pean Conference, Zurich, Switzerland, September 6-12,
2014, Proceedings, Part V 13, pages 740–755. Springer,
2014. 1, 3
[12] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee.
Improved baselines with visual instruction tuning. arXiv
preprint arXiv:2310.03744, 2023. 2
[13] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae
Lee. Visual instruction tuning. Advances in neural infor-
mation processing systems, 36, 2024. 1, 3, 6
[14] Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei
Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark,
and Ashwin Kalyan. Learn to explain: Multimodal rea-
soning via thought chains for science question answer-
ing. In The 36th Conference on Neural Information Pro-
cessing Systems (NeurIPS), 2022. 1, 3, 6
[15] Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao
Cheng, Zeyu Wang, Rae Ying Yee Wong, Zhuoming
Chen, Daiyaan Arfeen, Reyna Abhyankar, and Zhihao
Jia.
SpecInfer: Accelerating generative LLM serving
with speculative inference and token tree verification.
arXiv preprint arXiv:2305.09781, 2023. 1, 2, 4
[16] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya
Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas-
try, Amanda Askell, Pamela Mishkin, Jack Clark, et al.
Learning transferable visual models from natural lan-
guage supervision. In International conference on ma-
chine learning, pages 8748–8763. PMLR, 2021. 3, 6
[17] Mohammad Samragh,
Mehrdad Farajtabar,
Sachin
Mehta, Raviteja Vemulapalli, Fartash Faghri, Devang
Naik, Oncel Tuzel, and Mohammad Rastegari. Weight
subcloning: direct initialization of transformers using
larger pretrained ones. arXiv preprint arXiv:2312.09299,
2023. 3
[18] Andrea Santilli, Silvio Severino, Emilian Postolache,
Valentino Maiorca, Michele Mancusi, Riccardo Marin,
and Emanuele Rodol`a. Accelerating transformer infer-
ence for translation via parallel decoding. arXiv preprint
arXiv:2305.10427, 2023. 1
[19] Noam Shazeer. Fast transformer decoding: One write-
head is all you need. arXiv preprint arXiv:1911.02150,
2019. 1
[20] Ziteng Sun, Ananda Theertha Suresh, Jae Hun Ro, Ah-
mad Beirami, Himanshu Jain, and Felix Yu. SpecTr: Fast
speculative decoding via optimal transport. In Advances
in Neural Information Processing Systems (NeurIPS),
2023. 1, 4
[21] Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM
Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-
shot learning with frozen language models.
Advances
in Neural Information Processing Systems, 34:200–212,
2021. 1
[22] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and
Mohamed Elhoseiny.
Minigpt-4:
Enhancing vision-
language understanding with advanced large language
models. arXiv preprint arXiv:2304.10592, 2023. 1, 4
5
A. Appendix
A.1. Model Configurations
The LLaVA-7B model uses: (i) vision encoder, (ii) multi-layer perceptron (MLP) based image adapter/projector, and
(iii) LLaMA 7B language model. The visual encoder is CLIP ViT-L/14 with details present in [16], the MLP-based
image adapter has 2 linear layer with following sizes: 1024 × 4096 and 4096 × 4096. For the scenario when draft
model also has image adapter the sizes are 1024 × 1024 and 1024 × 1024.
The following configurations are used for our target and draft language model part which follows the LLaMA
architecture:
Table 1. Draft and target model configurations
target (7B)
draft (115M)
Layers
32
4
Attention heads
32
8
Intermediate dim
11,008
2,816
Hidden dim
2,048
1,024
Activation
SiLU
SiLU
A.2. System Prompts
We use the following systems prompts for the respective task. The special image token is used to include the image
data (<image>)
LLaVA-eval. We follow the prompt style given in [13], LLaVA has multiple questions and responses which we
divide into different samples.
<s> A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed,
and polite answers to the user’s questions. USER: <image>
Question Q1 ASSISTANT: response R1. USER: Question Q2 . . . .
COCO-caption. As COCO dataset doesn’t have any question prompts, we prompted the model with a prompt
similar to above.
<s> A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed,
and polite answers to the user’s questions. USER: <image>
Provide a detailed description of the given image ASSISTANT:
Science QA. We follow the prompt style provided in [14] with a single in-context example of the question, choices,
answer and reasoning to enable Chain-of-Thought (CoT) reasoning. Additionally we only consider the test samples
which have an associated image.
Question: question :Iques
i
Options: (0) option : Iopt
i1
(1) option : Iopt
i2
(2) option : Iopt
i3
Context: context : Icont
i
Answer: The answer is Ians
i
. BECAUSE: lectureIlect
i
explanation : Iexp
i
< image >
Question: question :Iques
test
Options: (0) option : Iopt
test,1 (1) option : Iopt
test,2 (2) option : Iopt
test,3
Context: context : Icont
test
Answer: The answer is
where, the subscript i is for in-context example.
In the SQA paper, the context field is provided by generating a caption for the associated image using an image
captioning model, however, these captions were often simple and didn’t provide a detailed description of the image
6
which is needed for answering the question. For this reason, the context field is filled with “hint” field provided in
the SQA dataset. For the in-context sample we choose a sample without any associated image as the target LLaVA
7B cannot consume multiple images. We leave it as a future work to experiment SPD with more than 1 in-context
examples.
7