Question:
According to the motivations presented in the paper, what key limitations do existing attention-based visual token pruning methods encounter?
A. They tend to retain redundant tokens due to token similarity and clustering.
B. They primarily focus on text tokens, neglecting important visual details.
C. They exhibit positional bias, causing essential tokens to be ignored.
D. They rely on training-based pruning methods, increasing inference complexity.

Answer: AC

However, these pruning techniques are typically constrained to around 50% token reduction, as further pruning often results in substantial performance degradation across tasks. As shown in Figure 1, we demonstrate that attention-only methods represented by FastV [9] have a tendency to keep tokens corresponding to the lower part of the image and often select clusters of tokens with high similarity. Our in-depth analysis of image tokens has highlighted two primary issues that constrain current attention-based pruning methods: positional bias of image attention and information redundancy in similar image tokens. These factors become especially problematic when aiming for more aggressive token pruning, and sharply degrading model performance.

Multi-Cue Adaptive Visual Token Pruning for Large Vision-Language Models
Bozhi Luan1
Wengang Zhou1
Hao Feng1
Zhe Wang2
Xiaosong Li2
Houqiang Li1
1 University of Science and Technology of China, 2 Huawei Technologies
{bzluan,haof}@mail.ustc.edu.cn, {zhwg,lihq}@ustc.edu.cn, {wangzhe226,lixiaosong20}@huawei.com
Abstract
As the computational needs of Large Vision-Language Mod-
els (LVLMs) increase, visual token pruning has proven
effective in improving inference speed and memory effi-
ciency.
Traditional pruning methods in LVLMs predom-
inantly focus on attention scores to determine token rel-
evance, overlooking critical aspects such as spatial posi-
tion and token similarity. To this end, we introduce Adapt-
Prune, a novel plug-and-play training-free pruning method
that builds on conventional attention-based pruning by inte-
grating spatial distance and token similarity with an adap-
tive NMS approach. Our method is based on several ob-
served phenomena in large models: the positional bias in
the model’s image attention and the redundancy of token
information ignored by previous approaches. By integrat-
ing attention, spatial, and similarity information, our ap-
proach ensures a comprehensive evaluation of token impor-
tance and substantially refines the pruning decisions. Our
method has been extensively tested across various LVLMs
and benchmarks, confirming its robustness and adaptabil-
ity. The results demonstrate that AdaptPrune consistently
outperforms existing methods across various pruning ra-
tios.
Code is available at https://github.com/
bzluan/AdaptPrune.
1. Introduction
Breakthroughs in Large Language Models (LLMs) [3, 12,
47] have revolutionized natural language processing with
impressive performance across a diverse range of tasks
[20, 50]. Building upon these advancements, Large Vision-
Language Models (LVLMs) [29, 30] extend LLMs by con-
verting images into hundreds or thousands of sequential
visual representations and combined with text prompts.
With the advancement of powerful visual understanding and
question-answering capabilities, there has also been a rapid
increase in computational complexity and memory usage.
A pioneering study FastV [9] has highlighted that while
image tokens account for the majority of computational
load, they receive far less attention than text tokens in
FastV
Input
Answer: Strawberry
Ratio=0.9
Answer: Apple
Ratio=0.5
Ours
Question: What is
that red fruit?
Question: What is
that red fruit?
Answer: Apple
Answer: Apple
Figure 1. Comparison between FastV [9] and our AdaptPrune at
different pruning ratios. FastV [9] utilizes only attention scores for
token pruning, while AdaptPrune incorporates attention, spatial,
and similarity information for a more holistic investigation.
the deeper layers of Large Vision-Language Models.
A
phenomenon known as “attention anchoring” [9, 52] has
also been identified, where attention becomes concentrated
on just a few specific image tokens, leaving most tokens
with substantially reduced focus. Quantitative analyses in
some studies [26, 55] further demonstrate that core attention
points show minimal variation across layers and generation
steps, reinforcing this observation.
Based on the above insights, some methods [9, 53, 55,
58] have been developed to prune image tokens with low at-
tention scores at specific layers, reducing the computational
load to accelerate inference while minimizing the impact on
accuracy. Image tokens are re-evaluated based on their aver-
age received attention scores. Tokens falling below a prede-
fined attention score threshold are then selectively discarded
in subsequent layers, streamlining the process by focusing
on the most impactful tokens. They not only bypass the
computational demand of the self-attention module but also
the Feed-Forward Network (FFN) module in deeper layers.
However, these pruning techniques are typically con-
strained to around 50% token reduction, as further pruning
arXiv:2503.08019v1  [cs.CV]  11 Mar 2025
often results in substantial performance degradation across
tasks. As shown in Figure 1, we demonstrate that attention-
only methods represented by FastV [9] have a tendency to
keep tokens corresponding to the lower part of the image
and often select clusters of tokens with high similarity. Our
in-depth analysis of image tokens has highlighted two pri-
mary issues that constrain current attention-based pruning
methods: positional bias of image attention and information
redundancy in similar image tokens. These factors become
especially problematic when aiming for more aggressive to-
ken pruning, and sharply degrading model performance.
To address these challenges, we propose AdaptPrune, a
pioneering training-free pruning method for image tokens
that incorporates three critical cues: attention scores, spatial
positions, and token similarity. As depicted by the compar-
ison in Figure 1, in contrast to FastV’s tendency to preserve
clusters of similar tokens and focus on the lower part of the
image, our AdpatPrune selects tokens that better represent
diverse semantic regions for more information.
We innovatively reframe the visual token pruning prob-
lem as a Non-Maximum Suppression (NMS) task. Draw-
ing inspiration from studies in the object detection area
[5, 21, 31, 32], we treat each token as a candidate ”detec-
tion” point, where its importance score reflects confidence
and similarity indicates information density. AdaptPrune
builds on attention scores and develops an adaptive NMS
approach, suppressing tokens within a specified range based
on spatial distance and token similarity. Our method initi-
ates token pruning from a single layer, applying the pruning
strategy consistently across subsequent layers and tokens
without further adjustments to avoid introducing additional
factors that could affect pruning outcomes.
We validate our findings through comprehensive statisti-
cal and visual analyses. In our experiments, we experiment
on 5 well-known LVLMs and evaluate our AdaptPrune on
multiple tasks including Image Captioning, General VQA,
Text-based VQA, and Multimodal Reasoning. Extensive
experiments demonstrate that AdaptPrune effectively ad-
dresses previous limitations, significantly minimizing per-
formance loss even at higher pruning ratios across diverse
models and benchmarks.
We summarize our contributions as follows:
• We identify and analyze the positional bias and informa-
tion redundancy phenomena in attention distribution, em-
phasizing that spatial information and token similarity are
also needed for a more holistic strategy.
• We propose AdaptPrune, an effective plug-and-play vi-
sual token pruning method that reframes the problem as
an adaptive NMS task, integrating three critical cues: at-
tention score, spatial distance, and token similarity.
• We perform extensive experiments on diverse vision-
language benchmarks based on different LVLMs to verify
the effectiveness of our AdaptPrune.
2. Related Work
Large Vision-Language Models (LVLMs). By employ-
ing a cross-modal projector that aligns visual encoders
[41] with Large Language Models [3, 8, 12–14, 42, 47],
LVLMs effectively bridged visual and language understand-
ing [29, 30, 62].
Despite their effectiveness in various
vision-language tasks, LVLMs still encounter challenges
such as detailed information understanding and hallucina-
tions [56]. To address this, recent advancements focused on
increasing image resolution [4, 10, 11], which greatly en-
hanced the performance on fine-grained tasks like TextVQA
[45] and DocVQA [38]. However, these approaches sub-
stantially raised the number of image tokens, increasing
from 576 in LLaVA-1.5 [30] to over 2,000 in LLaVA-NEXT
[29] and even exceeding 10,000 in InternVL2 [10], re-
sulting in significantly higher computational demands and
memory usage. Even with the KV Cache increasing the
speed of the decoding stage, the computational complexity
still scales quadratically with the token length in the prefill
stage. Consequently, research into the effective pruning of
redundant visual tokens becomes essential for optimizing
the efficiency of LVLMs in real-world applications.
Token Reduction. Recent advancements in adaptive at-
tention have improved computational efficiency in LLMs
by reducing redundant tokens dynamically [16, 23, 39, 46,
57]. StreamingLLM [52] retained fixed-position caches for
long-context processing, while FastGen [17] adapted KV
Cache management to retain important tokens based on re-
cent attention patterns. H2O [61] and ScissorHands [34]
further optimized inference by selectively pruning KV pairs
and retaining tokens with consistently high attention. Many
trainable methods [7, 15, 24, 27, 43, 44, 54] focused on vi-
sual token reduction in vision transformers [41] before im-
age tokens are projected into LLMs. However, when di-
rectly applied to LVLMs without training, these methods
lead to performance degradation because they fail to incor-
porate LLM’s understanding of the textual context.
FastV [9] proposed an image token pruning method for
LVLMs to address inefficiencies in image attention. This
method pruned tokens based on their attention scores start-
ing from a specific layer and achieved minimal impact on
overall performance. FastV [9] has been validated across
various benchmarks and models, with additional solutions
introduced ensuring compatibility with KV Cache. Later
methods [2, 28, 35, 49, 53, 55, 58] explored dynamic prun-
ing for each token or multi-layer pruning based on attention
scores. Despite its widespread usage in existing methods,
the ranking criteria for token pruning have received limited
exploration, which we aim to further improve.
Unlike most methods developed after FastV, which fo-
cus on multilayer or dynamic pruning for different output
tokens, we aim to explore a more fundamental and crucial
aspect of token pruning which is the cues of token ranking.
Layer 2
Layer 16
Layer 32
Average attention distribution on Nocaps dataset
Average attention distribution on TextVQA dataset
Figure 2. Visualization of attention positional bias phenomenon.
The average attention distribution projected onto image patches
exhibits a fixed pattern across different layers and tasks.
By addressing the phenomena we identified, We develop
a more holistic ranking standard for visual token pruning.
The detailed elaboration follows below.
3. Observations and Insights
In this section, we present several phenomena observed
through experiments across different datasets. Prior work
such as FastV [9] has revealed that image tokens consti-
tute a substantial portion of input tokens, and they receive
disproportionately less attention in deeper layers. Further
studies [9, 26, 58, 60] have shown that the model consis-
tently focuses on a subset of image tokens across multiple
layers and generation steps. Building upon these significant
findings, our experimental observations lead to new insights
into the attention mechanisms of LVLMs, motivating us to
propose strategies to improve token selection.
Positional Bias in Attention Distribution. As shown in
Figure 2, we observe fixed motifs and patterns in the atten-
tion score distribution of image tokens which will cause po-
sitional bias in token pruning. We use 1000 samples from
the Nocaps [1] dataset of the Image Captioning task and
the TextVQA [45] dataset of the Text-based VQA task for
statistics. Despite these samples coming from entirely dif-
ferent tasks, the model exhibits an almost identical atten-
tion distribution motifs for image tokens on the same layer.
Across different layers, the shapes changes slightly but the
tokens with the higher attention remain the same.
We observe that higher attention is distributed at the end
of an image sequence. We interpret this phenomenon as a
result of LVLMs inheriting the attention characteristics of
their LLM component. Image patches are encoded as a to-
ken sequence and processed almost the same as a text para-
graph during the pipeline of LVLMs. However, unlike text
tokens, where boundary locations usually contain key infor-
Input
Attention-only Prune
Attention score
Similarity of every token with the marked token
Figure 3.
Visualization of the information redundancy phe-
nomenon. The tokens with the highest attention score are kept,
most of which are adjacent and have high similarity.
mation, the key information of an image is usually located
in the central region [6]. Pruning methods that rely solely on
attention scores [9, 53, 55, 58] tend to keep tokens in certain
fixed positions and discard other essential tokens due to the
bias of the model. As a result, such pruning strategies lead
to performance degradation as the pruning ratio increases.
Information Redundancy in High Attention Tokens.
When token selection relies solely on attention scores, the
model tends to retain clusters of similar tokens, resulting in
redundant information and inefficient use of computational
resources. As shown in Figure 3, adjacent tokens with simi-
lar visual features in LVLMs often receive similar attention
scores, particularly in areas with flat backgrounds or repet-
itive textures. Spatially close tokens capture overlapping
features, making it difficult for the attention mechanism to
differentiate unique or critical information. While attention-
based pruning effectively captures central focus areas, it
often overlooks important background context, leading to
wasted attention on redundant information.
To improve
pruning efficiency and model performance, it is essential
to incorporate similarity and informational uniqueness into
the token selection process.
4. Method
As shown in Figure 4, our AdaptPrune reframes visual to-
ken pruning as a Non-Maximum Suppression (NMS) prob-
lem. Drawing inspiration from object detection [5, 31, 32],
we treat each token as a candidate “detection” point with an
importance score, which serves as a confidence measure. In
our AdaptPrune, tokens with high similarity and proximity
to the selected token are suppressed, retaining only diverse
and essential information for downstream tasks.
Adapt-
Prune optimizes inference by preserving token diversity and
reducing redundancy, ultimately enhancing computational
<system prompt>
+
What is funny
about the image?
Reshape with
Distribution
Correction
AdaptPrune
Vision Encoder & MLP
Tokenizer
The image is funny because the
rabbit is wearing sunglasses.
Prune R%
Image Tokens
Adaptive NMS
Distance Decay
Similarity Decay
System token
Image token
Text token
Transformer Layer
AdaptPrune
Transformer Layer
Attention score on
image patches
Final score after
NMS process
Figure 4. An overview of our AdaptPrune framework. AdaptPrune employs a simple and effective one-layer pruning strategy and incor-
porates an adaptive NMS algorithm to optimize the pruning process. AdaptPrune dynamically adjusts the suppression process using three
critical cues: attention score, spatial distance, and token similarity.
efficiency without sacrificing model performance.
4.1. Distribution Correction
Based on our findings of attention’s positional bias, we pro-
pose adjusting the distribution of attention scores across im-
age patches. Figure 8 of an influential study [6] points out
that more information is distributed around the center of the
image and suggests that the Gaussian function provides a
good approximation. We apply a Gaussian mask centered
at the image’s midpoint to the attention scores. This correc-
tion results in a more balanced attention distribution.
4.2. Factor Measurement
We employ three essential cues to calculate the scores of
tokens in different aspects, explained below in terms of their
measurement and computation.
Attention Scores Measurement. Like many existing
methods [9], we utilize the attention values from the pre-
vious layer in the LVLM’s current token generation step to
represent the importance of image tokens. This approach in-
tuitively reflects the dependency of the current output token
on all visual tokens and has been demonstrated by numer-
ous prior studies to be an effective scoring mechanism.
Distance Measurement. Our method refines attention
scoring by applying Soft NMS with two-dimensional spatial
distance. To calculate this distance, we map image tokens
back to their original patch positions. Unlike traditional
NMS, which directly removes overlapping areas, Soft NMS
gradually reduces scores based on token proximity, which
prevents the forced removal of dense neighboring informa-
tion. By leveraging a Gaussian decay function, we can con-
trol the range and intensity of distance suppression. This ap-
proach more efficiently chooses diverse regions across the
image to capture representative information.
Token Similarity Measurement.
In high-resolution
Text-based VQA benchmarks such as TextVQA [45] and
DocVQA [38], answers are often concentrated in small,
high-density regions, with each token representing unique
details of the answer. This contrasts with tasks like Image
Captioning or General VQA, which require broader con-
textual understanding. Drawing inspiration from the Zero-
TPrune [51] method, we incorporate token similarity as an
additional filtering factor. By calculating the cosine simi-
larity between key vectors, we implement an adaptive NMS
that dynamically adjusts decay parameters based on both
spatial distance and similarity. This adaptive pruning strat-
egy effectively removes redundant tokens while retaining
critical ones, enabling the model to better capture essential
details at any density within the answer region.
4.3. Adaptive NMS
We propose an adaptive NMS algorithm suitable for can-
didate token selection in visual token pruning to integrate
these three essential cues. At each iteration of the NMS pro-
cess, we select the token ti with the highest attention score
si. For the currently selected token, we apply a suppression
function to adjust the scores of nearby tokens tj within a
spatial range. The adaptive suppression function combines
decay factors based on spatial distance and token similarity.
We compute the spatial decay function Dspatial and simi-
larity decay function Dsimilarity as follows:
Dspatial = exp

−d(ti, tj)2
2σ2
d

,
(1)
Dsimilarity = exp

−(1 −cos(keyi, keyj))2
2σ2s
!
.
(2)
The decay functions are calculated using spatial distance
d(ti, tj) and cosine similarity cos(keyi, keyj) between key
vectors of the tokens, each modulated by decay factors σd
and σs, respectively. The updated score sj for each nearby
token tj is then calculated as:
sj : = sj · (1 −Dspatial · Dsimilarity) .
(3)
This function adaptively suppresses token scores using a
multiplicative decay. Initial scores sj are adjusted so that sj
reflects its spatial proximity and feature resemblance to ti.
This approach prioritizes spatially sparse tokens with dis-
tinct features, enhancing token selection flexibility through
the adaptable decay parameters σd and σs.
At each step, we select the highest-scoring token that
hasn’t yet been chosen and retain it. We then suppress the
scores of surrounding tokens based on their spatial distance
and similarity to the selected token. This iterative process
continues until the pruning preservation ratio is met, with
each newly selected token reducing the scores of its neigh-
bors. This approach ensures that we retain only the most in-
fluential and representative tokens while effectively pruning
those with redundant or less impactful information. Algo-
rithm 1 shows the pseudocode for AdaptPrune.
4.4. Discussion
To summarize, our AdaptPrune introduces an advanced
training-free token pruning strategy that unifies the cues of
attention scores, spatial distance, and token similarity into
cohesive importance ranking criteria of a balanced prun-
ing approach. Applying AdaptPrune during the prefilling
phase accelerates the prefilling speed while reducing the
KV Cache length, which further enhances the decoding pro-
cess. Our computational efficiency is almost identical to
that of FastV [9], as both are single-layer pruning methods.
Our adaptive strategy minimizes floating-point operations
per second (FLOPs), effectively enhancing both inference
speed and memory efficiency.
Unlike previous methods that primarily consider atten-
tion scores, AdaptPrune leverages spatial and similarity in-
formation to retain a more diverse token set, ensuring min-
imal impact on model accuracy even with substantial token
reduction. By benefiting both the prefilling and decoding
phases, this method offers a comprehensive optimization
for long-context multimodal tasks, balancing memory ef-
ficiency and high-quality output retention.
5. Experiments
We evaluate the effectiveness of our method across 5 differ-
ent models and 12 different benchmarks. We aim to provide
Algorithm 1 AdaptPrune
Require: Tokens T, scores S, positions P, keys K, dis-
tance decay parameter σd, similarity decay parameter
σs, pruning ratio ratio.
Ensure: Retained tokens R.
1: R ←∅;
2: N ←ratio × |T|;
3: S ←GaussianCorrection(S, P);
4: while |R| < N do
5:
i ←arg maxk∈T S[k];
6:
R ←R ∪{T[i]};
7:
for j ∈Neighbors(i) do
8:
d ←∥P[i] −P[j]∥;
9:
sim ←cos(K[i], K[j]);
10:
Dspatial ←exp

−d2
2σ2
d

;
11:
Dsimilarity ←exp

−(1−sim)2
2σ2s

;
12:
S[j] ←S[j] · (1 −Dspatial · Dsimilarity);
13:
end for
14: end while
15: return R.
a comprehensive comparison of the performance of visual
token pruning across diverse multimodal tasks.
5.1. Model Settings
For a fair comparison and efficient validation, we employ
the LMMs-Eval [59] evaluation framework across all exper-
iments. The models used include LLaVA-1.5 (7B and 13B
variants) [29], LLaVA-NEXT-7B [10], and InternVL2 (2B
and 8B variants) [10], with configurations based on the their
official settings. Detailed model architectures and settings
are provided in the supplementary material.
5.2. Datasets
To demonstrate the broad applicability of our approach, we
use several representative benchmark datasets that cover
a wide spectrum of task domains for evaluation, includ-
ing Nocaps [1] and Flickr30K [40] for Image Captioning,
VQAv2 [18], GQA [22], OK-VQA [36] and VizWiz [19]
for General Visual Question Answering, TextVQA [45],
ChartQA [37] and DocVQA [38] for Text-based Visual
Question Answering and POPE [25], MME and MMBench-
EN [33] for Multimodal Reasoning. Detailed descriptions
of the datasets and corresponding evaluation metrics are
provided in the supplementary material.
5.3. Main Results
We begin by comparing the pruning result of FastV [9]
and AdaptPrune across 5 LVLMs and 12 benchmarks at a
high pruning ratio. Then we show experiments with various
pruning ratios and qualitative visualizations to showcase the
Method
Image Caption
General VQA
Text-based VQA
Multimodal Reasoning
Nocaps Flickr30K VQAv2 GQA OK-VQA VizWiz TextVQA ChartQA DocVQA POPE MME
MMB
LLaVA-1.5-7B [29]
105.8
75.2
76.6
61.9
53.4
54.2
45.9
18.1
28.1
87.0
1508.2
64.1
+ VTW [28] (K=5)
6.1
6.9
42.7
39.5
20.0
50.3
8.1
11.6
10.5
46.0
635.5
21.0
+ FastV [9]
74.2
47.4
63.8
51.0
43.9
54.9
38.5
14.6
18.0
69.7
1217.8
58.6
+ AdaptPrune
99.3
70.5
72.3
58.0
49.1
55.9
40.1
17.0
19.4
79.3
1393.2
59.4
LLaVA-1.5-13B [29]
109.4
79.5
79.2
63.3
58.3
56.6
48.7
18.1
30.3
87.1
1528.8
68.7
+ VTW [28] (K=5)
5.0
4.8
47.0
39.7
22.4
50.2
8.2
13.2
11.0
50.0
623.7
22.4
+ FastV [9]
94.6
65.4
68.7
54.6
51.4
57.2
38.3
15.2
17.5
74.8
1353.4
63.3
+ AdaptPrune
103.8
75.2
73.5
57.9
53.1
57.9
42.0
15.4
19.8
80.5
1461.2
65.1
LLaVA-NEXT-7B [29]
88.3
68.4
81.8
64.2
44.2
60.6
64.8
54.8
74.4
87.6
1519.5
67.2
+ FastV [9]
72.7
52.2
71.3
56.6
38.3
57.4
48.2
30.0
36.8
79.7
1352.7
61.3
+ AdaptPrune
82.1
61.7
76.7
59.6
41.0
58.9
55.8
33.8
45.0
84.3
1428.3
63.0
Table 1. Experiments on LLaVA with 90% pruning ratio for visual tokens. Under this experiment setting, the FLOPs of the LLaVA-1.5
[29] baseline model are reduced by around 81%, while the FLOPs of the LLaVA-NEXT [29] baseline model are reduced by around 87%.
Method
General VQA
Text-based VQA
Multimodal Reasoning
VQAv2
GQA
OK-VQA
VizWiz
TextVQA
ChartQA
DocVQA
POPE
MME
MMB
InternVL2-2B [10]
74.7
59.9
43.6
46.1
72.7
74.9
87.4
89.0
1444.6
72.3
+ FastV [9]
64.3
50.9
37.1
42.8
49.9
17.9
28.0
80.1
1339.3
65.2
+ AdaptPrune
66.5
52.1
39.2
44.2
52.7
22.8
31.1
82.5
1313.5
68.2
InternVL2-8B [10]
78.9
62.7
52.2
61.0
77.0
82.4
92.0
87.9
1628.7
82.0
+ FastV [9]
67.4
53.6
45.3
58.2
55.7
28.2
33.8
79.6
1510.0
71.2
+ AdaptPrune
71.6
55.7
47.5
59.9
57.4
32.8
42.9
82.1
1516.7
75.6
Table 2. Experiments on InternVL with 90% pruning ratio for visual tokens. InternVL [10] model FLOPs are reduced by around 88%.
pruning effects of our method. To ensure a fair comparison,
we align key variables such as the pruning layer and pruning
ratio across both methods. Specifically, the pruning layer is
set to 3 following the optimal setup of FastV [9].
Comparison on 90% Pruning Ratio. As shown in Ta-
bles 1 2 and 3, we evaluate 5 LVLMs using VTW [28],
FastV [9] and AdaptPrune. Since AdaptPrune introduces
a new ranking criterion in token selection process, we com-
pare its single-layer pruning performance with FastV and
VTW [28] for a fair evaluation. The evaluated LVLMs dif-
fer in model size, training data, and architecture. Since the
InternVL2 [10] framework does not include Image Caption-
ing tasks with CIDEr score [48], we omit those results. To
assess method resilience, we select a 90% pruning ratio to
test token pruning effectiveness, emphasizing AdaptPrune’s
ability to maintain performance with minimal token reten-
tion, unlike FastV, which struggles with aggressive pruning.
Our method outperforms FastV across all models and
benchmarks. Unlike previous attention-only methods that
suffer performance drops at high pruning ratios, our ap-
proach minimizes loss even under aggressive pruning.
Experiments on Nocaps [1], Flickr30k [40], and Gen-
eral VQA benchmarks show that our method retains more
informative tokens, preserving broader image content. As
shown in Figure 5 and 6, while FastV [9] suffers signifi-
cant losses in image semantic information under extreme
pruning, our analysis suggests that the semantic loss primar-
ily stems from suboptimal ranking functions. Many non-
informative tokens are included in the retained token set and
genuinely informative tokens are filtered out, which causes
the main performance decline. Our new pruning strategy
selects a more informative and representative token set and
preserves a wider array of image information.
For Text-based VQA benchmarks like TextVQA [45],
DocVQA [38], and ChartQA [37], where answers are of-
ten confined to small, information-dense regions, the prun-
ing algorithm must measure token density and redundancy.
This necessitates that the pruning algorithm should effec-
tively measure both the density and redundancy of token
information.
By incorporating token similarity into our
AdaptPrune, we dynamically control the suppression de-
gree, adjusting to varying levels of information clustering
and redundancy. This demonstrates the effectiveness of the
AdaptPrune strategy in leveraging multiple pruning cues to
preserve model performance across various conditions.
Comparison at Different Pruning Ratios. As shown
in Figure 5 We conduct experiments across a range of token
pruning ratios, from 0.5 to 0.95, to evaluate how well our
Figure 5. Performance comparison of FastV [9] and AdaptPrune on the LLaVA-1.5-7B model at different pruning ratios across eight
evaluation benchmarks. The green line represents the LLaVA-1.5-7B baseline results without pruning.
Input
Ratio=0.5
Ratio=0.9
Ratio=0.7
FastV
Ours
FastV
Ours
Figure 6. Visualization comparison between FastV [9] and our
AdaptPrune on LLaVA-1.5-7B [29] at different pruning ratios of
samples from Nocaps (upper) and TextVQA (lower). FastV’s re-
dundant tokens at fixed positions are marked with red bboxes.
method maintains model performance at varying levels of
token retention. By testing across these ratios, we aim to
understand not only the robustness of our approach under
different pruning intensities but also its adaptability across
models and benchmarks. Unlike prior approaches that show
steep performance degradation as pruning becomes more
aggressive, our method demonstrates flexibility by preserv-
ing key image semantic information, even at high pruning
ratios. This suggests that our method can dynamically ad-
Method
Time
GPU-Memory Score Latency
LLaVA-1.5-7B [29] 49:05
19G
87.0
0.327s
+ FastV [9]
27:30
14.4G
69.7
0.182s
+ AdaptPrune
31:25
14.5G
79.3
0.209s
Table 3. Real inference comparison on POPE dataset where the
output is only one token. Experiments are on LLaVA-1.5-7B [29]
on single 3090 GPU with the pruning ratio of 90%.
just to different levels of token sparsity, maintaining the
quality of outputs while improving efficiency.
Qualitative Comparison. As shown in Figure 6, to gain
deeper insights into the behavior of our method, we conduct
a detailed visualization analysis using the LLaVA-1.5-7B
model across all benchmarks. By sampling 1,000 images,
we can examine token selection and pruning patterns in a
more granular way. This analysis reveals how our approach
differs from previous methods in selecting informative to-
kens and filtering out redundant ones. Specifically, we ob-
serve that our method consistently retains tokens that cap-
ture critical spatial and semantic details, leading to richer
and more contextually accurate outputs. These visual com-
parisons illustrate how our adaptive token pruning strategy
enhances model efficiency without compromising on the es-
sential information needed for high-quality predictions.
5.4. Ablation Studies
To validate the effectiveness across the factors we intro-
duced in our AdaptPrune, we conduct in-depth ablation ex-
periments as detailed in Table 4 and 5. All ablation ex-
periments are conducted on LLaVA-1.5-7B and the perfor-
mance is assessed across benchmarks from different tasks.
Impact of Distribution Correction. We first evaluate a
simplified variant (Table 4 (b)) of our AdaptPrune, where
Method
Correction Distance Similarity Nocaps Flickr30K GQA OK-VQA DocVQA TextVQA POPE MMB
LLaVA-1.5-7B [29]
105.8
75.2
61.9
53.4
28.1
45.9
87.0
64.1
(a)
74.2
47.5
51.0
43.9
18.0
38.5
69.7
58.6
(b)
✓
95.9
68.8
55.4
48.2
18.2
37.5
77.9
59.8
(c)
✓
✓
98.9
70.0
56.2
47.1
17.3
36.9
78.8
60.1
(d)
✓
✓
✓
99.3
70.5
58.0
49.1
19.4
40.1
79.3
59.4
Table 4. Ablation study across factors introduced in our AdaptPrune. Method (a) represents FastV [9] and the term “Correction” stands for
our Distribution Correction operation. Experiments are conducted on LLaVA-1.5-7B [29] with the pruning ratio of 90%.
Method
GQA OK-VQA TextVQA MMB
LLaVA-1.5-7B [29]
61.9
53.4
45.9
64.1
+ FastV [9]
51.0
43.9
38.5
58.6
+ AdaptPrune
58.0
49.1
40.1
59.4
(a) + FitPrune [55]
45.9
32.3
27.0
47.3
(b) + Skip
54.7
45.4
35.3
59.1
(c) + Random
56.0
40.0
20.2
56.7
(d) + Random 3x3
56.2
42.2
22.7
56.2
(e) + Max-pooling 3x3
56.0
47.7
33.2
58.9
(f) + Average-pooling
48.4
38.1
32.1
54.7
(g) + The last 10%
41.3
18.6
9.2
27.5
Table 5. Ablation study on alternative pruning strategies. Experi-
ments are on LLaVA-1.5-7B [29] with the pruning ratio of 90%.
we only apply a Distribution Correction process to mini-
mize the positional bias effect we observed. Following this
correction, we prune tokens based on the adjusted attention
scores. Compared with FastV [9] that used the attention
score directly (Table 4 (a)), this approach improves perfor-
mance across all benchmarks. This validates our observa-
tion that the model’s positional attention bias phenomenon.
The biased attention distribution does not accurately reflect
the true importance of information. The performance im-
provements across all benchmarks demonstrate that the Dis-
tribution Correction process allows new scores to more ac-
curately represent the importance of visual information.
Impact of Distance Decay. Based on the simplified vari-
ant (Table 4 (b)), we further introduce the distance decay
to reduce the redundancy (Table 4 (c)). We observed that
the selected tokens tend to cluster together, which usually
means the information is highly similar. To address this,
we incorporate the distance factor into our attention-based
pruning strategy using a Soft NMS approach. This method
suppresses the scores of tokens in the surrounding area of
each selected token based on their distance, effectively pro-
moting a more even token distribution across the image.
Experiment results show significant performance improve-
ments in Image Captioning, General VQA, and Multimodal
Reasoning tasks.
However, on Text-based VQA bench-
marks TextVQA [45] and DocVQA [38], performance is
below the attention-only approach of FastV [9].
We at-
tribute this to the enforced spatial dispersion of selected
tokens, which leads to information loss in tasks where an-
swers are concentrated in very small regions.
Impact of Similarity Decay.
To accommodate tasks
with varying information densities, we integrate similarity
information into our method (Table 4 (d)). To ensure each
retained token represents distinct information, we adjust the
decay effect on neighboring tokens based on the similarity
of their key vectors. Thus we propose the full version of
our AdaptPrune approach. AdaptPrune suppresses the sur-
rounding tokens’ attention score of the selected tokens with
distance and similarity information. Experimental results
show that our AdaptPrune consistently outperforms FastV
[9] across all tasks and achieves the best performance com-
pared to incomplete versions on almost all benchmarks.
Comparison with Alternative Pruning Strategy. As
shown in Table 5, we experimented with alternative scor-
ing and pruning strategies under our framework to validate
the effectiveness of our method. We performed a compre-
hensive comparison with methods such as average pooling,
max pooling, and random selection, with the specific im-
plementations detailed in the supplementary material. The
experimental results demonstrate that our method achieves
the highest performance at the same pruning ratio.
6. Conclusion
In this paper, we introduce AdaptPrune, a novel training-
free token pruning method designed to enhance the effi-
ciency of Large Vision-Language Models (LVLMs). Our
method addresses the key limitations of existing token prun-
ing strategies, which predominantly rely on attention scores
alone. By incorporating spatial information and token sim-
ilarity, AdaptPrune provides a more holistic approach to
token selection, ensuring that redundant tokens are sup-
pressed while retaining those that contribute to the model’s
performance. Through extensive experiments across mul-
tiple models and benchmarks, we demonstrate that Adapt-
Prune significantly improves both speed and memory effi-
ciency while maintaining minimal accuracy loss, even when
pruning up to 90% of the tokens. Our results validate the
importance of considering multiple cues of token relevance
in optimizing the performance of multimodal models.
References
[1] Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen,
Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Ste-
fan Lee, and Peter Anderson. Nocaps: Novel object caption-
ing at scale. In Proceedings of the IEEE/CVF International
Conference on Computer Vision, pages 8948–8957, 2019. 3,
5, 6, 1
[2] Kazi Hasan Ibn Arif, JinYi Yoon, Dimitrios S Nikolopou-
los, Hans Vandierendonck, Deepu John, and Bo Ji. Hired:
Attention-guided token dropping for efficient inference
of high-resolution vision-language models in resource-
constrained environments. arXiv preprint arXiv:2408.10945,
2024. 2
[3] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang,
Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei
Huang, et al.
Qwen technical report.
arXiv preprint
arXiv:2309.16609, 2023. 1, 2
[4] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan
Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren
Zhou. Qwen-vl: A versatile vision-language model for un-
derstanding, localization, text reading, and beyond. arXiv
preprint arXiv:2308.12966, 1(2):3, 2023. 2
[5] Navaneeth Bodla, Bharat Singh, Rama Chellappa, and
Larry S Davis. Soft-nms–improving object detection with
one line of code. In Proceedings of the IEEE/CVF Inter-
national Conference on Computer Vision, pages 5561–5569,
2017. 2, 3
[6] Ali Borji and Laurent Itti. State-of-the-art in visual atten-
tion modeling. IEEE Transactions on Pattern Analysis and
Machine Intelligence, 35(1):185–207, 2012. 3, 4
[7] Jieneng Chen, Luoxin Ye, Ju He, Zhao-Yang Wang, Daniel
Khashabi, and Alan Yuille. Llavolta: Efficient multi-modal
models via stage-wise visual context compression.
arXiv
preprint arXiv:2406.20092, 2024. 2
[8] Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui
He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v:
Improving large multi-modal models with better captions.
arXiv preprint arXiv:2311.12793, 2023. 2
[9] Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Jun-
yang Lin, Chang Zhou, and Baobao Chang. An image is
worth 1/2 tokens after layer 2: Plug-and-play inference ac-
celeration for large vision-language models. arXiv preprint
arXiv:2403.06764, 2024. 1, 2, 3, 4, 5, 6, 7, 8
[10] Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhang-
wei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng
Luo, Zheng Ma, et al. How far are we to gpt-4v? closing
the gap to commercial multimodal models with open-source
suites. arXiv preprint arXiv:2404.16821, 2024. 2, 5, 6, 1
[11] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen,
Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu,
Lewei Lu, et al. Internvl: Scaling up vision foundation mod-
els and aligning for generic visual-linguistic tasks. In Pro-
ceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition, pages 24185–24198, 2024. 2
[12] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhang-
hao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yong-
hao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-
source chatbot impressing gpt-4 with 90%* chatgpt quality.
See https://vicuna. lmsys. org, 2(3):6, 2023. 1, 2
[13] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat
Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N
Fung, and Steven Hoi.
Instructblip:
Towards general-
purpose vision-language models with instruction tuning.
Proceedings of the Advances in Neural Information Process-
ing Systems, 36, 2024.
[14] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina
Toutanova.
Bert:
Pre-training of deep bidirectional
transformers for language understanding.
arXiv preprint
arXiv:1810.04805, 2018. 2
[15] Shuangrui Ding, Peisen Zhao, Xiaopeng Zhang, Rui Qian,
Hongkai Xiong, and Qi Tian. Prune spatio-temporal tokens
by semantic-aware temporal accumulation. In Proceedings
of the IEEE/CVF International Conference on Computer Vi-
sion, pages 16945–16956, 2023. 2
[16] Qichen Fu, Minsik Cho, Thomas Merth, Sachin Mehta, Mo-
hammad Rastegari, and Mahyar Najibi. Lazyllm: Dynamic
token pruning for efficient long context llm inference. arXiv
preprint arXiv:2407.14057, 2024. 2
[17] Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei
Han, and Jianfeng Gao. Model tells you what to discard:
Adaptive kv cache compression for llms.
arXiv preprint
arXiv:2310.01801, 2023. 2
[18] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Ba-
tra, and Devi Parikh.
Making the v in vqa matter: Ele-
vating the role of image understanding in visual question
answering.
In Proceedings of the IEEE/CVF Conference
on Computer Vision and Pattern Recognition, pages 6904–
6913, 2017. 5, 1, 3, 4
[19] Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi
Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham.
Vizwiz grand challenge: Answering visual questions from
blind people. In Proceedings of the IEEE/CVF Conference
on Computer Vision and Pattern Recognition, pages 3608–
3617, 2018. 5, 1
[20] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou,
Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Mea-
suring massive multitask language understanding.
arXiv
preprint arXiv:2009.03300, 2020. 1
[21] Jan Hosang, Rodrigo Benenson, and Bernt Schiele. Learn-
ing non-maximum suppression.
In Proceedings of the
IEEE/CVF Conference on Computer Vision and Pattern
Recognition, pages 4507–4515, 2017. 2
[22] Drew A Hudson and Christopher D Manning. Gqa: A new
dataset for real-world visual reasoning and compositional
question answering. In Proceedings of the IEEE/CVF Con-
ference on Computer Vision and Pattern Recognition, pages
6700–6709, 2019. 5, 1
[23] Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui
Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H Abdi,
Dongsheng Li, Chin-Yew Lin, et al. Minference 1.0: Accel-
erating pre-filling for long-context llms via dynamic sparse
attention. arXiv preprint arXiv:2407.02490, 2024. 2
[24] Zhenglun Kong, Peiyan Dong, Xiaolong Ma, Xin Meng,
Wei Niu, Mengshu Sun, Xuan Shen, Geng Yuan, Bin Ren,
Hao Tang, et al. Spvit: Enabling faster vision transformers
via latency-aware soft token pruning. In Proceedings of the
European Conference on Computer Vision, pages 620–640.
Springer, 2022. 2
[25] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin
Zhao, and Ji-Rong Wen.
Evaluating object hallucina-
tion in large vision-language models.
arXiv preprint
arXiv:2305.10355, 2023. 5, 2
[26] Yuhong
Li,
Yingbing
Huang,
Bowen
Yang,
Bharat
Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick
Lewis, and Deming Chen.
Snapkv:
Llm knows what
you are looking for before generation.
arXiv preprint
arXiv:2404.14469, 2024. 1, 3
[27] Youwei Liang, Chongjian Ge, Zhan Tong, Yibing Song,
Jue Wang, and Pengtao Xie. Not all patches are what you
need: Expediting vision transformers via token reorganiza-
tions. arXiv preprint arXiv:2202.07800, 2022. 2
[28] Zhihang Lin, Mingbao Lin, Luxi Lin, and Rongrong Ji.
Boosting multimodal large language models with visual
tokens withdrawal for rapid inference.
arXiv preprint
arXiv:2405.05803, 2024. 2, 6
[29] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee.
Improved baselines with visual instruction tuning. In Pro-
ceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition, pages 26296–26306, 2024. 1, 2, 5,
6, 7, 8, 3
[30] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.
Visual instruction tuning. Proceedings of the Advances in
Neural Information Processing Systems, 36, 2024. 1, 2
[31] Songtao Liu, Di Huang, and Yunhong Wang. Adaptive nms:
Refining pedestrian detection in a crowd. In Proceedings of
the IEEE/CVF Conference on Computer Vision and Pattern
Recognition, pages 6459–6468, 2019. 2, 3
[32] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian
Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C
Berg. Ssd: Single shot multibox detector. In Proceedings
of the European Conference on Computer Vision, pages 21–
37. Springer, 2016. 2, 3
[33] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang
Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He,
Ziwei Liu, et al. Mmbench: Is your multi-modal model an
all-around player? In Proceedings of the European Confer-
ence on Computer Vision, pages 216–233. Springer, 2025. 5,
2, 3
[34] Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang,
Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshu-
mali Shrivastava. Scissorhands: Exploiting the persistence of
importance hypothesis for llm kv cache compression at test
time. Proceedings of the Advances in Neural Information
Processing Systems, 36, 2024. 2
[35] Zuyan Liu, Benlin Liu, Jiahui Wang, Yuhao Dong, Guangyi
Chen, Yongming Rao, Ranjay Krishna, and Jiwen Lu. Ef-
ficient inference of vision instruction-following models with
elastic cache. arXiv preprint arXiv:2407.18121, 2024. 2
[36] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and
Roozbeh Mottaghi. Ok-vqa: A visual question answering
benchmark requiring external knowledge. In Proceedings of
the IEEE/CVF Conference on Computer Vision and Pattern
Recognition, pages 3195–3204, 2019. 5, 1
[37] Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty,
and Enamul Hoque. Chartqa: A benchmark for question an-
swering about charts with visual and logical reasoning. arXiv
preprint arXiv:2203.10244, 2022. 5, 6, 2
[38] Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar.
Docvqa: A dataset for vqa on document images. In Proceed-
ings of the IEEE/CVF Winter Conference on Applications of
Computer Vision, pages 2200–2209, 2021. 2, 4, 5, 6, 8
[39] Shuai Peng, Di Fu, Baole Wei, Yong Cao, Liangcai Gao, and
Zhi Tang. Vote&mix: Plug-and-play token reduction for ef-
ficient vision transformer. arXiv preprint arXiv:2408.17062,
2024. 2
[40] Bryan A Plummer, Liwei Wang, Chris M Cervantes,
Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik.
Flickr30k entities: Collecting region-to-phrase correspon-
dences for richer image-to-sentence models. In Proceedings
of the IEEE/CVF International Conference on Computer Vi-
sion, pages 2641–2649, 2015. 5, 6, 1
[41] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya
Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry,
Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn-
ing transferable visual models from natural language super-
vision. In Proceedings of the International Conference on
Machine Learning, pages 8748–8763, 2021. 2
[42] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee,
Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and
Peter J Liu. Exploring the limits of transfer learning with a
unified text-to-text transformer. Journal of Machine Learn-
ing Research, 21(140):1–67, 2020. 2
[43] Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan
Yan. Llava-prumerge: Adaptive token reduction for efficient
large multimodal models. arXiv preprint arXiv:2403.15388,
2024. 2
[44] Dachuan Shi, Chaofan Tao, Anyi Rao, Zhendong Yang,
Chun Yuan, and Jiaqi Wang. Crossget: Cross-guided ensem-
ble of tokens for accelerating vision-language transformers.
arXiv preprint arXiv:2305.17455, 2023. 2
[45] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang,
Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus
Rohrbach. Towards vqa models that can read. In Proceed-
ings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 8317–8326, 2019. 2, 3, 4, 5, 6, 8
[46] Dingjie Song, Wenjun Wang, Shunian Chen, Xidong Wang,
Michael Guan, and Benyou Wang. Less is more: A sim-
ple yet effective token reduction method for efficient multi-
modal llms. arXiv preprint arXiv:2409.10994, 2024. 2
[47] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier
Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste
Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al.
Llama:
Open and efficient foundation language models.
arXiv preprint arXiv:2302.13971, 2023. 1, 2
[48] Ramakrishna Vedantam, C Lawrence Zitnick, and Devi
Parikh. Cider: Consensus-based image description evalua-
tion. In Proceedings of the IEEE/CVF Conference on Com-
puter Vision and Pattern Recognition, pages 4566–4575,
2015. 6, 1
[49] Zhongwei Wan, Ziang Wu, Che Liu, Jinfa Huang, Zhihong
Zhu, Peng Jin, Longyue Wang, and Li Yuan. Look-m: Look-
once optimization in kv cache for efficient multimodal long-
context inference. arXiv preprint arXiv:2406.18139, 2024.
2
[50] Alex Wang.
Glue: A multi-task benchmark and analysis
platform for natural language understanding. arXiv preprint
arXiv:1804.07461, 2018. 1
[51] Hongjie Wang, Bhishma Dedhia, and Niraj K Jha.
Zero-
tprune: Zero-shot token pruning through leveraging of the
attention graph in pre-trained transformers. In Proceedings
of the IEEE/CVF Conference on Computer Vision and Pat-
tern Recognition, pages 16070–16079, 2024. 4
[52] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han,
and Mike Lewis. Efficient streaming language models with
attention sinks. arXiv preprint arXiv:2309.17453, 2023. 1, 2
[53] Long Xing, Qidong Huang, Xiaoyi Dong, Jiajie Lu, Pan
Zhang, Yuhang Zang, Yuhang Cao, Conghui He, Jiaqi Wang,
Feng Wu, et al.
Pyramiddrop: Accelerating your large
vision-language models via pyramid visual redundancy re-
duction. arXiv preprint arXiv:2410.17247, 2024. 1, 2, 3
[54] Yizhe Xiong, Hui Chen, Tianxiang Hao, Zijia Lin, Jun-
gong Han, Yuesong Zhang, Guoxin Wang, Yongjun Bao,
and Guiguang Ding. Pyra: Parallel yielding re-activation for
training-inference efficient task adaptation. In Proceedings
of the European Conference on Computer Vision, pages 455–
473. Springer, 2025. 2
[55] Weihao Ye, Qiong Wu, Wenhao Lin, and Yiyi Zhou.
Fit
and prune:
Fast and training-free visual token pruning
for multi-modal large language models.
arXiv preprint
arXiv:2409.10197, 2024. 1, 2, 3, 8
[56] Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang,
Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong
Chen. Woodpecker: Hallucination correction for multimodal
large language models.
arXiv preprint arXiv:2310.16045,
2023. 2
[57] Gaotong Yu, Yi Chen, and Jian Xu. Balancing performance
and efficiency: A multimodal large language model prun-
ing method based image text interaction.
arXiv preprint
arXiv:2409.01162, 2024. 2
[58] Junyang Zhang, Mu Yuan, Ruiguang Zhong, Puhan Luo,
Huiyou Zhan, Ningkang Zhang, Chengchen Hu, and Xi-
angyang Li.
A-vl:
Adaptive attention for large vision-
language models. arXiv preprint arXiv:2409.14846, 2024.
1, 2, 3
[59] Kaichen
Zhang,
Bo
Li,
Peiyuan
Zhang,
Fanyi
Pu,
Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan
Zhang, Jingkang Yang, Chunyuan Li, et al. Lmms-eval: Re-
ality check on the evaluation of large multimodal models.
arXiv preprint arXiv:2407.12772, 2024. 5, 1
[60] Yichi Zhang, Bofei Gao, Tianyu Liu, Keming Lu, Wayne
Xiong, Yue Dong, Baobao Chang, Junjie Hu, Wen Xiao,
et al.
Pyramidkv:
Dynamic kv cache compression
based on pyramidal information funneling. arXiv preprint
arXiv:2406.02069, 2024. 3
[61] Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen,
Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian,
Christopher R´e, Clark Barrett, et al. H2o: Heavy-hitter ora-
cle for efficient generative inference of large language mod-
els. Proceedings of the Advances in Neural Information Pro-
cessing Systems, 36, 2024. 2
[62] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo-
hamed Elhoseiny.
Minigpt-4: Enhancing vision-language
understanding with advanced large language models. arXiv
preprint arXiv:2304.10592, 2023. 2
Multi-Cue Adaptive Visual Token Pruning for Large Vision-Language Models
Supplementary Material
In the supplementary material, we provide detailed de-
scriptions of the model architectures and settings, bench-
mark datasets along with their respective evaluation metrics,
and alternative pruning strategies. Additionally, we present
fine-grained results on the MMBench and MME bench-
marks. We also offer visualization comparisons between
FastV [9] and our AdaptPrune method at various pruning
ratios, along with the average attention distribution across
each layer of LLaVA-1.5-7B [29] on different datasets.
A. Model Settings
For a fair comparison and efficient validation, we employ
the LMMs-Eval [59] evaluation framework across all ex-
periments. The models used include LLaVA-1.5 (7B and
13B variants) [29], LLaVA-NEXT-7B [10], and InternVL2
(2B and 8B variants) [10]. We set the configurations based
on their official settings.
LLaVA-1.5. The model architecture of LLaVA-1.5 [29]
replaces the linear projection in LLaVA [30] with an MLP to
map visual features into a shared embedding space with the
language model. It uses CLIP-ViT-L as the visual encoder
at a resolution of 336×336, with Vicuna as the language de-
coder. The image is divided into 24×24 patches, generating
576 image tokens. We implement our AdaptPrune by restor-
ing the image sequences to a two-dimensional arrangement
on LLaVA-1.5.
LLaVA-NEXT. LLaVA-NEXT [29] is an adaptive high-
resolution model that divides input images into smaller grid
patches, each encoded at the native resolution of 336×336,
and then merges them into a large feature map. By remov-
ing image padding, it adjusts seamlessly to any aspect ra-
tio. To add global context and reduce stitching artifacts, it
also concatenates features from a downsampled image with
the merged map, allowing efficient support for any reso-
lution. Since the final image tokens are merged into two
image sequences, we restore each image sequence to a two-
dimensional arrangement and then perform our AdaptPrune
under a unified pruning budget.
InternVL2. InternVL2 [10] is a high-resolution model,
employing a dynamic high-resolution strategy by dividing
images into 448×448 tiles, with aspect ratios matched from
a predefined set to retain natural proportions, it scales up to
40 tiles to accommodate 4K resolution. A 448×448 thumb-
nail of the entire image is included alongside the tiles to pro-
vide global context. Pixel shuffling reduces each 448×448
tile to 256 visual tokens, enabling efficient high-resolution
processing with a reduced token count. Since multiple sub-
images tiles are directly inputted into the model for pro-
cessing, we restore each sub-image to a two-dimensional
arrangement and then perform AdaptPrune under a unified
pruning budget. In our AdaptPrune, each sub-image is pro-
cessed individually. During AdaptPrune token selection, we
consider scores across all tokens but ensure that the adaptive
suppression process does not cross sub-image boundaries.
B. Datasets
To demonstrate the broad applicability of our approach, we
use several representative datasets that cover a wide spec-
trum of task domains for evaluation, including Image Cap-
tioning, General Visual Question Answering, Text-based
Visual Question Answering, and Multimodal Reasoning.
Image Captioning. In Image Captioning, models gen-
erate a descriptive sentence for each image. We use the
Nocaps [1] and Flickr30K [40] datasets as benchmarks.
Nocaps [1] is the first large-scale benchmark dataset fea-
turing 166,100 human-generated captions for 15,100 im-
ages, designed to evaluate image captioning on novel ob-
jects. The Flickr30k [40] dataset is a standard benchmark
for sentence-based image description, featuring 31,000 im-
ages, each accompanied by five reference sentences pro-
vided by human annotators. We prompt the model with the
phrase ”Provide a one-sentence caption for the provided im-
age.” for both benchmarks and use the CIDEr score [48] for
measurement metric.
General Visual Question Answering. Visual Question
Answering (VQA) requires the model to generate an an-
swer for a given image-question pair based on the content
of the image. We evaluate the model’s performance on four
commonly used and representative datasets: VQAv2 [18],
OK-VQA [36], VizWiz [19], and GQA [22]. VQAv2 [18]
is a manually annotated open-ended question and answer
dataset about images, requiring an understanding of visu-
als, language, and common sense to answer the questions.
OK-VQA [36] is a benchmark for knowledge-based visual
question answering, featuring over 14,000 questions that re-
quire external knowledge to answer, designed to challenge
current VQA models and encourage the integration of ex-
ternal data sources. VizWiz [19] dataset consists of im-
ages and spoken questions from blind users, each with 10
crowdsourced answers, challenging models to predict an-
swers or identify unanswerable questions.
GQA [22] is
a visual question answering dataset featuring real images
with balanced Q&A pairs, scene graph annotations, and pre-
extracted visual features from advanced detection models.
The model is prompted with the instruction: ”Answer the
question using a single word or phrase.” for all benchmarks.
Category (dev)
LLaVA-1.5-7B [29] FastV [9] 90% AdaptPrune 90% FastV [9] 75% AdaptPrune 75%
Action Recognition
90.7
85.2
88.9
87.0
90.7
Attribute Comparison
50.0
50.0
54.5
52.3
54.5
Attribute Recognition
79.7
68.9
75.7
77.0
79.7
Celebrity Recognition
79.8
76.8
74.7
78.8
78.8
Function Reasoning
75.9
72.2
70.9
75.9
70.9
Future Prediction
45.0
30.0
37.5
40.0
42.5
Identity Reasoning
93.3
86.7
95.6
95.6
97.8
Image Emotion
78.0
58.0
78.0
78.0
76.0
Image Quality
35.8
22.6
30.2
28.3
32.1
Image Scene
96.2
90.4
94.2
96.2
97.1
Image Style
77.4
73.6
71.7
77.4
77.4
Image Topic
83.3
80.6
83.3
83.3
83.3
Nature Relation
41.7
39.6
37.5
37.5
37.5
Object Localization
39.5
35.8
38.3
37.0
38.3
OCR
59.0
59.0
59.0
59.0
59.0
Physical Property Reasoning
50.7
53.3
49.3
53.3
50.7
Physical Relation
33.3
41.7
41.7
41.7
41.7
Social Relation
88.4
53.5
55.8
72.1
69.8
Spatial Relationship
17.8
17.8
17.8
17.8
17.8
Structured Image-Text Understanding
26.9
30.8
26.9
28.2
26.9
Table 6. MMBench [33] finegrained comparison between FastV and AdaptPrune at 90% and 75% pruning ratios.
We use exact match accuracy as the metric for measure-
ment.
Multimodal Reasoning.
Multimodal reasoning de-
mands more sophisticated perception, knowledge, and rea-
soning skills than VQA, which makes it a more compre-
hensive task for evaluating the integrated capabilities of
LVLMs. We assess these skills with the datasets MME,
MMBench [33], and POPE [25]. MMBench [33] is a sys-
tematically designed benchmark for evaluating LVLMs, of-
fering a diverse set of multi-choice questions, rigorous qual-
ity control, and a unique CircularEval strategy to ensure
accurate and holistic model assessments. MME is a com-
prehensive evaluation benchmark for LVLMs, designed to
assess their perception and cognition abilities across 14 sub-
tasks, with manually crafted instruction-answer pairs to en-
sure a fair comparison and minimize data leakage. POPE
[25] is an innovative evaluation method designed to more
effectively and flexibly assess object hallucination in large
vision-language models by utilizing a polling-based query
approach.
The model is prompted with the instruction:
”Answer the question using a single word or phrase.” for
MME. The model is prompted with the instruction: ”An-
swer with the option’s letter from the given choices di-
rectly.” for MBench [33]. We use accuracy as the metric for
MME and POPE [25] and GPT Evaluation Score by GPT-
3.5-Turbo-0613 as the metric for MMBench [33].
Text-based VQA. Text-based visual question answer-
ing focuses on understanding textual information within im-
ages. We evaluate this capability with the datasets TextVQA
[45], DocVQA [38], and ChartQA [37].
The TextVQA
[45] dataset focuses on visual question answering in scene
text scenarios, challenging models to read and interpret text
within images to answer related questions. The DocVQA
[38] dataset offers 50,000 questions across over 12,000
document images, challenging models to understand docu-
ment structure and content for Visual Question Answering.
ChartQA [37] is a large-scale benchmark with over 32.7K
questions, combining human-written and generated queries
from chart summaries, designed to test complex visual and
logical reasoning on charts. The model is prompted with
the instruction: ”Answer the question using a single word
or phrase.” for all benchmarks. We use exact match accu-
racy as the metric for TextVQA [45] and ChartQA [37] and
ANLS Score as the metric for DocVQA [38].
C. Pruning Strategy Ablation
We conduct a series of ablation studies on our pruning
strategy with comparison methods commonly used in other
works [7, 9, 55]. Our experiments include: (a) a single-
layer implementation of FitPrune, which uses the cross at-
tention between image and text multiplied by the self atten-
tion among image tokens; (b) selecting every other token
from the top 20% tokens; (c) randomly selecting 10% to-
kens from all image tokens; (d) randomly selecting one to-
ken from each 3x3 grid and then choosing the top 10% to-
kens; (e) selecting the highest scoring token from each 3x3
grid, and then choosing the top 10% tokens; (f) applying av-
erage pooling with a stride of 1 and a kernel size of 3 over
the two-dimensional image; (g) selecting the last 10% of the
L2-category (dev)
LLaVA-1.5-7B [29] FastV [9] 90% AdaptPrune 90% FastV [9] 75% AdaptPrune 75%
Attribute Reasoning
70.4
68.3
68.3
71.9
69.3
Coarse Perception
77.4
68.6
74.7
76.0
76.7
Finegrained Perception (Cross-Instance)
55.2
53.1
55.9
54.5
56.6
Finegrained Perception (Instance-Level)
65.9
61.1
62.8
64.2
65.2
Logic Reasoning
33.1
30.5
30.5
32.2
32.2
Relation Reasoning
57.4
45.2
45.2
51.3
50.4
Table 7. MMBench [33] L2-category finegrained comparison between FastV and AdaptPrune at 90% and 75% pruning ratios.
Category
LLaVA-1.5-7B [29] FastV [9] 90% AdaptPrune 90% FastV [9] 75% AdaptPrune 75%
Code Reasoning
67.5
45.0
62.5
70.0
52.5
Numerical Calculation
70.0
47.5
57.5
52.5
55.0
Text Translation
107.5
125.0
117.5
120.0
115.0
Commonsense Reasoning
112.9
111.4
112.9
120.0
117.1
Artwork
119.5
106.5
113.8
120.0
117.5
Celebrity
137.1
127.4
133.2
138.5
134.7
Count
155.0
95.0
138.3
143.3
145.0
Color
170.0
130.0
155.0
170.0
170.0
Position
128.3
86.7
115.0
130.0
126.7
OCR
140.0
125.0
125.0
147.5
132.5
Landmark
163.8
136.5
154.0
157.5
166.0
Scene
158.0
147.5
165.8
159.8
163.3
Existence
190.0
135.0
165.0
160.0
170.0
Posters
146.6
128.2
144.9
142.5
149.3
Table 8. MME finegrained comparison between FastV and AdaptPrune at 90% and 75% pruning ratios.
image token sequence. The experiment results demonstrate
that our method is the most effective.
D. Finegrained Results
D.1. MMBench Finegrained Results
As shown in Table 6, in the MMBench [33] fine-grained
comparison between FastV [9] and AdaptPrune at 90%
and 75% pruning ratios, significant performance improve-
ments are evident with AdaptPrune in several categories.
Specifically, AdaptPrune shows enhanced outcomes in Ac-
tion Recognition, Attribute Recognition, Future Prediction,
Identity Reasoning, Image Emotion, Image Quality, and
Image Scene. These results underline AdaptPrune’s ability
to retain crucial visual information for complex understand-
ing and response capabilities within dynamic environments.
The MMBench [33] L2-category comparison in Table
7 reveals that AdaptPrune generally improves performance
in categories such as Coarse and Finegrained Perception.
This indicates that AdaptPrune’s nuanced pruning approach
could be beneficial for detailed perceptual analysis.
D.2. MME Finegrained Results
In the MME comparison in Table 8, AdaptPrune’s strat-
egy demonstrates considerable success in categories includ-
ing Code Reasoning, Commonsense Reasoning, Artwork,
Count, Color, Position, Landmark, Scene, Existence, and
Posters. These improvements suggest that AdaptPrune is
particularly effective in contexts requiring an intricate un-
derstanding of visual and contextual elements, arguably due
to its capability to preserve essential tokens that encapsulate
critical information for interpreting complex scenarios.
E. Visual Attention Distribution
In Figure 8 and Figure 9, we present the average atten-
tion score distribution across each layer of LLaVA-1.5-7B.
We randomly choose 1,000 samples from the VQAv2 [18]
benchmark of the General VQA task and the MMBench
[33] benchmark of the Multimodal Reasoning task for anal-
ysis. The model exhibits an almost identical attention distri-
bution pattern for image tokens within the same layer, even
though these samples come from entirely different tasks.
The pattern changes slightly across different layers, the pat-
tern changes slightly, but some of the tokens receiving the
highest attention remain consistent. The detailed statistical
results further reveal that the model’s attention scores ex-
hibit specific positional bias.
Ratio=0.5
Input
Ratio=0.7
Ratio=0.9
Ratio=0.5
Input
Ratio=0.7
Ratio=0.9
Ratio=0.5
Input
Ratio=0.7
Ratio=0.9
Figure 7. Uncurated random sample comparison on VQAv2 [18] benchmark images. For each comparison group with 8 images, the top
row shows FastV [9] results and the bottom row shows AdaptPrune results, with columns from left to right for input images and pruning
ratios of 0.5, 0.7, and 0.9.
Layer 1
Layer 6
Layer 5
Layer 4
Layer 3
Layer 2
Layer 7
Layer 8
Layer 9
Layer 14
Layer 13
Layer 12
Layer 11
Layer 10
Layer 15
Layer 16
Layer 17
Layer 22
Layer 21
Layer 20
Layer 19
Layer 18
Layer 23
Layer 24
Layer 25
Layer 30
Layer 29
Layer 28
Layer 27
Layer 26
Layer 31
Layer 32
Figure 8. Visualization of the average attention distribution across each layer of LLaVA-1.5-7B. The attention scores are computed from
1,000 random samples of VQAv2 [18].
Layer 1
Layer 6
Layer 5
Layer 4
Layer 3
Layer 2
Layer 7
Layer 8
Layer 9
Layer 14
Layer 13
Layer 12
Layer 11
Layer 10
Layer 15
Layer 16
Layer 17
Layer 22
Layer 21
Layer 20
Layer 19
Layer 18
Layer 23
Layer 24
Layer 25
Layer 30
Layer 29
Layer 28
Layer 27
Layer 26
Layer 31
Layer 32
Figure 9. Visualization of the average attention distribution across each layer of LLaVA-1.5-7B. The attention scores are computed from
1,000 random samples of MMBench [33].