Question:
Based on the latest knowledge, the correct option among the following is？
A. Both TransMLA and MHA2MLA support the MHA and GQA architectures, with TransMLA acting as an extension of MHA2MLA specifically for GQA.
B. The goal is to replace the Multi-Query Attention (MQA) mechanism with the more efficient Multi-Head Attention (MHA) architecture in pre-existing models.
C. The primary objective is to develop a novel attention mechanism that eliminates the need for positional encodings by compressing the complete KV cache.
D. By utilizing pre-trained models (such as MHA or GQA) and adapting them to the MLA architecture, substantial memory savings can be achieved without the need for further fine-tuning, assuming the model has already been optimized for MLA.

Answer: D

Towards Economical Inference: Enabling
DeepSeek’s Multi-Head Latent Attention in Any Transformer-based LLMs
Tao Ji♠, Bin Guo♡, Yuanbin Wu♡,
Qipeng Guo♢, Lixing Shen♣, Zhan Chen♣, Xipeng Qiu♠, Qi Zhang♠, Tao Gui♠#
♠Fudan University ♡East China Normal University ♣Hikvision Inc ♢Shanghai Al Lab
{taoji, tgui}@fudan.edu.cn
{binguo@stu, ybwu@cs}.ecnu.edu.cn
Abstract
Multi-head Latent Attention (MLA) is an in-
novative architecture proposed by DeepSeek,
designed to ensure efficient and economical
inference by significantly compressing the Key-
Value (KV) cache into a latent vector. Com-
pared to MLA, standard LLMs employing
Multi-Head Attention (MHA) and its vari-
ants such as Grouped-Query Attention (GQA)
exhibit significant cost disadvantages.
En-
abling well-trained LLMs (e.g., Llama) to
rapidly adapt to MLA without pre-training
from scratch is both meaningful and challeng-
ing. This paper proposes the first data-efficient
fine-tuning method for transitioning from MHA
to MLA (MHA2MLA), which includes two
key components: for partial-RoPE, we remove
RoPE from dimensions of queries and keys that
contribute less to the attention scores, for low-
rank approximation, we introduce joint SVD
approximations based on the pre-trained pa-
rameters of keys and values. These carefully
designed strategies enable MHA2MLA to re-
cover performance using only a small fraction
(3‰ to 6‰) of the data, significantly reducing
inference costs while seamlessly integrating
with compression techniques such as KV cache
quantization. For example, the KV cache size
of Llama2-7B is reduced by 92.19%, with only
a 0.5% drop in LongBench performance.1
1
Introduction
The rapid advancement of large language mod-
els (LLMs) has significantly accelerated progress
toward artificial general intelligence (AGI), with
model capabilities scaling predictably with param-
eter counts (Kaplan et al., 2020). However, these
gains come at a steep cost: escalating computa-
tional demands for training and degraded infer-
ence throughput, resulting in substantial energy
consumption and carbon emissions (Strubell et al.,
2019).
1Our source code is publicly available at https://github.
com/JT-Ushio/MHA2MLA.
As downstream tasks grow increasingly com-
plex, long-context processing and computationally
intensive inference have become central to LLM ap-
plications (An et al., 2024). A key bottleneck lies in
the memory footprint of the Key-Value (KV) cache
inherent to the Multi-Head Attention (MHA, 2017)
mechanism, which scales linearly with sequence
length and model size. To mitigate this, variants
like Grouped-Query Attention (GQA, 2023) and
Multi-Query Attention (MQA, 2019) have been ex-
plored. However, these methods reduce not only
the KV cache size but also the number of parame-
ters in the attention, leading to performance degra-
dation. The DeepSeek introduces Multi-Head La-
tent Attention (MLA, 2024), an attention mecha-
nism equipped with low-rank key-value joint com-
pression. Empirically, MLA achieves superior per-
formance compared with MHA, and meanwhile
significantly reduces the KV cache during infer-
ence, thus boosting the inference efficiency.
A critical yet unexplored question arises: Can
LLMs originally well-trained for MHA be
adapted to enabling MLA for inference? The in-
herent architectural disparities between MHA and
MLA render zero-shot transfer impractical, while
the prohibitive cost of pretraining from scratch
makes this transition both technically challenging
and underexplored in existing research. To address
this gap, we propose the first carefully designed
MHA2MLA framework that maximizes parame-
ter reuse from pre-trained MHA networks while
aligning the KV cache storage and inference pro-
cess with MLA’s paradigm (Figure 1). Our frame-
work features two pivotal technical innovations:
partial rotary position embedding (partial RoPE)
and low-rank approximation. The primary objec-
tive of MHA2MLA is to achieve data-efficient per-
formance recovery - restoring architecture-induced
capability degradation using minimal fine-tuning
data.
The inherent incompatibility between MLA’s
1
arXiv:2502.14837v1  [cs.CL]  20 Feb 2025
SVD Init.
d
FKjplKCPow2dKRPFBaWEUrhuTU=">AC1XicjVHLSsNA
FD2Nr1pfUZdugkWom5KIqEsfm24EBfuAWsokndZgXkwmg
pS6Erf+gFv9JfEP9C+8M0ZQi+iEJGfOvefM3HvdJPBTa
dsvBWNicmp6pjhbmptfWFwyl1caZwJj9e9OIhFy2UpD/
yI16UvA95KBGehG/Cme3mk4s0rLlI/js7kdcI7IRtEft/
3mCSqa5oVtnlzHjJ5IcLhce1g1DXLdtXWyxoHTg7KyNdJ
bD7jHD3E8JAhBEcESTgAQ0pPGw5sJMR1MCROEPJ1nGOE
mkzyuKUwYi9pO+Adu2cjWivPFOt9uiUgF5BSgsbpIkpT
xBWp1k6nmlnxf7mPdSe6m7X9Hdzr5BYiQti/9J9Zv5Xp2
qR6GNP1+BTYlmVHVe7pLprqibW1+qkuSQEKdwj+KCsKe
Vn32tCbVtaveMh1/1ZmKVXsvz83wpm5JA3Z+jnMcNLaq
zk5153S7vH+Yj7qINayjQvPcxT5qOEGdvK/wgEc8GU1jZ
Nwadx+pRiHXrOLbMu7fAUFzlbo=</latexit>(a) MHA
(b) MHA2MLA
x
x
s
uqGxcKCtYWbCnJdKrBvJhMCqXUlbj1B9zqL4l/oH/hnTEFtYhOSHLm3HvOzL3XjX0vkb9OmFMTk3PzObm8vMLi0vL5srqRKlgvEai/xINFwn4b4X8pr0pM8bseBO4Pq87t4cqni9x0XiReG57Me8FThXodf1mCOJaptmkW3dNgNHXotgc
HK8P2ybBbtk62WNg3IGCsjWaWS+oIkOIjCkCMARQhL24SCh5xJl2IiJa2FAnCDk6TjHEHnSpTFKcMh9oa+V7S7zNiQ9soz0WpGp/j0ClJa2CRNRHmCsDrN0vFUOyv2N+B9lR369PfzbwCYiWuif1LN8r8r07VItHFnq7Bo5pizajqWOaS6
q6om1tfqpLkEBOncIfigjDTylGfLa1JdO2qt46Ov+lMxao9y3JTvKtb0oDLP8c5Di62S+VKqXK2U6geZKPOYR0bKNI8d1HFEU5RI+8eHvGEZ6NuDI074/4z1ZjINGv4toyHD0/XlcA=</latexit>(c) MLA
A
C0XicjVHLSsNAFD2N7/qunQTLIKrkMbax05041LRPsBWSdJRQ/NiMhFDEcStP+BWf0r8A/0L74wp6KLohMzcOfecM3PnOrHvJcI03wva1PTM
7Nz8QnFxaXltbS23k6ilLus5UZ+xLuOnTDfC1lLeMJn3ZgzO3B81nGhzLfuWU8aLwTGQx6wf2dehdea4tCLroOZE/SLKAltHd/WpbBq7u
3uNZl03DctqWpZFQbVh1c2qXjFMNcrIx3FUekMPA0RwkSIAQwhBsQ8bCX3nqMBETFgfI8I4RZ7KM9yjSNqUWIwYNqFDmq9pd56jIe2lZ6LULp
3i089JqWObNBHxOMXyNF3lU+Us0UneI+Up75bR6uReAaECN4T+pRsz/6uTtQhcoaFq8KimWCGyOjd3SdWryJvrP6oS5BATJuMB5TnFrlKO31l
XmkTVLt/WVvkPxZSo3Ls5N8WnvCU1eNxFfXLQtoxKzaidVMv7B3mr57GJLexQP+vYxGO0SJvjme84FU71TLtQXv8pmqFXLOBX0N7+gJhOJXk
</latexit>x
RoPE
8lEKEbd+LWH3CrHyT+gf6Fd8YIuih2QmbunHvOmbl3nNj3EmGa7wVtYXFpeaW4Wlpb39jc0rd3WkmUcpc13ciPeMexE+Z7IWsKT/isE3NmB47P2s74Uubd4wnXhTeiEnMeoE9Cr2h59qCoL6+ZzmRP0gmAS1Ze2od9LMxn/b1crViqmHMDsrIRyPS32BhgAguUgRgCEo9mEjoa+LKkzEhPW
QEcYp8lSeYoSaVNiMWLYhI5pHtGum6Mh7aVnotQuneLTz0lp4Ig0EfE4xfI0Q+VT5SzRWd6Z8pR3m9Dq5F4BoQK3hP6n+2HOq5O1CAxrmrwqKZYIbI6N3dJVfkzY1fVQlyiAmT8YDynGJXKX/6bChNomqXvbV/kMxJSr3bs5N8SlvOd8Dt04q1Vqldn1arl/kT13EPg5xTO95hjqu0ECTv
DM84wWvmqXdaw/a4zdVK+SaXfwZ2tMXlFCYnw=</latexit>Wkr
J
hkCmXsyp249Qfc6v+If6B/4U2cgi6KzTDJzbnOTeWJHnclEuf+S0hcWl5ZX8amFtfWNzS9/eafIwiW3WsEMvjNuWyZnBqwhXOGxdhQz07c81rKGlzLfGrGYu2FwI8YR6/rmIHD7rm0Kgnr6XscKPYePfVrS1qRz0Eud4WjS04uVUlkNY3ZQRDbqof6ODhyEsJHAB0MAQbEHE5y+W1RQRkRYF
ylhMUWuyjNMUCBtQixGDJPQIc0D2t1maEB76cmV2qZTPpjUho4Ik1IvJhieZqh8olylugs71R5yruNabUyL59QgTtC/9NmfPqZC0CfZyrGlyqKVKIrM7OXBLVFXlz41dVghwiwmTsUD6m2FbKaZ8NpeGqdtlbU+U/FVOicm9n3ARf8pbzPXDzpFSplqrXp8XaRfbUezjEMf0nmeo4Qp1NMj7
Hi94xZvW1R60R+3ph6rlMs0u/gzt+Ru5z5kR</latexit>Wdkv
Wdq
ckv
qnope
RoPE
krope
RoPE
RoPE
Partial
RoPE
Partial
RoPE
4
9Qfc6v+If6B/4U2cgi6KzTDJzbnOTeWJHnclEuf+S0hcWl5ZX8amFtfWNzS9/eafIwiW3WsEMvjNuWyZnBqwhXOGxdhQz07c81rKGlzLfGrGYu2FwI8YR6/rmIHD7rm0Kgnr6XscKPYePfVrS1qRz0Eud4WjS04uVUlkNY3ZQRDbqof6ODhyEsJHAB0MAQbEHE5y+W1RQRkRYFylhMUWuyjNMUCBtQixGDJPQIc0D2t1
maEB76cmV2qZTPpjUho4Ik1IvJhieZqh8olylugs71R5yruNabUyL59QgTtC/9NmfPqZC0CfZyrGlyqKVKIrM7OXBLVFXlz41dVghwiwmTsUD6m2FbKaZ8NpeGqdtlbU+U/FVOicm9n3ARf8pbzPXDzpFSplqrXp8XaRfbUezjEMf0nmeo4Qp1NMj7Hi94xZvW1R60R+3ph6rlMs0u/gzt+Ru5z5kR</latexit>Wdkv
ckv
Linear
Apply RoPE
Cached
Aligned
Split
q
k
3
ueIyIrGzYsavY8gPdlp+p+gfwF9wZHKldRGUsz9w595wzc+eGWRzlyvPuataTp82nm9u1V+83N7ZtV+9HuWikIwPmYiFPA6DnMdRyocqUjE/ziQPkjDm4/Dig86Pv3CZRyL9rBYZP0mCszQ6jVigCJrZb6ehiOf5IqGlvFzOyqniX1UpRcaXy5nd8NxW6DX7zqe6/t93/cpaPf8rtd2mq5nRgPVOBL2LaY
Q4ChQAKOFIriGAFy+iZowkNG2AlKwiRFkclzLFEnbUEsToyA0Auaz2g3qdCU9tozN2pGp8T0S1I6eEcaQTxJsT7NMfnCOGt0nXdpPXdFrSGlVdCqMI5oY/pVsz/1elaFE7RMzVEVFNmEF0dq1wK8yr65s69qhQ5ZITpeE5STEzytU7O0aTm9r12wYm/9swNar3rOIW+KNvSQ1edFZH4x8t9lxO5/ajcFh1
epN7GEf76mfXQzwEUcYkvc3XOMGvyxmfbeurB/qFat0rzBg2H9/AsNMJzA</latexit>qrope
knope
v
Wuk
Wuv
3
CrHyT+gf6Fd8YUdFHshMzcOfecM3Pv2JHnxqJc/shpC4tLyv51cLa+sbmlr6904zDhDus4YReyNu2FTPDVhDuMJj7Ygzy7c91rJHlzLfumc8dsPgRowj1vWtYeAOXMcSBPX0PdMOvX489mlJWxPzoJcmd5OeXqyUymoYs4MislEP9XeY6COEgwQ+GAIij1YiOnroIyIsK6SAnjFLkqzBgbQJsRgxLEJHNA9p18nQgPbS
M1Zqh07x6OekNHBEmpB4nGJ5mqHyiXKW6CzvVHnKu41ptTMvn1CBW0L/02Z8+pkLQIDnKsaXKopUoiszslcEtUVeXPjV1WCHCLCZNynPKfYUcpnw2liVXtsreWyn8qpkTl3sm4Cb7kLed74OZJqVItVa9Pi7WL7Knz2Mchjuk9z1DFepokHeKF7ziTO1B+1Re/qharlMs4s/Q3v+BqnDmKg=</latexit>Wuq
3
CrHyT+gf6Fd8YUdFHshMzcOfecM3Pv2JHnxqJc/shpC4tLyv51cLa+sbmlr6904zDhDus4YReyNu2FTPDVhDuMJj7Ygzy7c91rJHlzLfumc8dsPgRowj1vWtYeAOXMcSBPX0PdMOvX489mlJWxPzoJfe8UlPL1ZKZTWM2UER2aiH+jtM9BHCQIfDAExR4sxPR1UEZEWFdpIRxilyVZ5igQNqEWIwYFqEjmoe062RoQHvpGS
u1Q6d49HNSGjgiTUg8TrE8zVD5RDlLdJZ3qjzl3ca02pmXT6jALaH/6abMeXWyFoEBzlUNLtUKURW52QuieqKvLnxqypBDhFhMu5TnlPsKOW0z4bSxKp2VtL5T8VU6Jy72TcBF/ylvM9cPOkVKmWqtenxdpF9tR57OMQx/SeZ6jhCnU0yDvFC17xpnag/aoPf1QtVym2cWfoT1/A6KcmKU=</latexit>Wqr
3
ueIyIrGzYsavY8gPdlp+p+gfwF9wZHKldRGUsz9w595wzc+eGWRzlyvPuataTp82nm9u1V+83N7ZtV+9HuWikIwPmYiFPA6DnMdRyocqUjE/ziQPkjDm4/Dig86Pv3CZRyL9rBYZP0mCszQ6jVigCJrZb6ehiOf5IqGlvFzOyqniX1UpRcaXy5nd8NxW6DX7zqe6/t93/cpaPf8rtd2mq5nRgPVOBL2LaY
Q4ChQAKOFIriGAFy+iZowkNG2AlKwiRFkclzLFEnbUEsToyA0Auaz2g3qdCU9tozN2pGp8T0S1I6eEcaQTxJsT7NMfnCOGt0nXdpPXdFrSGlVdCqMI5oY/pVsz/1elaFE7RMzVEVFNmEF0dq1wK8yr65s69qhQ5ZITpeE5STEzytU7O0aTm9r12wYm/9swNar3rOIW+KNvSQ1edFZH4x8t9lxO5/ajcFh1
epN7GEf76mfXQzwEUcYkvc3XOMGvyxmfbeurB/qFat0rzBg2H9/AsNMJzA</latexit>qrope
Wuv
v
knope
qnope krope
X
icjVHLbtNAFD0xUNpAWwPLsrAaIbGyHCfNYxeVDcsikaRSUkX2ZNpatT3ueIyIrGzYsavY8gPdlp+p+gfwF9wZHKldRGUsz9w595wzc+eGWRzlyvPu
ataTp82nm9u1V+83N7ZtV+9HuWikIwPmYiFPA6DnMdRyocqUjE/ziQPkjDm4/Dig86Pv3CZRyL9rBYZP0mCszQ6jVigCJrZb6ehiOf5IqGlvFzOyqn
iX1UpRcaXy5nd8NxW6DX7zqe6/t93/cpaPf8rtd2mq5nRgPVOBL2LaYQ4ChQAKOFIriGAFy+iZowkNG2AlKwiRFkclzLFEnbUEsToyA0Auaz2g3q
dCU9tozN2pGp8T0S1I6eEcaQTxJsT7NMfnCOGt0nXdpPXdFrSGlVdCqMI5oY/pVsz/1elaFE7RMzVEVFNmEF0dq1wK8yr65s69qhQ5ZITpeE5STEz
ytU7O0aTm9r12wYm/9swNar3rOIW+KNvSQ1edFZH4x8t9lxO5/ajcFh1epN7GEf76mfXQzwEUcYkvc3XOMGvyxmfbeurB/qFat0rzBg2H9/AsNMJ
zA</latexit>qrope
Wuk
WWk
Wv
Wq
Wv
Wk
Wq
q
k
v
krope
qq
krope
Figure 1: The diagram illustrates the MHA, MLA, and our MHA2MLA. It can be seen that the “cached” part is fully
aligned with MLA after MHA2MLA. The input to the attention module is also completely aligned with MLA (the
aligned region below ). Meanwhile, the parameters in MHA2MLA maximize the use of pre-trained parameters
from MHA (the aligned region above ).
inference acceleration mechanism and RoPE ne-
cessitates architectural compromises. DeepSeek’s
solution preserves PEs in limited dimensions while
compressing others, requiring strategic removal of
RoPE dimensions (converting them to NoPE) in
MHA to achieve MLA alignment. While higher
removal ratios enhance compression efficiency,
they exacerbate performance degradation, creat-
ing an efficiency-capability trade-off.
Through
systematically exploring RoPE removal strategies,
we identify that contribution-aware dimension se-
lection (retaining top-k dimensions ranked by at-
tention score impact) optimally balances these
competing objectives. Although previous studies
have investigated training partial-RoPE LLMs from
scratch (Black et al., 2021; Barbero et al., 2024),
our work pioneers data-efficient fine-tuning for full-
to-partial RoPE conversion in LLMs.
MLA reduces memory footprint by projecting
keys and values into a low-rank latent representa-
tion space (stored in the KV cache). MHA2MLA
can also apply low-rank approximation to the val-
ues and keys stripped of RoPE (NoPE dimen-
sions). By performing Singular Value Decompo-
sition (SVD) on the pre-trained parameter matri-
ces Wv and Wk corresponding to the NoPE sub-
spaces, we compress these components into a latent
space while maximizing the retention of knowledge
learned by the original model.
Our main contributions are:
• we introduce MHA2MLA, the first parameter-
efficient fine-tuning framework that adapts pre-
trained MHA-based LLMs to the MLA archi-
tecture using only 3‰ to 6‰ of training data
without training from scratch.
• we demonstrate that the MHA2MLA architecture
can be integrated with KV-cache quantization
to achieve more economical inference (up to a
96.87% reduction).
• we conduct experiments across four model sizes
(from 135M to 7B, covering both MHA and
GQA), and detailed ablation studies to provide
guidance and insights for MHA2MLA.
2
Preliminary
2.1
Multi-Head Attention (MHA)
Given an input sequence {x1, . . . , xl} ∈Rl×d,
standard MHA (Vaswani et al., 2017) projects
each token xi into queries q(h)
i
= xiW (h)
q
, keys
k(h)
i
= xiW (h)
k
, and values v(h)
i
= xiW (h)
v
,
where W (h)
q
, W (h)
k
, W (h)
v
∈Rd×dh for each head
h ∈{1, . . . , nh}. The Rotary positional encod-
ing (RoPE, 2024) is applied to queries and keys
(e.g., q(h)
i,rope = RoPE(q(h)
i
)), followed by scaled
dot-product attention2:
o(h)
i
= Softmax

q(h)
i,ropek(h)⊤
≤i,rope

v(h)
≤i ,
MHA(xi) =
h
o(1)
i , . . . , o(nh)
i
i
Wo,
(1)
where Wo ∈R(nhdh)×d and [·, ·] means vec-
tor concatenate. During autoregressive inference,
MHA stores the KV cache {k(h)
rope, v(h)}nh
h=1 of size
2We ignore here the
1
√
d scaling factor for ease of notation.
2
O(2lnhdh), growing linearly with sequence length
l, posing memory bottlenecks.
Variants:
Grouped-Query
Attention
(GQA,
2023) shares keys/values across
ng
groups
(ng ≪nh) to reduce the KV cache. For each head
h, it maps to group g = ⌊h/ng⌋:
o(h)
i
= Softmax

q(h)
i,ropek(g)⊤
≤i,rope

v(g)
≤i ,
GQA(xi) =
h
o(1)
i , . . . , o(nh)
i
i
Wo.
(2)
Multi-Query Attention (MQA, 2019) is a special
case of GQA with ng = 1, i.e., all heads share a sin-
gle global key/value. While reducing the KV cache
to O(2lngdh), these methods degrade performance
due to parameter pruning.
2.2
Multi-Head Latent Attention (MLA)
MLA (DeepSeek-AI et al., 2024) introduces a hy-
brid architecture that decouples PE from latent KV
compression. For each head h, the input xi is pro-
jected into two complementary components:
Position-Aware Component
A subset of dimen-
sions retains PE to preserve positional sensitivity:
q(h)
i,rope, ki,rope = RoPE

xiWdqW (h)
qr , xiWkr

,
where Wdq ∈Rd×dq, W (h)
qr
∈Rdq×dr, Wkr ∈
Rd×dr project queries/keys into a RoPE-preserved
component of dimension dr.
Position-Agnostic Component
The remaining
dimensions dc are stripped of PE (i.e., NoPE),
k(h)
i,nope and v(h)
i
and compressed into a shared latent
vector c(h)
i,kv:
q(h)
i,nope = xiWdqW (h)
qc ,
ci,kv = xiWdkv,
k(h)
i,nope, v(h)
i
= ci,kvW (h)
uk , ci,kvW (h)
uv ,
where W (h)
qc
∈Rdq×dc, Wdkv ∈Rd×dkv, W (h)
uk ∈
Rdkv×dc, W (h)
uv ∈Rdkv×dh. Note that dr + dc =
dh. The attention output of MLA combines both
components:
o(h)
i
= Softmax

q(h)
i,ropek(h)⊤
≤i,rope + qi,nopek(h)⊤
≤i,nope

· v(h)
≤i
MLA(xi) =
h
o(1)
i , . . . , o(nh)
i
i
· Wo.
(3)
Unlike MHA and its variants, MLA stores the latent
vector ckv and k(h)
i,rope (O (ldr + ldkv))) instead of
full-rank ki, vi (O(2lnhdh)), where (dr + dkv) ≪
2nhdh.
Why does MLA need to separate RoPE and
NoPE?
MLA introduces matrix merging tech-
niques for the NoPE portion during inference, ef-
fectively reducing memory usage. For the dot prod-
uct operation q(h)
i,nopek(h)⊤
j,nope, the following identity
transformation can be applied 3:
qi,nopek⊤
j,nope = (xiWdqWqc) (cj,kvWuk)⊤
= xi

WdqWqcW ⊤
uk

c⊤
j,kv
where
WdqWqcW ⊤
uk

can be pre-merged into a
single matrix, and cj,kv is already stored in the
KV cache. As for the RoPE portion, the RoPE(·)
function multiplies the input vector by the rotation
matrix (e.g., RoPE(qi) = qiRi, Ri’s specific form
will be introduced in Section 3.1). Therefore, the
identity transformation becomes as follows:
qi,ropek⊤
j,rope = (xiWdqWqrRi) (xjWkrRj)⊤
= xi

WdqWqcRj−iW ⊤
kr

x⊤
j
Since
WdqWqcRj−iW ⊤
kr

is related to the rela-
tive position j −i, it cannot be merged into a fixed
matrix. Considering that the relative distances in
LLMs can be very long, such as 128K, the RoPE
portion is better suited to be computed using the
original form.
3
MHA2MLA
3.1
Partial-RoPE
To enable migration from standard MHA to MLA,
we propose partial-RoPE finetuning, a strategy that
removes RoPE from a targeted proportion of di-
mensions and converts them into NoPE. Critically,
while prior work has explored training LLMs with
partial-RoPE from scratch (achieving marginally
better perplexity than full-RoPE (Black et al., 2021;
Barbero et al., 2024)), no existing method ad-
dresses how to efficiently adapt pre-trained full-
RoPE models (e.g., Llama) to partial-RoPE with-
out costly retraining. Our work bridges this gap by
systematically evaluating partial-RoPE variants to
identify the most data-efficient fine-tuning protocol
for recovering model performance post-adaptation.
3To simplify the notation, we omit the superscript (h).
Matrices Wuv and Wo can also be merged, please refer to
Appendix C by DeepSeek-AI et al. (2024).
3
(a)
(b)
(c)
(d)
RoPE
Figure 2: Illustration of Shigh, Slow, Suniform, S2-norm.
Where dh = 8 and r = 2.
Layer 3
Head 4
Layer 8
Head 23
Layer 16
Head 12
Layer 23
Head 14
Llama2-7B
      High Frequencies
Low Frequencies
2.5
5.0
7.5
10.0
12.5
15.0
17.5
20.0
Mean 2-norm
Figure 3: Visualization of Head-wise 2-norm Contri-
bution for Llama2-7B. We randomly selected 4 heads,
and the red dashed box highlights the top-4 frequency
subspaces chosen when r = 4. It can be seen that differ-
ent heads tend to focus on different frequency subspaces,
which validates the rationality of our S2-norm method.
MHA’s Full-RoPE
encodes positional informa-
tion into queries and keys through frequency-
specific rotations. Formally, given a query vector
qi ∈Rdh and key vector ki ∈Rdh, we partition
them into 2D chunks:
qi, ki =
h
q[2k,2k+1]
i
i
0≤k< dh
2
,
h
k[2k,2k+1]
i
i
0≤k< dh
2
,
where q[2k,2k+1]
i
∈R2 denotes the k-th 2D sub-
space. Each chunk undergoes a rotation by position-
dependent angles θk = β−2k/dh, forming a spec-
trum of wavelengths. High-frequency components,
e.g., k = 0, rotate rapidly at 1 radian per token.
Low-frequency components, e.g., k = dh
2 −1, ro-
tate slowly at ∼β1/dh radians per token. The base
wavelength β, typically set to 104 (Su et al., 2024)
or 5×105.
Formally, for each 2D chunk q[2k,2k+1]
i
and
k[2k,2k+1]
i
, the rotation matrix at position i is de-
fined as:
R[2k,2k+1]
i
(θk) =
cos(iθk)
−sin(iθk)
sin(iθk)
cos(iθk)

.
Thus, applying RoPE to queries and keys becomes:
qi,rope =
h
R[2k,2k+1]
i
(θk)q[2k,2k+1]
i
i
0≤k< dh
2
,
ki,rope =
h
R[2k,2k+1]
i
(θk)k[2k,2k+1]
i
i
0≤k< dh
2
.
Full-RoPE to Partial-RoPE Strategies
Given r
retained rotational subspaces(r = dr
2 ≪total sub-
spaces dh
2 , we propose four strategies (illustrated
in Figure 2) to select which r subspaces preserve
RoPE encoding:
High-Frequency Preservation retain the r
fastest-rotating (high-frequency) subspaces:
Shigh = {k | 0 ≤k < r} .
It is consistent with the p-RoPE method proposed
in Barbero et al. (2024), where they explored set-
tings in which r constituted 25%, 50%, and 75% of
the total subspaces, and observed a slight advantage
over full-RoPE in LLMs trained from scratch.
Low-Frequency Preservation retain the r
slowest-rotating (low-frequency) subspaces:
Slow =

k
 dh
2 −r ≤k < dh
2

.
It was chosen as a controlled experiment for the
high-frequency strategy.
Uniform Sampling select r subspaces with
equidistant intervals:
Suniform =

kdh
2r
  0 ≤k < r

This balances high- and low-frequency components
through geometric spacing. In practice, 2r typically
divides dh. It is similar to the partial RoPE used in
GPT-Neo (Black et al., 2021).
Head-wise 2-norm Contribution Barbero et al.
(2024) were the first to propose the 2-norm
contribution to investigate whether these fre-
quencies are utilized and how they are help-
ful.
This approach is based on the observa-
tion that, according to the Cauchy-Schwarz in-
equality, the influence of the k-th frequency sub-
space on the attention logits is upper-bounded
by the 2-norm of the corresponding query and
key components, i.e.,

D
q[2k,2k+1]
i
, k[2k,2k+1]
j
E ⩽

q[2k,2k+1]
i

k[2k,2k+1]
j

. For each head h, we
compute the mean 2-norm score for each subspace
4
x
Wdv
Wdk
ck
cv
knope
l
CxhQW1+j8Qgl1PdHI=">AC0XicjVHLSsNAFD2Nr1pfVZdugkVwFdJY+9gV3bi
saFuhVUnSUPzYjIplFIQt/6AW/0p8Q/0L7wzpqAL0QmZuXPuOWfmznVi30uEab7
ltLn5hcWl/HJhZXVtfaO4udVJopS7rO1GfsQvHDthvheytvCEzy5izuzA8VnXGR
7LfHfEeOJF4bkYx+wysG9D78ZzbUHQVd+J/EyDmiZjKbXxZJpHBwc1hs13TQsq
2FZFgWVulUzK3rZMNUoIRutqPiKPgaI4CJFAIYQgmIfNhL6eijDREzYJSaEcYo8l
WeYokDalFiMGDahQ5pvadfL0JD20jNRapdO8enpNSxR5qIeJxieZqu8qlyluhv
3hPlKe82ptXJvAJCBe4I/Us3Y/5XJ2sRuEFd1eBRTbFCZHVu5pKqV5E3179VJcg
hJkzGA8pzil2lnL2zrjSJql2+ra3y74opUbl3M26KD3lLavCsi/rvQcylWjelo
pNY+yVuexg13sUz9raOIELbTJm+MJz3jRzrSxdq89fFG1XKbZxo+hPX4CXHaV4g
=</latexit>v
Wv
knope
v
>
AC13icjVHLTsJAFD3UF+Kr4tJNlZi4IsUYdEl04xITeRgpC0DTugr7ZRICHFn3PoDbvWPjH+gf+GdsS6IDJNZ+6ce86ZuXfs0OW
xM2PjLa0vLK6l3PbWxube/ou/l6HCSRw2pO4AZR07Zi5nKf1QXLmuGEbM82UNe3gp840Ri2Ie+DdiHLKOZw183ueOJQjq6vm2Hb
i9eOzRMmlM2wfdUVcvlIqmGsb8oIB0VAP9HW30EMBAg8MPgTFLizE9LVQgomQsA4mhEUcZVnmCJH2oRYjBgWoUOaB7RrpahPe+kZK
7VDp7j0R6Q0cESagHgRxfI0Q+UT5SzRed4T5SnvNqbVTr08QgXuCP1PN2MuqpO1CPRxrmrgVFOoEFmdk7okqivy5savqgQ5hITJuEf5i
GJHKWd9NpQmVrXL3loq/6mYEpV7J+Um+JK3XOyB6yfFUrlYvj4tVC7Sp85iH4c4pvc8QwVXqKJG3vd4wSvetFvtQXvUn6oWibV7OHP
0J6/Acl4lyI=</latexit>Wv
ckv
Wdkv
(a) SVDsplit
(b) SVDjoint
U v⌃1/2
v
⌃1/2
v
V >
v
⌃1/2
k
V >
k
=
">ADKXicjVHPb9MwGH0Ng3WFQRlHLtGqSZxKMk2DYwWXHTuN/pCarnISt7OaxCFxJlV/yL+E26cgF25jBvXceKzl0rdqm04Sv
z8vde/Nl+GolcOc7PivVo4/GTzepW7emz7ecv6i93urksoB3AhnJrO+znEci4R0lVMT7acZ7Ee8508/6nrvnGe5kMknNUv5M
GaTRIxFwBRo3ro+TIK81lM07yzGE3tVcI7EZOYEXs6d9/uL2zvc8HC+xUrta6mPSXTUb3hNB0z7HXglqCBcrRl/Ts8hJAIUCAG
RwJFOAJDTs8ALhykxA0xJy4jJEydY4EaeQtScVIwYqf0ndBqULIJrXVmbtwB/SWiNyOnjT3ySNJlhPXfbFMvTLJm78qem0y9txnN
fpkVE6twRuxDvqXyf326F4Ux3pseBPWUGkZ3F5QphTkVvXN7pStFCSlxGodUzwgHxrk8Z9t4ctO7Pltm6pdGqVm9Dkptgd96l3T
B7u3rXAfd/aZ72Dw8Pmi0PpRXcVr7OIN3ec7tHCENjqU/RV/cIW/1hfrm/XDuriWpXS8wo3hvXrH9yVuhw=</latexit>
U k⌃1/2
k
7
dUk4WZTKeSuGRctw=">AC0XicjVHLSsNAFD2N7/qunQTLIKrkMbax05041LRPs
BWSdJRQ/NiMhFDEcStP+BWf0r8A/0L74wp6KLohMzcOfecM3PnOrHvJcI03wva1PT
M7Nz8QnFxaXltbS23k6ilLus5UZ+xLuOnTDfC1lLeMJn3ZgzO3B81nGhzLfuWU8
8aLwTGQx6wf2dehdea4tCLroOZE/SLKAltHd/WpbBq7u3uNZl03DctqWpZFQbVh1
c2qXjFMNcrIx3FUekMPA0RwkSIAQwhBsQ8bCX3nqMBETFgfI8I4RZ7KM9yjSNqUWI
wYNqFDmq9pd56jIe2lZ6LULp3i089JqWObNBHxOMXyNF3lU+Us0UneI+Up75bR6uR
eAaECN4T+pRsz/6uTtQhcoaFq8KimWCGyOjd3SdWryJvrP6oS5BATJuMB5TnFrlK
O31lXmkTVLt/WVvkPxZSo3Ls5N8WnvCU1eNxFfXLQtoxKzaidVMv7B3mr57GJLexQ
P+vYxGO0SJvjme84FU71TLtQXv8pmqFXLOBX0N7+gJhOJXk</latexit>x
A
ADNXicjVFPS8MwFH+tbv7XqUcvxSF4mq3I9Dj04lHRTcHOkXbZDGub2qaDUfa5/CaKB28ievILePAlRpgOmSltXn7/mpd4cBSYduPh
jk1XSjOzM7NLywuLa+UVtcaKc8Sn9Z9HvDk0iMpDVhE64KJgF7GCSWhF9ALr3ck+Ys+TVLGo3MxiGkzJN2IdZhPBEKtUux6PGingxCnvD
5s5b3+0BrF3DPWDckXcZ07O7tI32akPVE0Qjc04woet0plu2KrY0Xji7KoMcJLz2AC23g4EMGIVCIQGAdAIEUnytwIYsSbkiCVYMc
VTGMI8ejNUVQRHv47eLqSqMRrmVmqtw+/iXAN0GnBVvo4ahLsJZ/sxSfqWSJ/pWdq0y5twHOns4KERVwg+gk37fyvz7Zi4AOHKgeGPY
UK0R25+uUTJ2K3Lk10pXAhBgxWbeRT7D2lfP7nC3lSVXv8myJ4l+VUqJy7WtBm9yl3jBzu/rHC8auxWnWqme7pVrh/qZ2EDNmEb73M
fanAMJ1DH7Hv4MApG0bwzn8xn8+VLahrasw4/hvn+CRSYvk4=</latexit>
U kv⌃1/2
kv
⌃1/2
kv V >
kv
Wuk
Wuv
>
AC2nicjVHLSsNAFD2Nr1pfUXHlJloEV6UVqS6LblxWsA9oSknSaQ3Ni8ykUEI37sStP+BWP0j8A/0L74wp6KLYCZm5c+45Z+besSP
P5aJc/shpS8srq2v59cLG5tb2jr671+RhEjus4YReGLdtizPDVhDuMJj7Shmlm97rGWPrmW+NWYxd8PgTkwi1vWtYeAOXMcSBPX0A9
MOvT6f+LSkral51EuT8bSnFyulshrG/KCIbNRD/R0m+gjhIEPhgCYg8WOH0dVFBGRFgXKWExRa7KM0xRIG1CLEYMi9ARzUPadTI0o
L305Ert0Cke/TEpDZyQJiReTLE8zVD5RDlLdJ53qjzl3Sa02pmXT6jAPaH/6WbMRXWyFoEBLlUNLtUKURW52QuieqKvLnxqypBDhFhM
u5TPqbYUcpZnw2l4ap2VtL5T8VU6Jy72TcBF/ylos9cPOsVKmWqrfnxdpV9tR5HOIYp/SeF6jhBnU0yDvFC17xpnag/aoPf1QtVym
2cefoT1/A7WomK0=</latexit>Wuv
Wuk
Wk,nope
Wk,nope
Figure 4: Illustration of SVDsplit and SVDjoint. In the
multi-head setting, we adhere to the standard MLA ap-
proach, performing SVD on the merged multi-heads
rather than on each head individually (e.g., Ukv ∈
Rnhdh×nhdkv.
in an LLM over long sequences 4. Then, we pro-
pose to rank all subspaces by their 2-norm score
and select the top-r:
S2-norm = top-r
0≤k< dh
2

q[2k,2k+1]
∗

k[2k,2k+1]
∗

.
This head-specific selection adaptively preserves
rotation-critical subspaces. Figure 3 visualizes the
2-norm of Llama2-7B’s four heads.
We will analyze the effectiveness of the four
strategies in Section 4.3 and conduct an ablation
study on the essential hyperparameter r in Ap-
pendix D. For all strategies, non-selected subspaces
(k /∈S) become NoPE dimensions, enabling seam-
less integration with MLA’s latent compression.
3.2
Low-rank Approximation
After transitioning from full RoPE to partial
RoPE, we obtain the first component of the
KV cache in MLA, represented as: ki,rope =
h
R[2k,2k+1]
i
(θk)k[2k,2k+1]
i
i
k∈S. Our next goal is to
derive the second component, ci,kv ∈Rdkv, which
serves as a low-rank representation of ki,nope and
vi.
Given the keys ki = xiWk and values vi =
xiWv in MHA, we first extract the subspace of
Wk corresponding to ki,nope, i.e., the dimensions
not included in S, yielding: ki,nope = xiWk,nope.
We propose two Singular Value Decomposition
(SVD)-based strategies (Illustrated in Figure 4) to
4The 2-norm calculation detail is placed in Appendix A.
preserve pre-trained knowledge while achieving
rank reduction:
Decoupled SVD (SVDsplit)
Separately decom-
pose Wk,nope and Wv into truncated SVDs, allo-
cating dkv/2 dimensions to each:
Wk,nope = UkΣkV ⊤
k ,
Wv = UvΣvV ⊤
v ,
where Uk, Uv, Vk, Vv
∈Rdh× dkv
2 , Σk, Σv
∈
R
dkv
2 × dkv
2 . The down-projection matrices Wd∗
and up-projection matrices Wu∗become:
Wdk = UkΣ1/2
k
,
Wuk = Σ1/2
k
V ⊤
k ,
Wdv = UvΣ1/2
v
,
Wuv = Σ1/2
v
V ⊤
v .
The low-rank representation ci,kv can be con-
structed using ci,kv = [xiWdk, xiWdv].
Joint SVD (SVDjoint)
To preserve interactions
between Knope and V , we jointly factorize the con-
catenated matrix:
[Wk,nope, Wv] = UkvΣkvV ⊤
kv,
where Ukv, Vkv ∈Rdh×dkv, Σkv ∈Rdkv×dkv. The
latent projection is then:
Wdkv = UkvΣ1/2
kv ,
Wuk =Σ1/2
kv Vkv[:, : −dv], Wuv =Σ1/2
kv Vkv[:, dv :].
This jointly optimizes the latent space for both keys
and values, i.e., ci,kv = xiWdkv, retaining cross-
parameter dependencies critical for autoregressive
generation 5. Section 4.3 shows SVDjoint outper-
forming SVDsplit, validating that joint factorization
better preserves pre-trained knowledge.
4
Experiment
We evaluate our method on LLMs of varying
scales (SmolLM-135M/360M/1B7, Llama2-7B)
pre-trained with MHA or GQA. We chose the
SmolLM-series6 because its pretraining data and
framework are both open-source, which can mini-
mize the gap in fine-tuning data and processes. We
chose Llama2-7B7 because it is one of the widely
used open-source LLMs (but its pretraining data
is not open-source, there is a potential gap in fine-
tuning data).
5We describe the economical inference process of
MHA2MLA in Appendix B.
6https://huggingface.co/collections/
HuggingFaceTB/smollm-6695016cad7167254ce15966
7https://huggingface.co/meta-llama/Llama-2-7b
5
Model
Tokens
KV Mem.
Avg.
MMLU
ARC
PIQA
HS
OBQA
WG
135MSmolLM
600B
44.50
29.80
42.43
68.06
41.09
33.60
52.01
- GQA
dkv =128
44.25
29.82
42.05
68.34
41.03
33.20
51.07
- GQA2MLA
dkv =32
2.25B
-68.75%
43.06 -1.19
29.50
40.48
66.59
37.99
33.80
49.96
dkv =16
(3.8‰)
-81.25%
41.84 -2.41
28.66
39.95
65.02
36.04
31.60
49.80
dkv =8
-87.50%
40.97 -3.28
28.37
38.04
64.69
33.58
30.80
50.36
360MSmolLM
600B
49.60
33.70
49.82
71.87
51.65
37.60
52.96
- GQA
dkv =128
49.63
34.01
50.02
71.33
51.43
38.20
52.80
- GQA2MLA
dkv =32
2.25B
-68.75%
47.91 -1.72
32.94
48.36
70.73
48.16
36.00
51.30
dkv =16
(3.8‰)
-81.25%
46.94 -2.69
31.55
45.73
70.51
45.80
36.60
51.46
dkv =8
-87.50%
45.04 -4.59
30.54
43.33
68.50
42.83
35.00
50.04
1B7SmolLM
1T
55.90
39.27
59.87
75.73
62.93
42.80
54.85
- MHA
dkv =128
55.93
39.11
59.19
75.95
62.92
43.40
55.09
- MHA2MLA
dkv =32
6B
-68.75%
54.76 -1.17
38.11
57.13
76.12
61.35
42.00
53.83
dkv =16
(6.0‰)
-81.25%
54.65 -1.28
37.87
56.81
75.84
60.41
42.60
54.38
dkv =8
-87.50%
53.61 -2.32
37.17
55.50
74.86
58.55
41.20
54.38
7BLlama2
2T
59.85
41.43
59.24
78.40
73.29
41.80
64.96
- MHA
dkv =256
60.22
41.63
60.89
77.80
71.98
45.00
63.38
- MHA2MLA
dkv =64
6B
-68.75%
59.51 -0.71
41.36
59.51
77.37
71.72
44.20
62.90
dkv =32
(3.0‰)
-81.25%
59.61 -0.61
40.86
59.74
77.75
70.75
45.60
62.98
dkv =16
-87.50%
58.96 -1.26
40.39
59.29
77.75
69.70
43.40
63.22
Table 1: Commonsense reasoning ability of four LLMs with MHA2MLA or GQA2MLA. The six benchmarks
include MMLU (2021), ARC easy and challenge (ARC, 2018), PIQA (2020), HellaSwag (HS, 2019), OpenBookQA
(OBQA, 2018), Winogrande (WG, 2021).
We denote the architectural migration using
MHA2MLA and GQA2MLA, respectively.8 Both
adopt data-efficient full-parameter fine-tuning,
with the head-wise 2-norm selection (S2-norm, r =
dh
16) for Partial-RoPE and joint SVD factorization
(SVDjoint) for low-rank approximation as default
configurations. Our experiments address three crit-
ical questions:
1. How does MHA2MLA minimize accuracy
degradation induced by architectural shifts?
2. What does MHA2MLA achieve in the KV cache
reduction ratio?
3. Can MHA2MLA integrate with KV cache quan-
tization for compound gains?
4.1
Commonsense Reasoning Tasks
Main Results
As shown in Table 1, our method
achieves efficient architectural migration across
four model scales (135M to 7B) under varying
KV cache compression ratios (via latent dimen-
sion dkv). First, when comparing the performance
of our fine-tuning approach with the original LLM,
we observe only minor changes in performance
across the four base models: a -0.25% decrease
on the 135M, +0.03% on the 360M, +0.03% on
the 1B7, and +0.37% on the 7B. This suggests that
the fine-tuning data does not significantly degrade
or improve the performance of the original model,
8The details of the fine-tuning process (including data and
hyperparameters) are provided in Appendix C.
providing an appropriate experimental setting for
the MHA2MLA framework.
Next, as dkv decreases (e.g., from 32 to 16 to
8), the KV cache reduction increases (i.e., from
-68.75% to -81.25% to -87.5%), but the perfor-
mance loss becomes more challenging to recover
through fine-tuning. Figure 5 shows the fine-tuning
loss curves of 135M (representing GQA) and 7B
(representing MHA) under different compression
ratios. As the compression ratio increases, the loss
difference from the baseline becomes larger. Addi-
tionally, we observe that the fluctuation trends of
the loss curves are almost identical, indicating that
our architecture migration does not significantly
harm the model’s internal knowledge.
We also find that larger models experience less
performance degradation when transitioning to the
MLA architecture. For example, with compression
down to 18.75%, the performance drops by 2.41%
for 135M, 2.69% for 360M, 1.28% for 1B7, and
0.61% for 7B, revealing the potential scaling law
of MHA2MLA. Finally, from the 135M model to
the 7B model, the number of tokens required for
fine-tuning is only about 0.3% to 0.6% of the pre-
training tokens, demonstrating the data efficiency
of our method.
Overall,
whether
using
GQA2MLA
or
MHA2MLA, the architecture transition is achieved
with minimal cost, resulting in efficient and
economical inference.
6
0
2000
4000
6000
8000
10000
12000
#Step
1.6
1.7
1.8
1.9
Loss
MHA2MLA: Llama2-7B
MHA2MLA dkv = 16
MHA2MLA dkv = 32
MHA2MLA dkv = 64
MHA
0
2500
5000
7500
10000
12500
15000
17500
#Step
3
4
Loss
GQA2MLA: SmolLM-135M
GQA2MLA dkv = 8
GQA2MLA dkv = 16
GQA2MLA dkv = 32
GQA
Figure 5: The fine-tuning loss curves under different KV
cache storage ratios (with colors ranging from light to
dark representing 12.5%, 18.75%, 31.25%, and 100%).
4.2
Long Context Tasks
Settings
To evaluate the generative capabilities
of the model, we adopt LongBench (Bai et al.,
2024) as the benchmark for generation perfor-
mance. All models are tested using a greedy de-
coding strategy. The context window size is deter-
mined based on the sequence length used during
model fine-tuning. We use HQQ (Badri and Shaji,
2023) and Quanto9 to set caches with different lev-
els of precision to evaluate the performance of the
original model as the baseline. Since our method
is compatible with KV cache quantization, we also
conduct additional experiments to assess the com-
bined effect of both approaches.
Main
Results
As
evidenced
in
Table
2,
MHA2MLA achieves competitive or superior
efficiency-accuracy profiles compared to post-
training quantization methods on LongBench.
While 4-bit quantization incurs modest degradation
(-0.2% to -0.4%) at comparable compression ratios,
aggressive 2-bit quantization suffers severe perfor-
mance collapse (-6.2% to -9%) despite 87.5% KV
cache reduction. In contrast, MHA2MLA alone
attains 87.5% compression (at dkv =16) with only
3% accuracy loss, and further synergizes with 4-bit
quantization to reach 92.19%/96.87% compression
(dkv =64/16+Int4HQQ) while limiting degradation
to -0.5%/-3.2%, outperforming all 2-bit baselines.
This highlights that MHA2MLA’s latent space
design remains orthogonal to numerical precision
reduction, enabling compound efficiency gains
9https://huggingface.co/blog/
quanto-introduction
Model
Precision
KV Mem. Avg@LB
7BLlama2 BF16
100.0%
27.4
Int4HQQ
-75.00%
27.5
Int4Quanto
27.3
Int2HQQ
-87.50%
21.2
Int2Quanto
18.5
dkv =64
BF16
-68.75%
27.1
Int4HQQ
-92.19%
26.9
Int4Quanto
26.8
dkv =32
BF16
-81.25%
26.3
Int4HQQ
-95.31%
26.1
Int4Quanto
26.1
dkv =16
BF16
-87.50%
24.4
Int4HQQ
-96.87%
24.2
Int4Quanto
23.4
Table 2:
Evaluation results of Llama2-7B and
MHA2MLA on LongBench. Bold indicates compres-
sion ratios greater than or equal to Int2 quantization
while also achieving performance higher than Int2.
without destructive interference.
4.3
Ablation Study
Four Partial-RoPE strategies:
Shigh, Slow,
Suniform, S2-norm
Table 3 presents the results
of four strategies for converting full-RoPE to
partial-RoPE. First, when comparing the four
strategies with full-RoPE, we observed that the
low-frequency retention strategy, Slow, incurred
the greatest performance loss (a reduction of -
6.49%@135M and -1.21%@1B7), whereas the
high-frequency retention strategy, Shigh, experi-
enced significantly less degradation (a reduction of
-0.85%@135M and -0.76%@1B7), underscoring
the importance of high-frequency subspaces. Both
Suniform and S2-norm yielded better performance, the
Suniform preserves subspaces across the frequency
spectrum, while the S2-norm retains subspaces based
on their contribution to the attention scores. We
choose S2-norm as the default configuration because
the removed subspaces (i.e., NoPE) are more suit-
able for the (SVD-based) low-rank approximation.
Two SVD-based low-rank approximations:
SVDsplit, SVDjoint
The last two rows of each
group in Table 3 compare the effects of the two
SVD methods. We observe that, on both LLMs, the
SVDjoint method consistently outperforms SVDsplit,
yielding an average performance improvement of
0.92% on the 135M model and 0.74% on the 1B7
model. It indicates that SVDjoint emerges as the
7
Model
Tokens
Avg@CS
135MSmolLM
600B
44.50
- full-rope
2.25B
44.25
- Shigh
43.40 -0.85
- Slow
37.76 -6.49
- Suniform
43.76 -0.49
- S2-norm
43.77 -0.48
- Shigh + SVDjoint
2.25B
41.04 -3.21
- Suniform + SVDjoint
41.77 -2.48
- S2-norm + SVDjoint
41.84 -2.41
- S2-norm + SVDsplit
40.92 -3.33
1B7SmolLM
1T
55.90
- full-rope
6B
55.93
- Shigh
55.17 -0.76
- Slow
54.72 -1.21
- Suniform
55.31 -0.62
- S2-norm
55.10 -0.83
- Shigh + SVDjoint
6B
54.41 -1.52
- Suniform + SVDjoint
54.30 -1.63
- S2-norm + SVDjoint
54.65 -1.28
- S2-norm + SVDsplit
53.91 -2.02
Table 3: Reasoning ability of ablation studies. The
results of other models are provided in Appendix E.
clear default choice.
5
Related Work
Efficient Attention Architectures
The standard
Multi-Head Attention (MHA, 2017) mechanism’s
quadratic complexity in context length has spurred
numerous efficiency innovations.
While MHA
remains foundational, variants like Multi-Query
Attention (MQA) and Grouped-Query Attention
(GQA, 2023) reduce memory overhead by shar-
ing keys/values across heads—albeit at the cost of
parameter pruning and performance degradation.
Parallel efforts, such as Linear Transformers (Guo
et al., 2019; Katharopoulos et al., 2020; Choroman-
ski et al., 2021), RWKV (Peng et al., 2023), and
Mamba (Gu and Dao, 2023), replace softmax atten-
tion with linear recurrences or state-space models,
but struggle to match the expressiveness of stan-
dard attention in autoregressive generation.
Multi-Head Latent Attention (MLA, 2024) dis-
tinguishes itself by compressing KV caches into
low-rank latent vectors without pruning atten-
tion parameters.
Our work bridges MLA with
mainstream architectures (MHA/GQA), enabling
seamless migration via data-efficient fine-tuning.
Notably, while many linear attention variants
abandon softmax query-key interactions (e.g.,
through kernel approximations), architectures pre-
serving a query-key dot product structure—even
in factorized forms—remain compatible with our
MHA2MLA framework.
Economical Key-Value Cache
The memory
footprint of KV caches has become a critical bottle-
neck for long-context inference. Recent advances
fall into three categories:
Innovative
Architecture
methods
like
MLA (DeepSeek-AI et al., 2024), MiniCache (Liu
et al., 2024a), and MLKV (Zuhri et al., 2024) share
or compress KV representations across layers or
heads. While effective, cross-layer sharing risks
conflating distinct attention patterns, potentially
harming task-specific performance. Only MLA has
been successfully validated in Deepseek’s LLMs.
Quantization techniques such as GPTQ (Frantar
et al., 2022), FlexGen (Sheng et al., 2023), and
KIVI (Liu et al., 2024b) store KV caches in low-
bit formats (e.g., 2-bit), achieving memory savings
with precision loss.
Dynamic Pruning approaches like A2SF (Jo
and Shin, 2024) and SnapKV (Li et al., 2024)
prune “less important” tokens from the KV cache.
However, token pruning risks discarding critical
long-range dependencies, while head pruning (e.g.,
SliceGPT (Ashkboos et al., 2024), Sheared (Xia
et al., 2024), and Simple Pruning (Sun et al., 2024))
irreversibly reduces model capacity.
Our MHA2MLA method achieves the migration
of standard Transformer-based LLMs to the more
economical MLA architecture and has demon-
strated its ability to integrate with KV quantization
techniques to realize a ~97% cache saving. It is
also theoretically compatible with other methods
like pruning.
6
Conclusion
This work addresses the critical challenge of adapt-
ing pre-trained MHA-based LLMs (or variants)
to the KV-cache-efficient MLA architecture. By
introducing MHA2MLA with contribution-aware
partial-RoPE removal and SVD-driven low-rank
projection, we achieve near-lossless compression
of KV cache (up to 96.87% size reduction for
Llama2-7B) while requiring only 3‰ to 6‰of
training data. The framework demonstrates strong
compatibility with existing compression techniques
and maintains commonsense reasoning and long-
context processing capabilities, offering a practi-
cal pathway for deploying resource-efficient LLMs
without sacrificing performance. Our results un-
derscore the feasibility of architectural migration
for LLMs through targeted parameter reuse and
data-efficient fine-tuning.
8
Limitations
Verification on More LLMs
Considering that
MHA2MLA can significantly reduce inference
costs, it is worthwhile to validate it on larger and
more diverse open-source LLMs. However, con-
strained by our computation resources, models
like Llama3 require fine-tuning on a 128K con-
text length to mitigate performance degradation
from continued training, so we did not perform
such experiments. Furthermore, since Deepseek
has not yet open-sourced the tensor-parallel infer-
ence framework for MLA, it is currently challeng-
ing to explore models larger than 7B. This will be
addressed in our future work.
Parameter-Efficient MHA2MLA Fine-tuning
This paper primarily focuses on the data efficiency
of MHA2MLA. Since the architectural transforma-
tion does not involve the Feed-Forward (FFN) mod-
ule, future work could explore parameter-efficient
MHA2MLA fine-tuning, for example by freezing
the FFN module and/or freezing the parameters in
the queries and keys that correspond to the retained
RoPE. This could further reduce the cost of the
MHA2MLA transition.
References
Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury
Zemlyanskiy, Federico Lebrón, and Sumit Sanghai.
2023. GQA: training generalized multi-query trans-
former models from multi-head checkpoints. In Pro-
ceedings of the 2023 Conference on Empirical Meth-
ods in Natural Language Processing, EMNLP 2023,
Singapore, December 6-10, 2023, pages 4895–4901.
Association for Computational Linguistics.
Chenxin An, Shansan Gong, Ming Zhong, Xingjian
Zhao, Mukai Li, Jun Zhang, Lingpeng Kong, and
Xipeng Qiu. 2024. L-eval: Instituting standardized
evaluation for long context language models. In Pro-
ceedings of the 62nd Annual Meeting of the Associa-
tion for Computational Linguistics (Volume 1: Long
Papers), ACL 2024, Bangkok, Thailand, August 11-
16, 2024, pages 14388–14411. Association for Com-
putational Linguistics.
Saleh Ashkboos, Maximilian L. Croci, Marcelo Gen-
nari Do Nascimento, Torsten Hoefler, and James
Hensman. 2024. Slicegpt: Compress large language
models by deleting rows and columns. In The Twelfth
International Conference on Learning Representa-
tions, ICLR 2024, Vienna, Austria, May 7-11, 2024.
OpenReview.net.
Hicham Badri and Appu Shaji. 2023. Half-quadratic
quantization of large machine learning models.
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu,
Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao
Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang,
and Juanzi Li. 2024. Longbench: A bilingual, multi-
task benchmark for long context understanding. In
Proceedings of the 62nd Annual Meeting of the As-
sociation for Computational Linguistics (Volume 1:
Long Papers), ACL 2024, Bangkok, Thailand, Au-
gust 11-16, 2024, pages 3119–3137. Association for
Computational Linguistics.
Federico
Barbero,
Alex
Vitvitskyi,
Christos
Perivolaropoulos,
Razvan Pascanu,
and Petar
Velickovic. 2024. Round and round we go! what
makes rotary positional encodings useful?
CoRR,
abs/2410.06205.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng
Gao, and Yejin Choi. 2020. PIQA: reasoning about
physical commonsense in natural language. In The
Thirty-Fourth AAAI Conference on Artificial Intelli-
gence, AAAI 2020, The Thirty-Second Innovative Ap-
plications of Artificial Intelligence Conference, IAAI
2020, The Tenth AAAI Symposium on Educational
Advances in Artificial Intelligence, EAAI 2020, New
York, NY, USA, February 7-12, 2020, pages 7432–
7439. AAAI Press.
Sid Black, Leo Gao, Phil Wang, Connor Leahy,
and Stella Biderman. 2021.
GPT-Neo:
Large
Scale Autoregressive Language Modeling with Mesh-
Tensorflow. If you use this software, please cite it
using these metadata.
Krzysztof Marcin Choromanski, Valerii Likhosherstov,
David Dohan, Xingyou Song, Andreea Gane, Tamás
Sarlós, Peter Hawkins, Jared Quincy Davis, Afroz
Mohiuddin, Lukasz Kaiser, David Benjamin Be-
langer, Lucy J. Colwell, and Adrian Weller. 2021.
Rethinking attention with performers. In 9th Inter-
national Conference on Learning Representations,
ICLR 2021, Virtual Event, Austria, May 3-7, 2021.
OpenReview.net.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot,
Ashish Sabharwal, Carissa Schoenick, and Oyvind
Tafjord. 2018. Think you have solved question an-
swering? try arc, the AI2 reasoning challenge. CoRR,
abs/1803.05457.
DeepSeek-AI, Aixin Liu, Bei Feng, Bin Wang, Bingx-
uan Wang, Bo Liu, Chenggang Zhao, Chengqi Deng,
Chong Ruan, Damai Dai, Daya Guo, Dejian Yang,
Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fuli
Luo, Guangbo Hao, Guanting Chen, Guowei Li,
Hao Zhang, Hanwei Xu, Hao Yang, Haowei Zhang,
Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li,
Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Ji-
aqi Ni, Jiashi Li, Jin Chen, Jingyang Yuan, Junjie
Qiu, Junxiao Song, Kai Dong, Kaige Gao, Kang
Guan, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia,
Liang Zhao, Liyue Zhang, Meng Li, Miaojun Wang,
Mingchuan Zhang, Minghua Zhang, Minghui Tang,
Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang,
Peng Zhang, Qihao Zhu, Qinyu Chen, Qiushi Du,
9
R. J. Chen, R. L. Jin, Ruiqi Ge, Ruizhe Pan, Runxin
Xu, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan
Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng
Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuip-
ing Yu, Shunfeng Zhou, Size Zheng, Tao Wang, Tian
Pei, Tian Yuan, Tianyu Sun, W. L. Xiao, Wangding
Zeng, Wei An, Wen Liu, Wenfeng Liang, Wenjun
Gao, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xi-
anzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang,
Xiaojin Shen, Xiaokang Chen, Xiaosha Chen, Xiao-
tao Nie, and Xiaowen Sun. 2024. Deepseek-v2: A
strong, economical, and efficient mixture-of-experts
language model. CoRR, abs/2405.04434.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and
Dan Alistarh. 2022. GPTQ: accurate post-training
quantization for generative pre-trained transformers.
CoRR, abs/2210.17323.
Albert Gu and Tri Dao. 2023. Mamba: Linear-time
sequence modeling with selective state spaces. CoRR,
abs/2312.00752.
Qipeng Guo, Xipeng Qiu, Pengfei Liu, Yunfan Shao,
Xiangyang Xue, and Zheng Zhang. 2019.
Star-
transformer. In Proceedings of the 2019 Conference
of the North American Chapter of the Association for
Computational Linguistics: Human Language Tech-
nologies, NAACL-HLT 2019, Minneapolis, MN, USA,
June 2-7, 2019, Volume 1 (Long and Short Papers),
pages 1315–1325. Association for Computational
Linguistics.
Dan Hendrycks, Collin Burns, Steven Basart, Andy
Zou, Mantas Mazeika, Dawn Song, and Jacob Stein-
hardt. 2021. Measuring massive multitask language
understanding. In 9th International Conference on
Learning Representations, ICLR 2021, Virtual Event,
Austria, May 3-7, 2021. OpenReview.net.
Hyun-rae Jo and Dongkun Shin. 2024.
A2SF: ac-
cumulative attention scoring with forgetting factor
for token pruning in transformer decoder. CoRR,
abs/2407.20485.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B.
Brown, Benjamin Chess, Rewon Child, Scott Gray,
Alec Radford, Jeffrey Wu, and Dario Amodei. 2020.
Scaling laws for neural language models. CoRR,
abs/2001.08361.
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pap-
pas, and François Fleuret. 2020. Transformers are
rnns: Fast autoregressive transformers with linear
attention. In Proceedings of the 37th International
Conference on Machine Learning, ICML 2020, 13-18
July 2020, Virtual Event, volume 119 of Proceedings
of Machine Learning Research, pages 5156–5165.
PMLR.
Yuhong Li, Yingbing Huang, Bowen Yang, Bharat
Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai,
Patrick Lewis, and Deming Chen. 2024. Snapkv:
LLM knows what you are looking for before genera-
tion. In Advances in Neural Information Processing
Systems 38: Annual Conference on Neural Informa-
tion Processing Systems 2024, NeurIPS 2024, Van-
couver, BC, Canada, December 10 - 15, 2024.
Akide Liu, Jing Liu, Zizheng Pan, Yefei He, Reza Haf-
fari, and Bohan Zhuang. 2024a. Minicache: KV
cache compression in depth dimension for large lan-
guage models. In Advances in Neural Information
Processing Systems 38: Annual Conference on Neu-
ral Information Processing Systems 2024, NeurIPS
2024, Vancouver, BC, Canada, December 10 - 15,
2024.
Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong,
Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and
Xia Hu. 2024b. KIVI: A tuning-free asymmetric 2bit
quantization for KV cache. In Forty-first Interna-
tional Conference on Machine Learning, ICML 2024,
Vienna, Austria, July 21-27, 2024. OpenReview.net.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish
Sabharwal. 2018. Can a suit of armor conduct elec-
tricity? A new dataset for open book question an-
swering. In Proceedings of the 2018 Conference on
Empirical Methods in Natural Language Processing,
Brussels, Belgium, October 31 - November 4, 2018,
pages 2381–2391. Association for Computational
Linguistics.
Bo Peng, Eric Alcaide, Quentin Anthony, Alon Al-
balak, Samuel Arcadinho, Stella Biderman, Huanqi
Cao, Xin Cheng, Michael Chung, Leon Derczyn-
ski, Xingjian Du, Matteo Grella, Kranthi Kiran GV,
Xuzheng He, Haowen Hou, Przemyslaw Kazienko,
Jan Kocon, Jiaming Kong, Bartlomiej Koptyra, Hay-
den Lau, Jiaju Lin, Krishna Sri Ipsit Mantri, Ferdi-
nand Mom, Atsushi Saito, Guangyu Song, Xiangru
Tang, Johan S. Wind, Stanislaw Wozniak, Zhenyuan
Zhang, Qinghua Zhou, Jian Zhu, and Rui-Jie Zhu.
2023. RWKV: reinventing rnns for the transformer
era. In Findings of the Association for Computa-
tional Linguistics: EMNLP 2023, Singapore, Decem-
ber 6-10, 2023, pages 14048–14077. Association for
Computational Linguistics.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavat-
ula, and Yejin Choi. 2021. Winogrande: an adver-
sarial winograd schema challenge at scale. Commun.
ACM, 64(9):99–106.
Noam Shazeer. 2019. Fast transformer decoding: One
write-head is all you need. CoRR, abs/1911.02150.
Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan
Li, Max Ryabinin, Beidi Chen, Percy Liang, Christo-
pher Ré, Ion Stoica, and Ce Zhang. 2023. Flexgen:
High-throughput generative inference of large lan-
guage models with a single GPU. In International
Conference on Machine Learning, ICML 2023, 23-29
July 2023, Honolulu, Hawaii, USA, volume 202 of
Proceedings of Machine Learning Research, pages
31094–31116. PMLR.
Emma Strubell, Ananya Ganesh, and Andrew McCal-
lum. 2019. Energy and policy considerations for
10
deep learning in NLP. In Proceedings of the 57th
Annual Meeting of the Association for Computational
Linguistics, pages 3645–3650, Florence, Italy. Asso-
ciation for Computational Linguistics.
Jianlin Su, Murtadha H. M. Ahmed, Yu Lu, Shengfeng
Pan, Wen Bo, and Yunfeng Liu. 2024. Roformer: En-
hanced transformer with rotary position embedding.
Neurocomputing, 568:127063.
Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter.
2024. A simple and effective pruning approach for
large language models. In The Twelfth International
Conference on Learning Representations, ICLR 2024,
Vienna, Austria, May 7-11, 2024. OpenReview.net.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob
Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz
Kaiser, and Illia Polosukhin. 2017. Attention is all
you need. In Advances in Neural Information Pro-
cessing Systems 30: Annual Conference on Neural
Information Processing Systems 2017, December 4-9,
2017, Long Beach, CA, USA, pages 5998–6008.
Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi
Chen. 2024. Sheared llama: Accelerating language
model pre-training via structured pruning. In The
Twelfth International Conference on Learning Rep-
resentations, ICLR 2024, Vienna, Austria, May 7-11,
2024. OpenReview.net.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali
Farhadi, and Yejin Choi. 2019. Hellaswag: Can a
machine really finish your sentence? In Proceedings
of the 57th Conference of the Association for Compu-
tational Linguistics, ACL 2019, Florence, Italy, July
28- August 2, 2019, Volume 1: Long Papers, pages
4791–4800. Association for Computational Linguis-
tics.
Zayd Muhammad Kawakibi Zuhri, Muhammad Farid
Adilazuarda, Ayu Purwarianti, and Alham Fikri
Aji. 2024.
MLKV: multi-layer key-value heads
for memory efficient transformer decoding. CoRR,
abs/2406.09297.
A
The Calculation of 2-norm Score
To compute the 2-norm scores for each attention
head, we selected 1,024 samples from the train-
ing dataset. The proportions of the subsets and
sequence length used during the 2-norm compu-
tation are consistent with those used during fine-
tuning. First, we calculate the query vectors and
key vectors for each head. Then, for each rotational
subspace of the vectors, we compute the 2-norm
scores. Finally, the 2-norm scores of the query and
key vectors are aggregated within each subspace.
If the model employs Grouped-Query Attention
(GQA), the 2-norm scores are averaged within each
GQA group, and the scores are shared between the
groups.
B
Inference Process of MHA2MLA
During inference in the MHA2MLA model, our
input includes the hidden representation xi of the
i-th token, as well as the previously stored k(h)
<i,rope
and c<i,kv in the KV cache for the first i−1 tokens.
During the inference, our goal is to compute
the h-th head’s dot product of these two parts
q(h)
i,ropek(h)⊤
≤i,rope and q(h)
i,nopek(h)⊤
≤i,nope. For the RoPE
part, we can easily extract W (h)
q,rope and W (h)
k,rope
from the pre-trained parameter matrices W (h)
q
and
W (h)
k
(i.e., the rows corresponding to the sub-
space that retains RoPE) and then obtain the result
through a linear transformation:
q(h)
i,rope = xiW (h)
q,rope
k(h)
i,rope = xiW (h)
k,rope
k(h)
≤i,rope = [k(h)
<i,rope, k(h)
i,rope]
→q(h)
i,ropek(h)⊤
≤i,rope.
Note that k(h)
<i,rope is already stored in the KV cache
and can be directly retrieved.
For the NoPE part, q(h)
i,nope can still be easily
obtained through a linear transformation W (h)
q,nope
which extracted from the pre-trained parameter ma-
trix W (h)
q
by separating the rows corresponding to
the subspace with RoPE removed. However, k(h)
i,nope
requires two linear transformations: a dimension-
ality reduction transformation using Wdkv, and
a dimensionality expansion transformation using
W (h)
uk . Note that Wdkv is shared across all heads
in the current layer, and both Wdkv and W (h)
uk are
11
constrained by the SVD decomposition of the pre-
trained parameter matrices W (h)
k,nope and W (h)
v
, pre-
serving most of the pre-trained knowledge:
q(h)
i,nope = xiW (h)
q,nope
ci,kv = xiWdkv,
k(h)
i,nope = ci,kvW (h)
uk
k(h)
<i,nope = c<i,kvW (h)
uk .
During inference, the NoPE part can also leverage
the standard MLA matrix merging algorithm to
reduce memory consumption:
k(h)
≤i,nope = [c<i,kv, ci,kv]W (h)
uk
q(h)
i,nopek(h)⊤
≤i,nope = (xiW (h)
q,nope)(c≤i,kvW (h)
uk )⊤
= xi(W (h)
q,nopeW (h)⊤
uk
)c⊤
≤i,kv.
We
can
pre-multiply
the
parameter
ma-
trices
(W (h)
q,nopeW (h)⊤
uk
),
and
let
c(h)
i,q
=
xi(W (h)
q,nopeW (h)⊤
uk
).
In the end, the output
of MHA2MLA is as follows:
v(h)
i
=
o(h)
i
=Softmax

q(h)
i,ropek(h)⊤
≤i,rope+c(h)
i,q c⊤
≤i,kv

c≤i,kv
MHA2MLA(xi) =
h
. . . , o(h)
i
W (h)
uv , . . .
i
Wo.
Where W (h)
uv
and Wo can also perform matrix
merging to make inference more economical.
Why doesn’t MHA2MLA perform low-rank rep-
resentation on the query as DeepSeek does?
Firstly, we found that the economical inference of
MLA is not affected even if W (h)
q,nope is not decom-
posed into a dimension-reducing matrix (e.g., Wdq)
and a dimension-increasing matrix (e.g., W (h)
uq ).
Secondly, decomposing W (h)
q,nope introduces addi-
tional architectural migration loss (approximation
loss) and further reduces the number of LLM pa-
rameters. Therefore, we believe there is no need to
decompose W (h)
q,nope within the MHA2MLA frame-
work.
C
The Details of Fine-tuning
Data
We fine-tune our model using the pretrain-
ing corpus from SmolLM10. The dataset consists of
fineweb-edu-dedup, cosmopedia-v2, python-edu,
open-web-math, and StackOverflow.
The first
10https://huggingface.co/blog/smollm
three datasets are part of the smollm-corpus11 cu-
rated by HuggingFaceTB. Fineweb-edu-dedup is
a high-quality dataset filtered by HuggingFaceTB
from education-related webpages. Similarly, Hug-
gingFaceTB filtered Python code snippets from
The Stack to construct the python-edu dataset.
Cosmopedia-v2 is a high-quality dataset gener-
ated by a model based on 34,000 topics defined
by BISAC book classifications. Additionally, open-
web-math12 and StackOverflow13 are sourced from
high-quality mathematical texts available online
and posts from StackOverflow, respectively.
Hyperparameters
The fine-tuning hyperparame-
ters for models of all sizes are listed in Table 4. The
training process employs a warmup phase followed
by a decay strategy. A 1-sqrt decay strategy is
applied to ensure a smooth and gradual reduction.
D
Ablation Study on Partial-RoPE
Dimensions
To better determine the strategy and dimension-
ality for partial-RoPE, we conducted an ablation
study on the number of RoPE dimensions using
the 135MSmolLM model. The experimental results
are presented in Table 5. By comparing the per-
formance of four different strategies in varying di-
mensionalities, we observed that the low-frequency
strategy, Slow, suffered significant performance
degradation (-14.7%) when the dimensionality was
relatively low (≤4). In contrast, both Suniform and
S2-norm consistently demonstrated superior perfor-
mance regardless of dimensionality. Furthermore,
increasing the dimensionality from 4 to 8 provided
negligible performance gains. Based on these find-
ings, we selected a dimensionality of 4 for partial-
RoPE.
E
Detailed Results
In this section, we present the detailed results.
Detailed LongBench evaluation
is reported in
Table 6.
Detailed ablation experiment
is reported in Ta-
ble 7.
11https://huggingface.co/datasets/
HuggingFaceTB/smollm-corpus
12https://huggingface.co/datasets/
open-web-math/open-web-math
13https://huggingface.co/datasets/bigcode/
stackoverflow-clean
12
Metrics
135MSmolLM 360MSmolLM
1B7SmolLM
7BLlama2
n_batch × n_gpu
16×4
16×4
32×8
16×16
Learning Rate
1e-4
1e-4
1e-4
1e-4
Hardware
RTX3090
RTX3090
NVIDIA L20Y NVIDIA L20Y
Steps
18000
18000
12000
12000
Warmup ratio
5.0%
5.0%
8.3%
8.3%
Decay
10%
10%
16.7%
16.7%
Time
6h
12h
16h
28h
Seqlen
2048
2048
2048
4096
#Param.
dkv =128/256†
134.52M
361.82M
1.71B
6.61B†
dkv =32/64†
130.99M
351.38M
1.67B
6.37B†
dkv =16/32†
129.64M
347.38M
1.59B
5.99B†
dkv =8/16†
128.97M
345.39M
1.56B
5.79B†
Table 4: Training detail information across different models.
Model
Avg.
MMLU
ARC
PIQA
HS
OBQA
WG
135M
r=32
44.25
29.82
42.05
68.34
41.03
33.20
51.07
- NoPE r=0
38.99 -5.26
27.03
34.23
62.68
31.89
29.40
48.70
- Shigh
r=2
42.86 -1.39
29.58
40.91
66.54
38.48
32.00
49.64
r=4
43.40 -0.85
29.90
41.15
66.92
39.34
32.60
50.51
r=8
43.56 -0.69
29.90
40.89
67.63
40.41
32.20
50.36
- Slow
r=2
37.94 -6.31
26.95
33.56
60.28
31.51
27.80
47.51
r=4
37.76 -6.49
27.11
32.06
59.79
30.68
28.40
48.54
r=8
42.54 -1.71
29.34
39.58
67.36
37.86
32.00
49.09
- Suniform
r=2
43.16 -1.09
29.89
41.80
66.27
38.78
32.40
49.80
r=4
43.76 -0.49
29.87
41.29
67.36
40.22
32.80
50.99
r=8
43.74 -0.51
29.95
40.81
67.19
40.47
32.60
51.38
- S2-norm
r=2
43.13 -1.12
29.75
40.13
67.25
39.03
32.80
49.80
r=4
43.77 -0.48
30.14
41.69
67.57
39.53
33.00
50.67
r=8
43.88 -0.37
29.91
41.35
67.74
40.40
33.40
50.51
Table 5: The impact of positional encoding dimensionality on model performance.
0
2000
4000
6000
8000
10000
12000
#Step
1.8
2.0
2.2
2.4
Loss
MHA2MLA: SmolLM-1B7
MHA2MLA dkv = 8
MHA2MLA dkv = 16
MHA2MLA dkv = 32
MHA
0
2500
5000
7500
10000
12500
15000
17500
#Step
2.0
2.5
3.0
3.5
Loss
GQA2MLA: SmolLM-360M
GQA2MLA dkv = 8
GQA2MLA dkv = 16
GQA2MLA dkv = 32
GQA
Figure 6: The fine-tuning loss curves under different KV
cache storage ratios (with colors ranging from light to
dark representing 12.5%, 18.75%, 31.25%, and 100%).
Additional visualizations of fine-tuning loss
We present the loss of the other two models fine-
tuned, excluding the ones mentioned in the main
text, in Figure 6. We observe that as fine-tuning pro-
gresses, the gap in loss between our approach and
0
2000
4000
6000
8000
10000
12000
#Step
2.0
2.5
3.0
3.5
Loss
MHA2MLA: SmolLM-1B7
low
high
uniform
2-norm
full-rope
0
2500
5000
7500
10000
12500
15000
17500
#Step
2.5
3.0
3.5
4.0
Loss
GQA2MLA: SmolLM-135M
low
high
uniform
2-norm
full-rope
Figure 7: The fine-tuning loss curves under different
partial-RoPE strategy.
the baseline gradually decreases, and both exhibit
similar fluctuations, demonstrating the effective-
ness of our approach. In Figure 7, we show the loss
under different partial-RoPE strategies. Except for
Slow, the other three partial-RoPE schemes show
little difference from the baseline. Additionally,
13
0
2000
4000
6000
8000
10000
12000
#Step
1.8
2.0
2.2
2.4
Loss
MHA2MLA: SmolLM-1B7
full-rope
2-norm
uniform + SVDjoint
uniform + SVDsplit
0
2500
5000
7500
10000
12500
15000
17500
#Step
2.5
3.0
3.5
4.0
Loss
GQA2MLA: SmolLM-135M
full-rope
2-norm
uniform + SVDjoint
uniform + SVDsplit
Figure 8: The fine-tuning loss curves under the combi-
nation of S2-norm and different SVD strategies.
Slow has a higher probability of convergence fail-
ure. In Figure 8, we show the loss under different
SVD strategies. The loss curves on both 1B7SmolLM
and 135MSmolLM reveal that SVDjoint outperforms
SVDsplit.
14
dkv Precision
KV Avg.
S-Doc QA
M-Doc QA
Summ.
Few-shot
Synth.
Code
A
B
C
D
E
F
G
H
I
J
K
L
M
N
O
P
7BLlama2 (Length=4K)
BF16
100.0% 27.4 15.1 9.6 21.1 7.5
9.7 3.7 26.7 20.5
3.2 65.5 87.5 34.1 1.9 6.6 66.5 59.4
Int4HQQ
-75.00% 27.5 16.1 9.1 22.0 7.3
9.9 3.6 26.5 21.1
3.4 65.5 87.2 34.3 1.5 6.7 66.0 59.9
Int4Quanto
27.3 14.4 9.5 20.5 7.5
9.7 3.5 25.8 20.7
3.1 65.5 87.7 34.3 1.4 7.3 66.8 59.3
Int2HQQ
-87.50% 21.2 18.0 5.5 12.6 7.5
8.4 3.2 12.6 18.6
0.9 56.5 73.3 27.0 1.8 6.1 34.5 52.9
Int2Quanto
18.5
9.4 6.2 12.7 6.8
6.7 3.3
5.9 17.2
0.4 61.0 63.9 26.0 1.4 2.7 42.4 30.5
64
BF16
-68.75% 27.1 13.3 9.6 23.2 7.2 10.9 3.5 24.6 20.0 22.1 62.5 83.5 32.4 0.9 8.7 56.9 53.7
Int4HQQ
-92.19% 26.9 13.4 9.1 25.6 7.3 10.2 3.4 24.6 20.0 20.9 62.5 83.8 32.3 0.6 9.6 55.3 52.7
Int4Quanto
26.8 13.8 9.2 24.6 7.4 10.5 3.5 24.6 19.8 21.4 62.0 84.3 31.8 1.2 7.5 56.1 51.8
32
BF16
-81.25% 26.3 14.9 9.1 27.0 7.3
9.9 3.1 24.6 19.1 22.5 60.5 81.6 26.9 0.0 8.2 53.4 52.6
Int4HQQ
-95.31% 26.1 14.7 9.5 26.6 7.9 10.7 3.4 23.6 19.0 20.5 60.5 80.8 28.3 0.0 7.6 51.9 52.0
Int4Quanto
26.1 14.7 9.5 26.6 7.9 10.7 3.4 23.6 19.0 20.5 60.5 80.8 28.3 0.0 7.6 51.9 52.0
16
BF16
-87.50% 24.4 14.7 9.5 24.3 7.8 10.2 3.8 22.8 19.1 24.6 61.0 82.8 20.2 0.2 8.6 39.9 41.4
Int4HQQ
-96.87% 24.2 15.2 9.4 25.2 7.4 10.2 3.9 22.9 19.8 20.6 61.0 82.5 21.7 0.1 9.0 38.0 41.2
Int4Quanto
23.4 15.6 8.4 22.7 7.3 10.2 3.8 20.2 18.7 18.6 61.0 81.9 21.7 0.5 8.0 36.9 38.3
1B7SmolLM (Length=2K)
BF16
100.0% 18.7
2.6 6.3 19.9 5.4
8.6 2.7 23.5 18.4 20.2 46.5 70.2 32.4 2.2 3.2 21.3 16.5
Int4HQQ
-75.00% 18.6
2.5 6.2 19.1 5.5
8.2 2.7 23.4 18.3 20.0 46.5 69.4 32.1 2.7 3.2 21.5 16.0
Int4Quanto
18.6
2.6 6.2 17.4 5.1
8.6 2.6 23.0 18.1 20.1 46.0 70.2 31.9 2.9 3.6 21.9 16.7
Int2HQQ
-87.50% 16.3
2.5 5.6 13.0 4.8
7.5 2.7 14.8 16.3
9.3 46.0 70.4 26.9 2.6 3.4 18.3 16.8
Int2Quanto
13.3
1.6 3.8 10.3 3.9
7.3 1.4
5.9 13.4
6.3 40.0 64.3 14.6 3.1 3.5 15.6 17.5
32
BF16
-68.75% 16.0
2.6 6.1 16.9 4.6
9.3 2.0 22.8 15.1 19.9 50.0 57.1 29.8 1.7 2.4
9.4
6.7
Int4HQQ
-92.19% 15.9
2.7 5.7 16.3 5.0
8.5 1.8 23.0 15.0 18.5 50.0 56.2 30.2 1.8 3.2 10.0
6.8
Int4Quanto
15.4
2.5 5.7 16.1 5.7
8.7 2.1 20.9 13.8 17.6 50.0 55.0 29.5 1.7 2.8
9.6
5.4
16
BF16
-81.25% 16.5
2.6 6.2 17.2 4.5
9.7 2.1 22.0 15.3 21.0 47.5 55.5 31.7 1.2 3.3 15.8
8.5
Int4HQQ
-95.31% 16.2
2.5 6.1 16.2 4.5
8.9 2.0 20.6 15.4 19.7 47.5 55.6 30.6 1.2 4.0 16.3
8.0
Int4Quanto
15.6
2.5 5.7 15.6 4.3
8.8 1.6 21.2 15.7 17.6 47.0 55.7 27.4 1.7 3.6 15.6
6.2
8
BF16
-87.50% 15.3
2.4 5.9 17.9 4.8 10.1 1.8 25.1 15.2 20.6 42.5 49.0 31.4 2.7 3.3
7.1
4.4
Int4HQQ
-96.87% 15.0
2.4 5.7 16.9 4.7 10.1 2.0 23.5 14.7 20.3 42.5 47.6 30.6 2.6 3.6
7.7
4.5
Int4Quanto
14.2
2.7 5.4 16.9 4.1
8.8 1.5 22.2 14.4 17.2 42.0 47.9 29.9 1.5 3.3
7.0
3.0
360MSmolLM (Length=2K)
BF16
100.0% 13.5
2.4 6.4 14.3 5.0
8.8 2.5 18.0 17.5
7.1 47.5 37.5 24.9 1.5 3.4
8.1 10.4
Int4HQQ
-75.00% 13.4
2.7 6.1 14.1 5.5
8.4 3.0 16.2 15.4 11.2 47.5 37.5 23.4 1.3 3.7
9.0 10.1
Int4Quanto
13.3
2.4 6.2 13.7 5.4
8.7 2.6 15.4 17.4
7.3 47.5 37.3 24.4 1.0 3.7
8.4 11.0
Int2HQQ
-87.50% 10.8
2.7 4.7
8.3 5.4
5.9 1.9
9.9 10.0
8.4 45.2 27.5 14.2 2.1 4.2 10.0 11.9
Int2Quanto
8.6
2.6 2.2
4.4 3.9
4.8 1.4
5.6
8.9
2.9 44.0 26.8
9.6 1.0 1.9
7.2
9.7
32
BF16
-68.75% 13.5
2.3 5.9 13.4 5.5
9.8 2.7 20.4 14.5 11.5 41.0 31.2 29.6 1.2 3.5 15.4
7.9
Int4HQQ
-92.19% 12.5
2.6 5.7 12.1 5.1 10.2 2.7 14.6 12.5
8.8 41.0 30.3 27.8 1.9 2.7 14.5
7.6
Int4Quanto
12.3
2.0 5.2 11.9 5.0
9.1 3.0 15.4 14.9
8.3 41.0 28.3 27.0 0.9 3.9 13.8
7.8
16
BF16
-81.25% 11.6
2.2 5.2 13.0 4.8
9.5 3.2 13.4 13.4 11.3 32.0 26.1 22.5 1.1 5.0 14.9
7.7
Int4HQQ
-95.31% 11.2
2.6 5.6 12.0 5.1
8.8 2.9 13.4 12.4 10.8 32.0 24.8 21.8 2.1 3.7 14.0
7.2
Int4Quanto
10.9
1.9 4.9 11.5 4.2
8.8 2.6 12.2 12.2
9.5 32.5 25.8 18.5 1.4 4.6 15.5
7.8
8
BF16
-87.50%
9.9
1.9 4.7 11.7 4.5
8.5 2.8 13.0 12.9
9.4 34.0 17.2 15.3 1.4 3.2 11.4
6.9
Int4HQQ
-96.87% 10.0
2.2 4.8 11.0 4.2
8.2 2.6 13.1 12.8 11.7 33.5 17.3 14.8 0.8 4.4 11.6
7.4
Int4Quanto
9.3
1.8 3.6 11.3 4.0
8.0 3.0 10.6 12.0
7.4 31.5 19.8 10.3 0.8 4.8 12.1
7.6
Table 6: Evaluation results of all models on LongBench, including Task A: narrativeqa, B: qasper, C: multifieldqa_en,
D: hotpotqa, E: 2wikimqa, F: musique, G: gov_report, H: qmsum, I: multi_news, J: trec, K: triviaqa, L: samsum, M:
passage_count, N: passage_retrieval_en, O: lcc, P: repobench-p. Bold indicates compression ratios greater than or
equal to Int2 quantization while also achieving performance higher than Int2.
15
Model
Tokens
Avg@CS
MMLU
ARC
PIQA
HS
OBQA
WG
135MSmolLM
600B
44.50
29.80
42.43
68.06
41.09
33.60
52.01
- full-rope
2.25B
44.25
29.82
42.05
68.34
41.03
33.20
51.07
- Shigh
43.40 -0.85
29.90
41.15
66.92
39.34
32.60
50.51
- Slow
37.76 -6.49
27.11
32.06
59.79
30.68
28.40
48.54
- Suniform
43.76 -0.49
29.87
41.29
67.36
40.22
32.80
50.99
- S2-norm
43.77 -0.48
30.14
41.69
67.57
39.53
33.00
50.67
- Shigh + SVDjoint
2.25B
41.04 -3.21
28.16
37.55
64.91
34.91
32.00
48.70
- Suniform + SVDjoint
41.77 -2.48
28.58
38.69
65.67
36.17
32.00
49.49
- S2-norm + SVDjoint
41.84 -2.41
28.66
39.95
65.02
36.04
31.60
49.80
- S2-norm + SVDsplit
40.92 -3.33
28.04
37.85
65.56
34.60
29.8
49.65
1B7SmolLM
1T
55.90
39.27
59.87
75.73
62.93
42.80
54.85
- full-rope
6B
55.93
39.11
59.19
75.95
62.92
43.40
55.09
- Shigh
55.17 -0.76
38.56
57.72
75.73
60.93
44.00
54.06
- Slow
54.72 -1.21
37.82
56.47
75.35
60.06
43.20
55.41
- Suniform
55.31 -0.62
38.93
57.93
75.63
61.97
42.60
54.85
- S2-norm
55.10 -0.83
38.60
57.36
75.68
61.77
43.00
54.22
- Shigh + SVDjoint
6B
54.41 -1.52
37.97
56.74
75.14
59.75
42.00
54.85
- Suniform + SVDjoint
54.30 -1.63
37.82
56.30
75.08
60.35
42.40
53.91
- S2-norm + SVDjoint
54.65 -1.28
37.87
56.81
75.84
60.41
42.60
54.38
- S2-norm + SVDsplit
53.91 -2.02
37.64
55.50
75.46
59.52
42.40
52.96
Table 7: The complete results of the ablation experiment.
16