Question:
What generalization abilities were demonstrated by PH3 that exceeded initial expectations?
A. PH3 maintained effectiveness across different LM architectures, from GPT to LLaMa models.
B. The method worked across different task types, including factual QA and knowledge retrieval.
C. PH3 could only be applied to models trained on a specific dataset, limiting its usability.
D. Pruning attention heads did not impact general model performance outside of knowledge conflicts.

Answer: AB

We conduct a thorough analysis of the generalization ability of PH3. For cross-model generalization, PH3 is effective across a wide range of models. This shows that our method is not limited to small models, but can also be adopted on relatively large models, including the popular LLaMA2 series. For cross-relation generalization, by intervening on the attention heads discovered on World Capital, our method can also well resolve knowledge conflicts on other relation types. This indicates that PH3 does not identify attention heads specific to a certain type of relation. Instead, it identifies universal memory and context heads. For cross-format generalization, PH3 can transfer well from triple-form context to document-form context. This indicates that our method does not merely remember the relative positions of elements in context, but is capable of understanding the external context. Compared to the Gradient, our method has demonstrated superior generalizability. We also analyze the impact of the number of pruning heads in Appendix G.

Cutting Off the Head Ends the Conflict: A Mechanism for
Interpreting and Mitigating Knowledge Conflicts in Language Models
Zhuoran Jin1,2, Pengfei Cao1,2, Hongbang Yuan1,2, Yubo Chen1,2,
Jiexin Xu3, Huaijun Li3, Xiaojian Jiang3, Kang Liu1,2, Jun Zhao1,2
1 School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
2 The Laboratory of Cognition and Decision Intelligence for Complex Systems,
Institute of Automation, Chinese Academy of Sciences, Beijing, China
3 China Merchants Bank
{zhuoran.jin, pengfei.cao, yubo.chen, kliu, jzhao}@nlpr.ia.ac.cn
Abstract
Recently, retrieval augmentation and tool aug-
mentation have demonstrated a remarkable ca-
pability to expand the internal memory bound-
aries of language models (LMs) by providing
external context. However, internal memory
and external context inevitably clash, leading to
knowledge conflicts within LMs. In this paper,
we aim to interpret the mechanism of knowl-
edge conflicts through the lens of information
flow, and then mitigate conflicts by precise in-
terventions at the pivotal point. We find there
are some attention heads with opposite effects
in the later layers, where memory heads can
recall knowledge from internal memory, and
context heads can retrieve knowledge from ex-
ternal context. Moreover, we reveal that the piv-
otal point at which knowledge conflicts emerge
in LMs is the integration of inconsistent infor-
mation flows by memory heads and context
heads. Inspired by the insights, we propose a
novel method called Pruning Head via PatH
PatcHing (PH3), which can efficiently mitigate
knowledge conflicts by pruning conflicting at-
tention heads without updating model parame-
ters. PH3 can flexibly control eight LMs to use
internal memory (↑44.0%) or external context
(↑38.5%). Moreover, PH3 can also improve
the performance of LMs on open-domain QA
tasks. We also conduct extensive experiments
to demonstrate the cross-model, cross-relation,
and cross-format generalization of our method.
1
Introduction
Language models (LMs) (Brown et al., 2020; Tou-
vron et al., 2023; OpenAI, 2023) have memorized
a substantial amount of factual knowledge during
pre-training, and stored the knowledge within their
parameters as internal memory (i.e., parametric
knowledge) (Meng et al., 2022). During the infer-
ence phase, LMs rely on their internal memory to
understand and generate text. However, the internal
memory may be limited or outdated, making LMs
prone to producing factually incorrect content.
To alleviate the problem, one promising solu-
tion is to employ additional retrievers or tools to
augment LMs by providing external context (i.e.,
non-parametric knowledge). Nevertheless, inter-
nal memory and external context can often con-
tradict each other, which is known as knowledge
conflicts (Longpre et al., 2021; Chen et al., 2022;
Xie et al., 2023; Yu et al., 2023). Recent works have
mainly investigated the behavior and preference of
LMs, attempting to determine whether these mod-
els are more inclined towards internal memory or
external context when faced with knowledge con-
flicts. However, there is a limited understanding
of the underlying mechanism of knowledge con-
flicts. Insights into the mechanism will facilitate
precise interventions at the pivotal point to mitigate
knowledge conflicts, which can not only empower
LMs to more reliably adhere to internal memory
(e.g., ignoring misleading external context) but also
enhance faithfulness in generating text based on ex-
ternal context (e.g., correcting outdated memory).
In this paper, we reveal that the pivotal point
at which knowledge conflicts emerge in LMs is
the integration of inconsistent information flows by
various attention heads in later layers. To investi-
gate this, we consider a simple factual recall task
(i.e., subject attribute prediction) inspired by the
work of Yu et al. (2023). As illustrated in Figure
1, given the question (i.e., “What is the capital of
France?”) and the conflicting external context (i.e.,
“The capital of France is Rome.”), the model can
either use internal memory (i.e., “Paris”) or exter-
nal context (i.e., “Rome”) to predict the subject’s
attribute. Following this, we present a set of “top-
down” analyses to locate the pivotal point where
conflicts emerge and to identify the model compo-
nents that are significant in knowledge conflicts,
which primarily involves the following three steps:
Step 1: We start by answering the first ques-
tion “What function do model components serve
in knowledge conflicts?”. We knock out the acti-
arXiv:2402.18154v1  [cs.CL]  28 Feb 2024
The
capital
of
is
Rome
.
What
is
the
capital
of
France
?
France
Rome
Paris
:
Yes
The
France
Rome
Paris
Context Relation
Context Subject
Context Attribute
Question Relation
Question Subject
FFN
MHA
Context Head
(1)
(1)
Deep
Semantic
Shallow
Semantic
(3) Memory
Extraction
(3)
Context
Extraction
(2)
(2)
Relation
Propagation
Paris
Rome
Subject
Propagation
Pivotal Point of
Knowledge Conflict
Memory Head
Figure 1: An illustration of the mechanism of knowledge conflicts in LMs: (1) Enriching the semantic information
of context subject and context attribute; (2) Propagating question information to the last token through MHAs; (3)
Extracting attribute information through memory attention heads and context attention heads at later layers.
vations to examine the functionality of multi-head
attention (MHA) blocks and feed-forward network
(FFN) blocks. We find that FFNs enrich the seman-
tic information of input elements in early layers,
while MHAs play an important role in passing in-
formation to the last token in later layers; Step 2:
Based on this, the second question naturally arises,
namely “When and where do MHAs pass informa-
tion to the last token?”. We investigate the MHAs
by knocking out the attention weights from the last
token to other input elements. Results reveal that
the question information is first propagated to the
last token, and then the last token extracts attribute
information from the subject and the attribute in
the context; Step 3: Inspired by this, we aim to
answer the final question “How do MHAs extract
attribute information under knowledge conflicts?”.
We find that some attention heads in late MHAs
play opposite roles, where memory heads can re-
call attributes from internal memory, and context
heads can retrieve attributes from external context.
According to our findings, the mechanism by which
LMs use both internal memory and external context
can be summarized as three stages in Figure 1: (1)
Enriching semantic information; (2) Propagating
question information; and (3) Extracting attribute
information, where knowledge conflicts arise at
the third stage, due to the inconsistent information
flows between memory heads and context heads.
Inspired by our insights into knowledge conflicts,
we propose a minimally-invasive control method
called Pruning Head via PatH PatcHing (PH3),
which can efficiently mitigate knowledge conflicts
by intervening on attention heads without updating
model parameters. First, we use the path patching
(Goldowsky-Dill et al., 2023; Wang et al., 2023a)
technique to localize important memory heads and
context heads. Our method can avoid the noise in-
terference of other heads, enabling a more accurate
calculation of the importance score for the target
head. Then, we perform structured pruning on
those negative attention heads to mitigate conflicts.
In this way, our method can flexibly control LMs
to use internal memory or external context. Experi-
mental results on the World Capital dataset show
that our method can not only reliably and consis-
tently increase the average internal memory usage
rate of eight LMs by 44.0% (from 49.7% to 93.7%)
but also increase the external context usage rate by
38.5% (from 50.3% to 88.8%). PH3 also enables
LMs to generate answers more faithfully according
to retrieved passages in open-domain QA tasks. We
conduct extensive experiments to demonstrate the
cross-model (e.g., from GPT series to LLaMA2 se-
ries), cross-relation (e.g., from World Capital to
Official Language), and cross-format (e.g., from
triple format to document format) generalization.
Our contributions are summarized as follows:
• We perform an exploration into the mecha-
nism of interpreting knowledge conflicts, and
reveal that memory heads and context heads
at later layers can cause knowledge conflicts
when inconsistent information flows merge.
• We propose a novel method called Pruning
Head via PatH PatcHing (PH3), which can ef-
ficiently mitigate knowledge conflicts by prun-
ing those conflicting attention heads.
• We demonstrate that our PH3 can flexibly con-
trol LMs to use internal memory (↑44.0%) or
external context (↑38.5%). We also prove the
cross-model, cross-relation, and cross-format
generalization ability of our method.
2
Background
In this work, we mainly focus on the autoregressive
transformer-based language models. Given a se-
quence of input tokens x = [x1, · · · , xN], the LM
G first embeds each token xi into a vector x0
i ∈Rd
using an embedding matrix E ∈R|V|×d, over a
vocabulary V. The input embeddings are processed
by L transformer layers. Each layer consists of an
MHA and an FFN. Formally, the hidden state xℓ
i of
token xi at layer ℓis calculated as:
xℓ
i = xℓ−1
i
+ aℓ
i + mℓ
i,
(1)
where aℓ
i and mℓ
i are the outputs from the MHA
block and the FFN block in the ℓ-th layer. Then,
the vocabulary head ϕ(·) and the softmax function
σ(·) predict the output probability:
pL
i = σ
ϕ
xL
i

.
(2)
MHA.
A MHA block consists of M attention
heads, which are capable of aggregating global in-
formation from the hidden states (Halawi et al.,
2023; Wang et al., 2023b). An individual attention
head h in layer ℓconsists of three learnable ma-
trices, Wℓ,h
Q , Wℓ,h
K , Wℓ,h
V
∈Rd× d
M . Formally, for
the input Xℓ−1 =
h
xℓ−1
1
, · · · , xℓ−1
N
i
in layer ℓ:
Aℓ=
h
Hℓ,1; · · · ; Hℓ,Mi
Wℓ
o,
(3)
Hℓ,h = sℓ,hXℓ−1Wℓ,j
V ,
(4)
sℓ,h = σ




Xℓ−1Wℓ,h
Q

Xℓ−1Wℓ,h
K
T
p
d/M


(5)
where Aℓ=

aℓ
1, · · · , aℓ
N

is the MHA block’s
output. Wℓ,h
O ∈Rd×d is a learnable output matrix.
FFN.
A FFN block can work as a key-value mem-
ory to store factual knowledge (Geva et al., 2021),
enriching the hidden states of token i:
mℓ
i = f

xℓ−1
i
+ aℓ
i

Wℓ
1

Wℓ
2.
(6)
3
Experimental Setup
3.1
Tasks
In this paper, we conduct controlled experiments
to construct knowledge conflicts, wherein the inter-
nal memory is factual while the external context is
counterfactual. To avoid the LM being influenced
by other irrelevant factors (i.e., reasoning ability),
we adopt a simple factual recall task (Geva et al.,
2023), which requires predicting the corresponding
attribute am based on the given subject s and rela-
tion r. Building on previous work (Yu et al., 2023),
we use the World Capital dataset to interpret this
problem in §4, where the LM needs to predict the
capital city of the country based on the question q:
Q: What is the capital of {s}? A:
We retain those questions that the LM can correctly
predict the factual attributes am based on internal
memory, then provide the counterfactual attributes
ac in the external context c to construct conflicts:
The capital of {s} is {ac}. {q}
To mitigate knowledge conflicts, we further con-
struct three datasets for verifying the generaliza-
tion of our method in §5, including the Official
Language, Country, and Continent datasets. We
also generate a more complex World Capital D
dataset based on the World Capital dataset, us-
ing gpt-3.5-turbo to rewrite the external context
from triplet form into document form. More details
about these datasets are shown in Appendix B.
3.2
Models
We analyze two GPT-series LMs: GPT-2 XL (Rad-
ford et al., 2019) and GPT-J (Wang and Komat-
suzaki, 2021) in §4. Additionally, we also validate
the effectiveness of our method on six LMs: OPT-
1.3B, OPT-2.7B (Zhang et al., 2022), Pythia-6.9B,
Pythia-12B (Biderman et al., 2023), LLaMA2-7B
and LLaMA2-13B (Touvron et al., 2023) in §5.
4
Interpreting Knowledge Conflicts
We utilize a “top-down” analysis approach to lo-
cate the pivotal point where conflicts emerge and
to identify the model components that are signifi-
cant in knowledge conflicts. We start by examining
the functionality of model components by knocking
out activations, and reveal that MHAs in the middle
and late layers play a crucial role in passing infor-
mation to the last token (§4.1). Then, we further
investigate MHAs by knocking out the attention
weights. We find the question information is first
passed to the last token, then the last token extracts
information from the subject and the attribute in
the context (§4.2). Last, we discover that some
attention heads in later MHAs play opposite roles
in conflicts, where memory heads can recall knowl-
edge from internal memory, and context heads can
retrieve knowledge from external context (§4.3).
0
8
16
24
32
40
48
C Subject
C Relation
C Attribute
Q Subject
Q Relation
Last
0.0
0.2
0.4
0.6
0.8
(a) Effect of FFNs on internal memory.
0
8
16
24
32
40
48
0.0
0.1
0.2
0.3
0.4
0.5
0.6
(b) Effect of MHAs on internal memory.
0
8
16
24
32
40
48
Layer
0.0
0.1
0.2
0.3
0.4
Attribute Extraction Rate
MHA
FFN
(c) Extraction rate of internal memory.
0
8
16
24
32
40
48
C Subject
C Relation
C Attribute
Q Subject
Q Relation
Last
0.0
0.2
0.4
0.6
0.8
(d) Effect of FFNs on external context.
0
8
16
24
32
40
48
0.0
0.1
0.2
0.3
0.4
0.5
0.6
(e) Effect of MHAs on external context.
0
8
16
24
32
40
48
Layer
0.0
0.1
0.2
0.3
0.4
Attribute Extraction Rate
MHA
FFN
(f) Extraction rate of external context.
Figure 2: Effect of model components (FFNs and MHAs) in GPT-2 XL on the final prediction probability. Figures
2a and 2b (Figures 2d and 2e) show the effect of different model components and input elements when the model
predicts based on internal memory (external context). The deeper color indicates the greater the impact of knocking
out this part on the original prediction probability. Figure 2c (Figure 2f) shows the effect of MHAs and FFNs on the
last token’s attribute extraction rate when the model predicts based on internal memory (external context).
4.1
Examining Component Functionality
We start by exploring the functionality of model
components (including FFNs and MHAs across
various layers) in knowledge conflicts.
Experiment 1: Knocking Out Component.
We
examine which component in the transformer layer
is critical for the attribute prediction by knocking
out activations. Then, we divide the input into six
elements for analysis: context subject sc, context
relation rc, context attribute ac, question subject
sq, question relation rq, and the last token xN. To
measure the impact on the final prediction results,
we zero-out the updates to the specified input ele-
ment from the MHA and FFN blocks within each
layer. For example, to intervene in the update of
the ℓ-th MHA (FFN) to the input element sc, we set
aℓ′
i = 0 (mℓ′
i = 0) for i in the token range of sc and
ℓ′ = max (1, ℓ−W/2) , · · · , min (L, ℓ+ W/2),
where W denotes the window size. We define the
effect of a model component as the change in the
original prediction probability after knocking it out.
Results.
Figure 2 illustrates the effect of model
components (FFNs and MHAs) in GPT-2 XL with
the window size W = 5. Our observation reveals
that destroying the FFN blocks in the early layers
has a significant effect on the prediction probability
while destroying the FFN blocks at the late layers
shows minimal or no impact (Figures 2a and 2d).
Moreover, the MHA blocks at the middle and late
layers are crucial for the last token (Figures 2b and
2e). A possible explanation of the model’s behav-
ior on the factual recall task is that the early FFNs
first enrich the semantic information of input ele-
ments, and then the enriched semantic information
about attributes is extracted to the last token via
late MHAs, where knowledge conflicts may arise at
the later stage. To verify this hypothesis, we will
examine the attribute extraction function of MHAs.
Experiment 2: Extracting Attributes via MHAs.
We adopt the extraction rate (Geva et al., 2023) to
examine the attribute extraction function of MHAs.
We apply the early exit (Schuster et al., 2021; Geva
et al., 2022) to project the MHA update aℓ
N for the
last token xN over the vocabulary. Then we check
whether the top token tℓof each update aligns with
the attribute t∗predicted at the final layer L:
t∗= arg max
pL
N

,
(7)
tℓ= arg max

σ

ϕ

aℓ
N

.
(8)
We consider that the MHA correctly performs at-
tribute extraction when t∗= tℓ. For comparison,
we also examine the extraction rate of FFNs.
Results.
As illustrated in Figures 2c and 2f, it is
evident that the attribute extraction rate of MHAs
significantly exceeds that of FFNs. Moreover, at-
tribute extraction mainly takes place at the 24-48
layers. Results for GPT-J show similar trends in
Appendix C. The above findings motivate us to
conduct an in-depth study on the information flows
of MHAs from input elements to the last token.
0
8
16
24
32
40
48
Layer
40
20
0
20
40
60
Change in prediction probability (%)
C Subject
C Relation
C Attribute
Q Subject
Q Relation
(a) Prediction based on internal memory.
0
8
16
24
32
40
48
Layer
80
60
40
20
0
20
Change in prediction probability (%)
C Subject
C Relation
C Attribute
Q Subject
Q Relation
(b) Prediction based on external context.
0
8
16
24
32
40
48
Layer
30
20
10
0
10
20
30
Change in prediction probability (%)
C1 Subject
C1 Attribute
C2 Subject
C2 Attribute
Q Subject
Q Relation
(c) Prediction based on internal memory.
Figure 3: Relative change in the prediction probability when blocking the information flow from the input elements
to the last token. Figures 3a and 3b only provide conflicting context. Figure 3c provides both supporting and
conflicting context to internal memory, C1 denotes the supporting context, and C2 denotes the conflicting context.
4.2
Tracing Information Flow
The analysis presented above confirms that the last
token extracts attribute information for prediction
through MHA blocks. Following this, we explore
the order and importance of the information flow
from the various elements to the last token.
Experiment 3: Blocking Information Flow.
We
localize the information propagation from the input
elements (including sc, rc, ac, sq and rq) to the
last token by knocking out attention edges between
them. For example, to block the information flow
from the input element sc to the last token xN in the
layer ℓ, we set the attention weight sℓ,h [N, i] = 0
for i in the token range of sc, h = 1, · · · , M, and
ℓ′ = max (1, ℓ−W/2) , · · · , min (L, ℓ+ W/2).
In this way, we can restrict the last token from
attending to the target element. If blocking the
information propagation between them has a signif-
icant impact on the original prediction probability,
this indicates that it is a crucial information flow.
Results.
Figure 3 illustrates the information flow
in GPT-2 XL with the window size W = 9. We can
observe that in the early to middle layers, blocking
the attention to the question relation leads to a de-
crease in the prediction probability. Similarly, in
the subsequent layers, blocking the attention to the
question subject also results in a decrease in the
prediction probability. This suggests that the criti-
cal relation and subject information in the question
are sequentially transmitted to the last token.
Then, in the middle to late layers, blocking the
attention to the context subject and context attribute
has the opposite effect on the final prediction prob-
ability. Taking Figure 3a as an example (when the
model predicts the attribute based on internal mem-
ory), blocking the attention to the context attribute
can improve the prediction probability, however,
blocking the attention to the context subject can re-
duce the prediction probability. This suggests that
the last token can extract the internal knowledge
from the context subject, and extract the external
knowledge from the context attribute. In addition,
the last token also extracts a certain degree of inter-
nal knowledge from the question subject. Results
for GPT-J show consistent trends in Appendix C.
Overall, this shows that there are two specific
stages in the process of information flow passing to
the last token: (1) the question information is first
passed to the last token; (2) the last token extracts
or copies the attribute from the context subject or
the context attribute. In the later stage, knowledge
conflicts arise during the process of merging incon-
sistent information flows from MHAs.
Experiment 4: Extending to Conflicts between
Contexts.
We extend our analysis to a more com-
plex scenario in which the model is presented with
both supporting context and conflicting context rel-
ative to internal memory. Supporting context and
conflicting context contain am and ac respectively:
C1: The capital of {s} is {am}.
C2: The capital of {s} is {ac}. {q}
We find that GPT-2 XL prefers to choose attributes
consistent with internal memory 97.6% of the time.
Hence, we only analyze the cases where the model
makes predictions based on its internal memory.
Results.
As illustrated in Figure 3c, we can ob-
serve that the question information is first passed to
the last token in the first stage, which is consistent
with the trend of a single conflicting context. In the
second stage, a notable distinction is that the model
no longer extracts the memory attribute from the
subject; instead, it opts for a more straightforward
approach of copying the memory attribute from
the context. The above findings indicate that there
exists a mechanism within MHAs capable of distin-
guishing and selecting between internal knowledge
0
5
10
15
20
25
Head
0
8
16
24
32
40
48
Layer
(a) Memory head.
0
5
10
15
20
25
Head
0
8
16
24
32
40
48
0.00
0.05
0.10
0.15
0.20
0.25
0.30
(b) Context head.
24
32
40
48
0.0
0.1
0.2
0.3
Memory Extraction Rate
Memory Head
Context Head
Other Head
24
32
40
48
Layer
0.0
0.1
0.2
0.3
0.4
Context Extraction Rate
(c) Extraction rate of attention heads.
Figure 4: Memory heads and context heads in GPT-2 XL. Figure 4a shows the important score heatmap for
predicting based on internal memory. Figure 4b shows the important score heatmap for predicting based on external
context. Figure 4c illustrates the memory and context attribute extraction rate of different attention heads.
and external knowledge. This motivates us to con-
duct further analysis of MHAs.
4.3
Looking Deeper into Attention Heads
Attention heads serve as the fundamental compo-
nent of an MHA block. For example, GPT-2 XL
contains a total of 1,200 attention heads. This mo-
tivates us to conduct an investigation into the role
of attention heads in handling knowledge conflicts.
Experiment 5: Discovering Important Heads.
To discover the attention heads that are crucial for
predicting memory attributes or context attributes,
we compute the gradient-based importance score
(Michel et al., 2019; Bansal et al., 2023) for each
head. Given a dataset D with a set of inputs x and
outputs y, the importance score of an attention head
h captures the expected sensitivity of the model to
h and is computed as follows:
Il,h(D) = E(x,y)
Hl,hT ∂L(y | x)
∂Hl,h
 ,
(9)
where L(·) is the loss function of conditional au-
toregressive generation. The proxy score of head h
for predicting internal memory is calculated as:
Sl,h
m (Dm, D′
m) = Il,h(Dm) −Il,h(D′
m),
(10)
where (x, am) ∈Dm denotes the original outputs
are memory attributes, (x, ac) ∈D′
m denotes re-
placing the original outputs with context attributes.
In this way, we can also calculate the proxy score
of head h for predicting external context as:
Sl,h
c (Dc, D′
c) = Il,h(Dc) −Il,h(D′
c).
(11)
We compute the proxy score of each head across
different layers to discover important heads.
Results.
As shown in Figure 4a (Figure 4b), the
deeper color of the red square indicates a more
significant contribution from this attention head to
the model’s predictions based on internal memory
(external context). We can observe that there are a
specific number of attention heads within middle-
to-late layers that play opposite roles in predicting
attributes. Accordingly, we refer to those heads that
contribute to the prediction of memory attributes as
memory heads, and those that facilitate predicting
context attributes as context heads. Therefore, we
claim that they may serve in a mutually exclusive
capacity during knowledge conflicts. The heatmaps
of GPT-J are provided in the Appendix C.
Experiment 6: Extracting Specific Attributes
via Heads.
We further analyze the two types
of heads discovered above to verify their role in
knowledge conflicts. We rank the attention heads in
descending order based on their importance scores,
Sl,h
m for memory and Sl,h
c
for context, subsequently
identifying the top-5% of heads as memory heads
and context heads, respectively. For comparison,
we also randomly choose an additional 5% of the
attention heads as other heads. Then, we examine
their memory extraction rate when tℓ= am, and
context extraction rate when tℓ= ac.
Results.
As shown in Figure 4c, memory heads
and context heads are responsible for extracting dif-
ferent attribute information to the last token with a
significant difference between memory and context
extraction rates. Therefore, we discern that the piv-
otal point at which knowledge conflicts emerge in
LMs is the integration of inconsistent information
flows by memory heads and context heads.
5
Mitigating Knowledge Conflicts
Building on the above insights, we propose a novel
method called Pruning Head via PatH PatcHing
(PH3) to efficiently mitigate knowledge conflicts
by intervening on attention heads without the need
to update model parameters (§5.1). Then, we con-
duct extensive experiments to show that our method
can flexibly control LMs to use internal memory or
external context (§5.2). Moreover, we analyze the
generalization capability of our method (§5.3).
5.1
Method
Our method consists of two stages, first identifying
the important heads through path patching, then
intervening on these heads via structured pruning.
Localizing Memory Heads and Context Heads
via Path Patching.
When we use the gradient-
based method in §4.3 to estimate the importance
score of the target head h, it is subject to interfer-
ence from other heads. The calculated gradients
may not fully reflect the contribution of the target
head, but rather a mixture of the influences from
other heads. Therefore, we adopt the path patch-
ing technique (Goldowsky-Dill et al., 2023; Wang
et al., 2023a) to analyze the causal relationship be-
tween the head h and the output attribute (including
am and ac) in conflicts. To calculate the important
score Sℓ,h
c
of the target head h, our path patching
method consists of three steps shown in Figure 14:
1. Run on the original input x ∈Dc to record
the original activations of all heads;
2. Run on the corrupted input x to record the
corrupted activations of all heads, wherex is:
The capital of {s} is ⟨unk⟩. {q}
where ⟨unk⟩is the special token;
3. Run on the original input x, while keeping
all the heads frozen to their activations on x,
except for the target head h whose activation
is set onx. Then measure the important score
as the change of output logits.
The important score Sℓ,h
c
of head h is computed as:
Sl,h
c (Dc) = E(x)[(Px(ac) −Px(am))
−
Px(ac) −Px(am)

].
(12)
We adopt similar steps to calculate the importance
score Sℓ,h
m of the target head h for memory attribute
prediction in Appendix D. We also provide the
importance score heatmaps of memory and context
heads for various models in Appendix E, and our
method can clearly distinguish between them.
Pruning Attention Heads to Mitigate Knowledge
Conflicts.
By ranking all the attention heads in
ascending order based on the importance score Sl,h
c
(Sl,h
m ), we can prune the top-k% attention heads
that negatively impact the model’s capability to
predict context (memory) attributes, thereby en-
hancing the model’s ability to utilize external con-
text (internal memory). To prune a head h in layer
ℓin practice, we set Hℓ,h to be the zero matrix.
5.2
Experiment
Setups.
We evaluate our method on five datasets,
including World Capital, World Capital D,
Official Language, Country, and Continent.
To verify the generalization of PH3, we only calcu-
late the importance scores of the attention heads on
the World Capital dataset, and then directly eval-
uate PH3 on other datasets. We also select 1,000
test samples from an open-domain QA dataset NQ
(Kwiatkowski et al., 2019), providing the LM with
the top-5 retrieved passages, and ensuring that at
least one relevant passage is among them. We vali-
date the effectiveness of PH3 on eight LMs.
Metrics.
We use the internal memory usage rate
RM =
fm
fm+fc+fo and the external context usage
rate RC =
fc
fm+fc+fo to assess how effectively
the method controls the reliance of LMs on either
internal memory or external context, where fm is
the frequency of relying on internal memory, fc is
the frequency of relying on external context, and
fo is the frequency of other answers. For the open-
domain QA task, we use Recall to evaluate whether
the model can provide correct answers based on the
retrieved passages following Adlakha et al. (2023).
Baselines.
We compare with the following base-
lines: (1) Prompt: We instruct the LM to gener-
ate answers based on internal memory or external
context through specific prompts; (2) CAD: Shi
et al. (2023) leverage contrastive decoding (Li et al.,
2023b) to encourage the LM to attend to its con-
text during generation; (3) Gradient: We replace
our path patching method with the gradient-based
method to discover the attention heads. We select
the optimal pruning rate k on the development set
for both Gradient and PH3. More details about
hyperparameter settings are in Appendix B.2.
Results.
Table 1 shows the results of GPT-2 XL,
GPT-J and LLaMA2-7B, and more results of other
models are in Table 2. Throughout our experiments,
we note the following key observations:
World
World
Official
Capital
Capital D
Language
Country
Continent
Model
Method
RM
RC
RM
RC
RM
RC
RM
RC
RM
RC
Base
59.2
40.8
47.2
52.8
42.2
57.8
37.2
62.8
41.5
58.5
Prompt
12.5
81.2
21.3
71.2
20.3
74.4
24.5
75.5
16.2
43.4
↑Memory
Gradient
72.4
9.8
78.6
10.5
41.5
40.5
39.1
60.2
42.7
46.6
PH3 (Ours)
97.9
0.6
93.3
2.5
74.4
9.8
50.9
36.3
53.1
38.1
GPT-2 XL
Prompt
9.3
87.5
18.9
75.2
17.1
80.7
18.5
81.4
25.5
58.3
CAD
25.0
65.6
12.5
63.6
9.1
80.5
27.2
72.5
22.9
60.4
↑Context
Gradient
44.4
49.0
28.0
58.7
29.6
59.5
36.4
63.4
18.4
51.5
PH3 (Ours)
27.5
68.9
7.7
91.3
20.7
74.8
22.7
75.7
27.7
66.7
+ Prompt
3.6
95.1
5.2
94.4
9.6
88.7
12.6
86.0
20.9
63.8
Base
37.5
62.5
43.1
56.9
41.5
58.5
54.0
46.0
43.2
56.8
Prompt
29.8
67.1
31.6
62.1
23.1
69.1
22.4
77.6
12.5
86.0
↑Memory
Gradient
67.9
8.3
67.6
6.7
39.4
53.4
54.4
45.5
57.2
30.0
PH3 (Ours)
93.3
1.6
76.5
10.8
63.3
25.3
58.9
40.5
75.1
17.6
GPT-J
Prompt
31.9
64.5
15.8
76.4
16.2
70.6
17.9
82.1
7.2
91.4
CAD
2.5
89.9
13.4
68.2
4.7
89.9
17.0
81.8
13.0
80.3
↑Context
Gradient
6.1
88.3
7.9
67.8
5.7
76.4
29.2
70.5
36.8
60.7
PH3 (Ours)
0.2
99.3
0.1
98.4
2.3
90.6
9.5
86.7
8.0
64.9
+ Prompt
0.1
99.5
0.2
97.8
2.0
81.9
1.4
98.6
1.4
90.9
Base
46.3
53.7
95.5
4.0
18.8
80.3
52.9
46.8
30.9
69.1
Prompt
36.0
63.2
96.0
3.7
40.0
59.1
68.2
31.6
77.4
22.6
↑Memory
Gradient
81.0
5.8
95.1
1.6
50.1
47.4
60.0
38.0
64.5
24.5
PH3 (Ours)
98.1
1.2
98.0
1.3
73.7
17.8
76.9
20.6
90.5
8.8
LLaMA2-7B
Prompt
3.2
96.6
92.4
2.2
25.5
73.8
58.2
41.5
19.2
80.3
CAD
1.4
95.5
29.1
70.6
0.0
100.0
13.6
86.1
0.2
98.2
↑Context
Gradient
23.6
63.2
40.1
58.8
25.7
74.6
17.6
82.2
27.1
72.9
PH3 (Ours)
1.6
97.4
19.1
73.4
0.1
99.9
5.2
94.7
0.5
99.4
+ Prompt
0.4
98.8
10.6
85.3
0.0
100.0
2.8
97.0
0.0
100.0
Table 1: Experimental results of GPT-2 XL, GPT-J and LLaMA2-7B on five datasets. Bolds denote the best results.
(1) PH3 significantly outperforms other base-
lines. Experimental results show that PH3 can
not only increase the average internal memory us-
age rate of eight LMs by 44.0%, but also increase
the average external context usage rate by 38.5%.
When PH3 is combined with Prompt, it can more
effectively control the LMs to use external context.
(2) As shown in Table 3, PH3 can also achieve an
average 6.2% Recall improvement on open-domain
QA tasks. By pruning a small number of negative
context heads, PH3 can make LMs generate an-
swers more faithfully based on retrieved passages.
(3) Although Prompt and CAD can effectively
increase the external context usage rate, there are
limitations. CAD cannot directly enhance internal
memory, and Prompt may even have the opposite
effect. In contrast, our method offers a viable solu-
tion to enhance the internal memory usage rate.
5.3
Analysis
We conduct a thorough analysis of the generaliza-
tion ability of PH3. For cross-model generalization,
PH3 is effective across a wide range of models.
This shows that our method is not limited to small
models, but can also be adopted on relatively large
models, including the popular LLaMA2 series. For
cross-relation generalization, by intervening on the
attention heads discovered on World Capital, our
method can also well resolve knowledge conflicts
on other relation types. This indicates that PH3
does not identify attention heads specific to a cer-
tain type of relation. Instead, it identifies universal
memory and context heads. For cross-format gen-
eralization, PH3 can transfer well from triple-form
context to document-form context. This indicates
that our method does not merely remember the rela-
tive positions of elements in context, but is capable
of understanding the external context. Compared
to the Gradient, our method has demonstrated su-
perior generalizability. We also analyze the impact
of the number of pruning heads in Appendix G.
6
Conclusion
In this paper, we perform an exploration into the
mechanism of interpreting knowledge conflicts and
reveal that memory and context heads in later lay-
ers can cause knowledge conflicts when merging
inconsistent information flows. Based on our in-
sights, we propose a novel method called Pruning
Head via PatH PatcHing (PH3), which can miti-
gate knowledge conflicts by pruning those conflict-
ing attention heads. We prove that PH3 can flexi-
bly control LMs to use internal memory or exter-
nal context. We also demonstrate the cross-model,
cross-relation, and cross-format generalization.
Limitations
For further study, we conclude some limitations of
our work as follows:
• Similar to previous works on mechanism in-
terpretability that adopt tasks such as antonym
generation (Todd et al., 2023), fact recall
(Meng et al., 2022; Geva et al., 2023), arith-
metic operation (Hanna et al., 2023; Stolfo
et al., 2023), and text classification (Bansal
et al., 2023; Wang et al., 2023b), our work also
selects a relatively simpler task to interpret the
mechanism behind knowledge conflicts. Sim-
ple tasks enable us to better control variables
and minimize external distractions. In the fu-
ture, we plan to extend our analysis to more
complex and realistic scenarios, such as where
irrelevant information is present within the ex-
ternal context, or where the model needs to
reason with both internal and external knowl-
edge.
• Although our research has delved into the at-
tention heads in LMs, there may be more ba-
sic elements involved in knowledge conflicts.
Furthermore, the memory and context heads
we have discovered may not only be respon-
sible for extracting knowledge from internal
memory or external context. These heads may
also have other functions, such as helping the
model capture global dependencies of input
texts. By pruning these heads, the original ca-
pabilities of the model may be affected. There-
fore, we will further explore mitigating knowl-
edge conflicts through more subtle interven-
tion methods.
In summary, the mechanism behind knowledge
conflicts remains a largely unexplored area, and we
hope our work can offer some useful insights for
further research.
Ethics Statement
To enhance the reproducibility of our research, we
will make all source code and datasets publicly
available upon the acceptance of this paper. Our
work focuses on uncovering the mechanisms be-
hind knowledge conflicts in LM, thereby better
controlling the model in retrieval augmentation and
tool augmentation. Through effective intervention,
our method can make the LM more controllable
and trustworthy. On the one hand, it can prevent
prompt injections from attacking the model, and
on the other hand, it can correct the biased knowl-
edge that the model learned during pre-training.
Nonetheless, the impact of head pruning on the
model’s original capabilities remains unexplored.
These factors should be taken into careful consider-
ation for future research.
References
Vaibhav Adlakha, Parishad BehnamGhader, Xing Han
Lu, Nicholas Meade, and Siva Reddy. 2023. Eval-
uating correctness and faithfulness of instruction-
following models for question answering.
arXiv
preprint arXiv:2307.16877.
Hritik Bansal, Karthik Gopalakrishnan, Saket Dingliwal,
Sravan Bodapati, Katrin Kirchhoff, and Dan Roth.
2023. Rethinking the role of scale for in-context
learning: An interpretability-based case study at 66
billion scale. In Proceedings of the 61st Annual Meet-
ing of the Association for Computational Linguis-
tics (Volume 1: Long Papers), pages 11833–11856,
Toronto, Canada. Association for Computational Lin-
guistics.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory
Anthony, Herbie Bradley, Kyle O’Brien, Eric Hal-
lahan, Mohammad Aflah Khan, Shivanshu Purohit,
USVSN Sai Prashanth, Edward Raff, et al. 2023.
Pythia: A suite for analyzing large language mod-
els across training and scaling.
In International
Conference on Machine Learning, pages 2397–2430.
PMLR.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie
Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind
Neelakantan, Pranav Shyam, Girish Sastry, Amanda
Askell, Sandhini Agarwal, Ariel Herbert-Voss,
Gretchen Krueger, Tom Henighan, Rewon Child,
Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens
Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma-
teusz Litwin, Scott Gray, Benjamin Chess, Jack
Clark, Christopher Berner, Sam McCandlish, Alec
Radford, Ilya Sutskever, and Dario Amodei. 2020.
Language models are few-shot learners.
In Ad-
vances in Neural Information Processing Systems,
volume 33, pages 1877–1901. Curran Associates,
Inc.
Nick Cammarata, Shan Carter, Gabriel Goh, Chris Olah,
Michael Petrov, Ludwig Schubert, Chelsea Voss, Ben
Egan, and Swee Kiat Lim. 2020. Thread: Circuits.
Distill. Https://distill.pub/2020/circuits.
Hung-Ting Chen, Michael Zhang, and Eunsol Choi.
2022. Rich knowledge sources bring complex knowl-
edge conflicts: Recalibrating models to reflect con-
flicting evidence. In Proceedings of the 2022 Con-
ference on Empirical Methods in Natural Language
Processing, pages 2292–2307, Abu Dhabi, United
Arab Emirates. Association for Computational Lin-
guistics.
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao
Chang, and Furu Wei. 2022. Knowledge neurons in
pretrained transformers. In Proceedings of the 60th
Annual Meeting of the Association for Computational
Linguistics (Volume 1: Long Papers), pages 8493–
8502, Dublin, Ireland. Association for Computational
Linguistics.
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom
Henighan, Nicholas Joseph, Ben Mann, Amanda
Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al.
2021. A mathematical framework for transformer
circuits. Transformer Circuits Thread, 1.
Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir
Globerson. 2023. Dissecting recall of factual associa-
tions in auto-regressive language models. In Proceed-
ings of the 2023 Conference on Empirical Methods in
Natural Language Processing, pages 12216–12235,
Singapore. Association for Computational Linguis-
tics.
Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Gold-
berg. 2022. Transformer feed-forward layers build
predictions by promoting concepts in the vocabulary
space. In Proceedings of the 2022 Conference on
Empirical Methods in Natural Language Process-
ing, pages 30–45, Abu Dhabi, United Arab Emirates.
Association for Computational Linguistics.
Mor Geva, Roei Schuster, Jonathan Berant, and Omer
Levy. 2021. Transformer feed-forward layers are key-
value memories. In Proceedings of the 2021 Confer-
ence on Empirical Methods in Natural Language Pro-
cessing, pages 5484–5495, Online and Punta Cana,
Dominican Republic. Association for Computational
Linguistics.
Nicholas Goldowsky-Dill, Chris MacLeod, Lucas
Sato, and Aryaman Arora. 2023.
Localizing
model behavior with path patching. arXiv preprint
arXiv:2304.05969.
Danny Halawi, Jean-Stanislas Denain, and Jacob Stein-
hardt. 2023. Overthinking the truth: Understanding
how language models process false demonstrations.
arXiv preprint arXiv:2307.09476.
Michael Hanna, Ollie Liu, and Alexandre Variengien.
2023. How does GPT-2 compute greater-than?: In-
terpreting mathematical abilities in a pre-trained lan-
guage model. In Thirty-seventh Conference on Neu-
ral Information Processing Systems.
Yaru Hao, Li Dong, Furu Wei, and Ke Xu. 2021. Self-
attention attribution: Interpreting information inter-
actions inside transformer. Proceedings of the AAAI
Conference on Artificial Intelligence, 35(14):12963–
12971.
Roee Hendel, Mor Geva, and Amir Globerson. 2023.
In-context learning creates task vectors. In Find-
ings of the Association for Computational Linguis-
tics: EMNLP 2023, pages 9318–9333, Singapore.
Association for Computational Linguistics.
Zhuoran Jin, Pengfei Cao, Yubo Chen, Kang Liu, Xiao-
jian Jiang, Jiexin Xu, Qiuxia Li, and Jun Zhao. 2024.
Tug-of-war between knowledge: Exploring and re-
solving knowledge conflicts in retrieval-augmented
language models. arXiv preprint arXiv:2402.14409.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red-
field, Michael Collins, Ankur Parikh, Chris Alberti,
Danielle Epstein, Illia Polosukhin, Jacob Devlin, Ken-
ton Lee, Kristina Toutanova, Llion Jones, Matthew
Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob
Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural
Questions: A Benchmark for Question Answering
Research. Transactions of the Association for Com-
putational Linguistics, 7:453–466.
Daliang Li, Ankit Singh Rawat, Manzil Zaheer, Xin
Wang, Michal Lukasik, Andreas Veit, Felix Yu, and
Sanjiv Kumar. 2023a. Large language models with
controllable working memory. In Findings of the As-
sociation for Computational Linguistics: ACL 2023,
pages 1774–1793, Toronto, Canada. Association for
Computational Linguistics.
Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang,
Jason Eisner, Tatsunori Hashimoto, Luke Zettle-
moyer, and Mike Lewis. 2023b. Contrastive decod-
ing: Open-ended text generation as optimization. In
Proceedings of the 61st Annual Meeting of the As-
sociation for Computational Linguistics (Volume 1:
Long Papers), pages 12286–12312, Toronto, Canada.
Association for Computational Linguistics.
Shayne Longpre, Kartik Perisetla, Anthony Chen,
Nikhil Ramesh, Chris DuBois, and Sameer Singh.
2021. Entity-based knowledge conflicts in question
answering. In Proceedings of the 2021 Conference
on Empirical Methods in Natural Language Process-
ing, pages 7052–7063, Online and Punta Cana, Do-
minican Republic. Association for Computational
Linguistics.
Kevin Meng, David Bau, Alex Andonian, and Yonatan
Belinkov. 2022. Locating and editing factual associ-
ations in gpt. Advances in Neural Information Pro-
cessing Systems, 35:17359–17372.
Paul Michel, Omer Levy, and Graham Neubig. 2019.
Are sixteen heads really better than one?
In Ad-
vances in Neural Information Processing Systems,
volume 32. Curran Associates, Inc.
Ella Neeman, Roee Aharoni, Or Honovich, Leshem
Choshen, Idan Szpektor, and Omri Abend. 2023.
DisentQA: Disentangling parametric and contextual
knowledge with counterfactual question answering.
In Proceedings of the 61st Annual Meeting of the
Association for Computational Linguistics (Volume 1:
Long Papers), pages 10056–10070, Toronto, Canada.
Association for Computational Linguistics.
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas
Joseph, Nova DasSarma, Tom Henighan, Ben Mann,
Amanda Askell, Yuntao Bai, Anna Chen, et al. 2022.
In-context learning and induction heads.
arXiv
preprint arXiv:2209.11895.
OpenAI. 2023. Gpt-4 technical report.
Cheng Qian, Xinran Zhao, and Sherry Tongshuang Wu.
2023. "merge conflicts!" exploring the impacts of
external distractors to parametric knowledge graphs.
ArXiv, abs/2309.08594.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan,
Dario Amodei, Ilya Sutskever, et al. 2019. Language
models are unsupervised multitask learners. OpenAI
blog, 1(8):9.
Mansi Sakarvadia, Aswathy Ajith, Arham Khan, Daniel
Grzenda, Nathaniel Hudson, André Bauer, Kyle
Chard, and Ian Foster. 2023a. Memory injections:
Correcting multi-hop reasoning failures during in-
ference in transformer-based language models. In
Proceedings of the 6th BlackboxNLP Workshop: An-
alyzing and Interpreting Neural Networks for NLP,
pages 342–356, Singapore. Association for Compu-
tational Linguistics.
Mansi Sakarvadia, Arham Khan, Aswathy Ajith, Daniel
Grzenda, Nathaniel Hudson, André Bauer, Kyle
Chard, and Ian Foster. 2023b. Attention lens: A tool
for mechanistically interpreting the attention head
information retrieval mechanism.
Tal Schuster, Adam Fisch, Tommi Jaakkola, and Regina
Barzilay. 2021. Consistent accelerated inference via
confident adaptive transformers. In Proceedings of
the 2021 Conference on Empirical Methods in Natu-
ral Language Processing, pages 4962–4979, Online
and Punta Cana, Dominican Republic. Association
for Computational Linguistics.
Weijia Shi, Xiaochuang Han, Mike Lewis, Yulia
Tsvetkov, Luke Zettlemoyer, and Scott Wen-tau
Yih. 2023.
Trusting your evidence: Hallucinate
less with context-aware decoding. arXiv preprint
arXiv:2305.14739.
Alessandro Stolfo, Yonatan Belinkov, and Mrinmaya
Sachan. 2023. A mechanistic interpretation of arith-
metic reasoning in language models using causal
mediation analysis. In Proceedings of the 2023 Con-
ference on Empirical Methods in Natural Language
Processing, pages 7035–7052, Singapore. Associa-
tion for Computational Linguistics.
Eric Todd, Millicent L. Li, Arnab Sen Sharma, Aaron
Mueller, Byron C. Wallace, and David Bau. 2023.
Function vectors in large language models.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Al-
bert, Amjad Almahairi, Yasmine Babaei, Nikolay
Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti
Bhosale, et al. 2023.
Llama 2:
Open founda-
tion and fine-tuned chat models.
arXiv preprint
arXiv:2307.09288.
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov,
Sharon Qian, Daniel Nevo, Yaron Singer, and Stu-
art Shieber. 2020. Investigating gender bias in lan-
guage models using causal mediation analysis. In
Advances in Neural Information Processing Systems,
volume 33, pages 12388–12401. Curran Associates,
Inc.
Ben Wang and Aran Komatsuzaki. 2021.
GPT-J-
6B: A 6 Billion Parameter Autoregressive Lan-
guage Model. https://github.com/kingoflolz/
mesh-transformer-jax.
Kevin Ro Wang, Alexandre Variengien, Arthur Conmy,
Buck Shlegeris, and Jacob Steinhardt. 2023a. Inter-
pretability in the wild: a circuit for indirect object
identification in GPT-2 small. In The Eleventh Inter-
national Conference on Learning Representations.
Lean Wang, Lei Li, Damai Dai, Deli Chen, Hao Zhou,
Fandong Meng, Jie Zhou, and Xu Sun. 2023b. Label
words are anchors: An information flow perspective
for understanding in-context learning. In Proceed-
ings of the 2023 Conference on Empirical Methods
in Natural Language Processing, pages 9840–9855,
Singapore. Association for Computational Linguis-
tics.
Yike Wang, Shangbin Feng, Heng Wang, Weijia
Shi, Vidhisha Balachandran, Tianxing He, and Yu-
lia Tsvetkov. 2023c.
Resolving knowledge con-
flicts in large language models.
arXiv preprint
arXiv:2310.00935.
Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, and
Yu Su. 2023.
Adaptive chameleon or stubborn
sloth: Unraveling the behavior of large language
models in knowledge conflicts.
arXiv preprint
arXiv:2305.13300.
Yi Yang, Hanyu Duan, Ahmed Abbasi, John P. Lalor,
and Kar Yan Tam. 2023. Bias a-head? analyzing
bias in transformer-based language model attention
heads.
Qinan Yu, Jack Merullo, and Ellie Pavlick. 2023. Char-
acterizing mechanisms for factual recall in language
models. In Proceedings of the 2023 Conference on
Empirical Methods in Natural Language Processing,
pages 9924–9959, Singapore. Association for Com-
putational Linguistics.
Fred Zhang and Neel Nanda. 2024. Towards best prac-
tices of activation patching in language models: Met-
rics and methods.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel
Artetxe, Moya Chen, Shuohui Chen, Christopher De-
wan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mi-
haylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel
Simig, Punit Singh Koura, Anjali Sridhar, Tianlu
Wang, and Luke Zettlemoyer. 2022. Opt: Open pre-
trained transformer language models.
Wenxuan Zhou, Sheng Zhang, Hoifung Poon, and
Muhao Chen. 2023.
Context-faithful prompting
for large language models. In Findings of the As-
sociation for Computational Linguistics: EMNLP
2023, pages 14544–14556, Singapore. Association
for Computational Linguistics.
A
Related Work
A.1
Investigating Knowledge Conflict
Previous research (Longpre et al., 2021; Chen et al.,
2022; Yu et al., 2023; Xie et al., 2023; Wang et al.,
2023c; Neeman et al., 2023; Jin et al., 2024) on
knowledge conflicts primarily seek to answer the
question: do language models prefer internal mem-
ory or external context? Yu et al. (2023) find that
language models are more inclined to internal mem-
ory as the frequency of a fact in the pre-training
corpus increases. Xie et al. (2023) demonstrate that
large language models (LLMs) are highly receptive
to external conflicting evidence. They also reveal
that when both supportive and contradictory evi-
dence to their internal memory are present, LLMs
show a strong confirmation bias and tend to cling
to their parametric memory. The above observed
phenomena contribute to a better understanding
of knowledge conflicts. However, the underlying
mechanism of knowledge conflicts remains unclear.
We observe that knowledge conflicts arise when the
late attention heads integrate different information
flows from internal memory and external context.
A.2
Resolving Knowledge Conflict
Existing work (Shi et al., 2023; Zhou et al., 2023;
Li et al., 2023a; Yu et al., 2023; Qian et al., 2023)
has conducted preliminary exploration into the mit-
igation of knowledge conflicts. Shi et al. (2023)
propose a simple method to encourage the LM to
attend to the external context via contrastive decod-
ing (Li et al., 2023b). Yu et al. (2023) use head
attribution to identify individual attention heads
that either promote the memorized answer or the
in-context answer, then scale the value vector of
these heads to increase the rate of the in-context an-
swers. Our work is inspired by their exploration of
attention heads, and we propose further analysis to
improve understanding of the way knowledge con-
flicts are formed. Furthermore, while most existing
methods (Shi et al., 2023; Yu et al., 2023) primarily
focus on improving the model’s faithfulness to the
context, enabling the model to adhere to its internal
memory remains a challenging task.
A.3
Mechanistic Interpretability
Recently, there has been a growing interest in
the mechanistic interpretability (Cammarata et al.,
2020; Elhage et al., 2021) of parametric knowledge
in LMs, with efforts focusing on reverse engineer-
ing the computational processes of model parame-
ters. Dai et al. (2022) use a knowledge attribution
method (Hao et al., 2021) to identify the knowledge
neurons in FFNs. Meng et al. (2022) reveal that
FFNs at a range of middle layers can recall facts by
using the causal mediation analysis method (Vig
et al., 2020). Geva et al. (2023) find that knowledge
extraction is typically done via attention heads. Be-
sides, there are some works investigating LMs in
mathematical reasoning (Hanna et al., 2023; Stolfo
et al., 2023) and in-context learning (Hendel et al.,
2023; Olsson et al., 2022; Bansal et al., 2023). Be-
sides, there are some studies (Yang et al., 2023;
Sakarvadia et al., 2023a,b; Zhang and Nanda, 2024)
focused on interpreting attention heads in LMs.
Our work is highly inspired by previous wisdom in
mechanistic interpretability, focusing on interpret-
ing and mitigating knowledge conflicts in LMs.
B
Implement Details
B.1
Datasets
We construct Official Language, Country, and
Continent datasets by sampling knowledge triples
from Wikidata. The Official Language dataset
requires the LM to predict the official language of
the given city or country:
The official language of {s} is {ac}.
Q: What is the official language of {s}? A:
The Country dataset requires the LM to predict the
country to which the given city belongs:
The city {s} is located in {ac}.
Q: Which country is the city {s} in ? A:
The Continent dataset requires the LM to predict
the continent on which the given country is located:
{s} is in the continent of {ac}.
Q: Which continent is {s} located in ? A:
We also generate a more complex World Capital
D dataset based on the World Capital dataset, us-
ing gpt-3.5-turbo to rewrite the external context
from triplet form into document form.
B.2
Hyperparameter Settings
Our implementation is based on HuggingFace’s
Transformers1, PyTorch2 and baukit3.
For the
Prompt method, we use the following prompt to
enhance the internal memory:
Please answer the question based on your
internal memory, ignoring the given context.
1https://github.com/huggingface/transformers/
2https://github.com/pytorch/pytorch/
3https://github.com/davidbau/baukit/
0
4
8
12
16
20
24
28
C Subject
C Relation
C Attribute
Q Subject
Q Relation
Last
0.0
0.2
0.4
0.6
0.8
Figure 5: Effect of FFNs in GPT-J on internal memory.
0
4
8
12
16
20
24
28
C Subject
C Relation
C Attribute
Q Subject
Q Relation
Last
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.35
0.40
Figure 6: Effect of MHAs in GPT-J on internal memory.
and we use the following prompt to enhance the
external context:
Please answer the question based on the
given context, ignoring your internal memory.
For Gradient and PH3, we select the optimal prun-
ing rate k ∈{1, 3, 5, 7, 9, 15} on the development
set with 200 samples. To mitigate knowledge con-
flicts, setting the pruning rate k of PH3 to 5 usually
achieves excellent results. For enhancing the open-
domain QA capabilities, we usually set the pruning
rate k of PH3 to 3. Details about the models used in
this paper are in Table 4. All experiments are con-
ducted with NVIDIA GeForce RTX A6000 GPUs.
C
Additional Results for GPT-J
We provide here additional results for GPT-J. Fig-
ures 5 and 6 show the effect of FFNs and MHAs
on internal memory, and Figures 7 and 8 show the
effect of FFNs and MHAs on external context. Fig-
ures 9 and 10 illustrate the information flow in GPT-
J with the window size W = 9. Figure 11 shows
the information flow in GPT-J when providing both
supporting context and conflicting context relative
to internal memory. Figures 12 and 13 show the
gradient-based important scores of memory heads
and context heads in GPT-J.
0
4
8
12
16
20
24
28
C Subject
C Relation
C Attribute
Q Subject
Q Relation
Last
0.800
0.825
0.850
0.875
0.900
0.925
0.950
0.975
Figure 7: Effect of FFNs in GPT-J on external context.
0
4
8
12
16
20
24
28
C Subject
C Relation
C Attribute
Q Subject
Q Relation
Last
0.78
0.80
0.82
0.84
0.86
0.88
Figure 8: Effect of MHAs in GPT-J on external context.
D
Method Details
To calculate the important score Sℓ,h
m of the target
head h, our path patching method consists of the
following three steps:
1. Run on the original input x ∈Dm to record
the original activations of all heads;
2. Run on the corrupted input x to record the
corrupted activations of all heads, wherex is:
The capital of ⟨unk⟩is {ac}.
Q: What is the capital of ⟨unk⟩? A:
where ⟨unk⟩is the special token;
3. Run on the original input x, while keeping
all the heads frozen to their activations on x,
except for the target head h whose activation
is set onx. Then measure the important score
as the change of output logits.
The important score Sℓ,h
m of head h is computed as:
Sl,h
m (Dm) = E(x)[(Px(am) −Px(ac))
−
Px(am) −Px(ac)

].
(13)
E
Heatmaps of Attention Heads
We calculate the important scores of memory heads
and context heads via our path patching method,
0
4
8
12
16
20
24
28
Layer
80
60
40
20
0
20
40
60
80
Change in prediction probability (%)
C Subject
C Relation
C Attribute
Q Subject
Q Relation
Figure 9: Relative change in the GPT-J’s prediction
probability based on internal memory.
0
4
8
12
16
20
24
28
Layer
100
80
60
40
20
0
20
Change in prediction probability (%)
C Subject
C Relation
C Attribute
Q Subject
Q Relation
Figure 10: Relative change in the GPT-J’s prediction
probability based on external context.
then provide the heatmaps for GPT-2 XL (Figures
15 and 16), GPT-J (Figures 17 and 18), OPT-1.3B
(Figures 19 and 20), OPT-2.7B (Figures 21 and
22), Pythia-6.9B (Figures 23 and 24), Pythia-12B
(Figures 25 and 26), LLaMA2-7B (Figures 27 and
28) and LLaMA2-13B (Figures 29 and 30). The
red squares indicate heads that have a significant
positive impact, while the blue squares represent
heads that have a negative effect.
F
Additional Experimental Results
We report experimental results in Table 2 and 3.
G
Number of Pruning Heads
As shown in Figures 31, 32, 33, 34, 35, 36, 37,
38, 39, 40, 41 and 42, we analyze the impact of
the number of pruning heads (sparsity ratio) on the
Gradient and PH3 methods.
0
4
8
12
16
20
24
28
Layer
60
40
20
0
20
Change in prediction probability (%)
C1 Subject
C1 Attribute
C2 Subject
C2 Attribute
Q Subject
Q Relation
Figure 11: Relative change in the GPT-J’s prediction
probability based on internal memory when providing
both supporting context and conflicting context.
0
4
8
12
16
Head
0
4
8
12
16
20
24
28
Layer
0.0
0.2
0.4
0.6
0.8
1.0
1.2
Figure 12: Memory Heads of GPT-J.
0
4
8
12
16
Head
0
4
8
12
16
20
24
28
Layer
0.0
0.1
0.2
0.3
0.4
0.5
0.6
Figure 13: Context Heads of GPT-J.
The capital of France is Rome.
What is the capital of France?
The capital of France is Rome.
What is the capital of France?
Original Input
Rome
𝜕ℒ
)
𝜕𝚮𝑙,𝑗(
)
Original Input
Rome
Original
Corrupted Input
Paris
Original Input
ℙ( Rome ) - ℙ(            )
Paris
Original Input
Corrupted Input
Original Head Activation
Corrupted Head Activation
FFN Activation
(1) Gradient-based method.
(2) Path patching method (Ours).
(a) Original run.
(c) Run with replacing activations.
(b) Corrupted run.
Gradient Flow
Freezing with Original Activation
Patching with Corrupted Activation
Rome
Context Attribute (Original Prediction)
Memory Attribute
Paris
Patched Path
Figure 14: Illustration of gradient-based method and our path patching method.
0
5
10
15
20
25
Head
0
8
16
24
32
40
48
Layer
0.005
0.000
0.005
0.010
0.015
Figure 15: Memory Heads of GPT-2 XL.
0
5
10
15
20
25
Head
0
8
16
24
32
40
48
Layer
0.06
0.04
0.02
0.00
0.02
0.04
0.06
Figure 16: Context Heads of GPT-2 XL.
0
4
8
12
16
Head
0
4
8
12
16
20
24
28
Layer
0.05
0.00
0.05
0.10
0.15
0.20
0.25
Figure 17: Memory Heads of GPT-J.
0
4
8
12
16
Head
0
4
8
12
16
20
24
28
Layer
0.15
0.10
0.05
0.00
0.05
Figure 18: Context Heads of GPT-J.
0
4
8
12
16
20
24
28
32
Head
0
4
8
12
16
20
24
Layer
0.050
0.025
0.000
0.025
0.050
0.075
0.100
0.125
Figure 19: Memory Heads of OPT-1.3B.
0
4
8
12
16
20
24
28
32
Head
0
4
8
12
16
20
24
Layer
0.04
0.02
0.00
0.02
0.04
0.06
0.08
Figure 20: Context Heads of OPT-1.3B.
0
4
8
12
16
20
24
28
32
Head
0
4
8
12
16
20
24
28
32
Layer
0.04
0.02
0.00
0.02
0.04
0.06
0.08
0.10
Figure 21: Memory Heads of OPT-2.7B.
0
4
8
12
16
20
24
28
32
Head
0
4
8
12
16
20
24
28
32
Layer
0.02
0.00
0.02
0.04
0.06
Figure 22: Context Heads of OPT-2.7B.
0
4
8
12
16
20
24
28
32
Head
0
4
8
12
16
20
24
28
32
Layer
0.02
0.00
0.02
0.04
0.06
0.08
Figure 23: Memory Heads of Pythia-6.9B.
0
4
8
12
16
20
24
28
32
Head
0
4
8
12
16
20
24
28
32
Layer
0.02
0.00
0.02
0.04
0.06
0.08
0.10
Figure 24: Context Heads of Pythia-6.9B.
0
8
16
24
32
40
Head
0
4
8
12
16
20
24
28
32
36
Layer
0.02
0.01
0.00
0.01
0.02
0.03
0.04
0.05
Figure 25: Memory Heads of Pythia-12B.
0
8
16
24
32
40
Head
0
4
8
12
16
20
24
28
32
36
Layer
0.03
0.02
0.01
0.00
0.01
0.02
0.03
Figure 26: Context Heads of Pythia-12B.
0
4
8
12
16
20
24
28
32
Head
0
4
8
12
16
20
24
28
32
Layer
0.02
0.00
0.02
0.04
0.06
0.08
0.10
Figure 27: Memory Heads of LLaMA2-7B.
0
4
8
12
16
20
24
28
32
Head
0
4
8
12
16
20
24
28
32
Layer
0.04
0.02
0.00
0.02
0.04
0.06
0.08
0.10
Figure 28: Context Heads of LLaMA2-7B.
0
8
16
24
32
40
Head
0
8
16
24
32
40
Layer
0.02
0.00
0.02
0.04
0.06
0.08
0.10
0.12
0.14
Figure 29: Memory Heads of LLaMA2-13B.
0
8
16
24
32
40
Head
0
8
16
24
32
40
Layer
0.02
0.00
0.02
0.04
0.06
0.08
0.10
Figure 30: Context Heads of LLaMA2-13B.
0
5
10
15
20
Sparsity Ratio (%)
0
20
40
60
80
100
Usage Rate (%)
Memory Usage Rate (PH3)
Context Usage Rate (PH3)
Memory Usage Rate (Gradient)
Context Usage Rate (Gradient)
Figure 31: Impact of GPT-2 XL’s sparsity ratio on im-
proving internal memory usage rate.
0
5
10
15
20
Sparsity Ratio (%)
0
20
40
60
80
100
Usage Rate (%)
Memory Usage Rate (PH3)
Context Usage Rate (PH3)
Memory Usage Rate (Gradient)
Context Usage Rate (Gradient)
Figure 32: Impact of GPT-2 XL’s sparsity ratio on im-
proving external context usage rate.
0
5
10
15
20
Sparsity Ratio (%)
0
20
40
60
80
100
Usage Rate (%)
Memory Usage Rate (PH3)
Context Usage Rate (PH3)
Memory Usage Rate (Gradient)
Context Usage Rate (Gradient)
Figure 33: Impact of GPT-J’s sparsity ratio on improv-
ing internal memory usage rate.
0
5
10
15
20
Sparsity Ratio (%)
0
20
40
60
80
100
Usage Rate (%)
Memory Usage Rate (PH3)
Context Usage Rate (PH3)
Memory Usage Rate (Gradient)
Context Usage Rate (Gradient)
Figure 34: Impact of GPT-J’s sparsity ratio on improv-
ing external context usage rate.
0
5
10
15
20
Sparsity Ratio (%)
0
20
40
60
80
100
Usage Rate (%)
Memory Usage Rate (PH3)
Context Usage Rate (PH3)
Memory Usage Rate (Gradient)
Context Usage Rate (Gradient)
Figure 35: Impact of OPT-2.7B’s sparsity ratio on im-
proving internal memory usage rate.
0
5
10
15
20
Sparsity Ratio (%)
0
20
40
60
80
100
Usage Rate (%)
Memory Usage Rate (PH3)
Context Usage Rate (PH3)
Memory Usage Rate (Gradient)
Context Usage Rate (Gradient)
Figure 36: Impact of OPT-2.7B’s sparsity ratio on im-
proving external context usage rate.
0
5
10
15
20
Sparsity Ratio (%)
0
20
40
60
80
100
Usage Rate (%)
Memory Usage Rate (PH3)
Context Usage Rate (PH3)
Memory Usage Rate (Gradient)
Context Usage Rate (Gradient)
Figure 37: Impact of Pythia-6.9B’s sparsity ratio on
improving internal memory usage rate.
0
5
10
15
20
Sparsity Ratio (%)
0
20
40
60
80
100
Usage Rate (%)
Memory Usage Rate (PH3)
Context Usage Rate (PH3)
Memory Usage Rate (Gradient)
Context Usage Rate (Gradient)
Figure 38: Impact of Pythia-6.9B’s sparsity ratio on
improving external context usage rate.
0
5
10
15
20
Sparsity Ratio (%)
0
20
40
60
80
100
Usage Rate (%)
Memory Usage Rate (PH3)
Context Usage Rate (PH3)
Memory Usage Rate (Gradient)
Context Usage Rate (Gradient)
Figure 39: Impact of LLaMA2-7B’s sparsity ratio on
improving internal memory usage rate.
0
5
10
15
20
Sparsity Ratio (%)
0
20
40
60
80
100
Usage Rate (%)
Memory Usage Rate (PH3)
Context Usage Rate (PH3)
Memory Usage Rate (Gradient)
Context Usage Rate (Gradient)
Figure 40: Impact of LLaMA2-7B’s sparsity ratio on
improving external context usage rate.
0
5
10
15
20
Sparsity Ratio (%)
0
20
40
60
80
100
Usage Rate (%)
Memory Usage Rate (PH3)
Context Usage Rate (PH3)
Memory Usage Rate (Gradient)
Context Usage Rate (Gradient)
Figure 41: Impact of LLaMA2-13B’s sparsity ratio on
improving internal memory usage rate.
0
5
10
15
20
Sparsity Ratio (%)
0
20
40
60
80
100
Usage Rate (%)
Memory Usage Rate (PH3)
Context Usage Rate (PH3)
Memory Usage Rate (Gradient)
Context Usage Rate (Gradient)
Figure 42: Impact of LLaMA2-13B’s sparsity ratio on
improving external context usage rate.
World
World
Official
Capital
Capital D
Language
Country
Continent
Model
Method
RM
RC
RM
RC
RM
RC
RM
RC
RM
RC
Base
40.5
59.5
36.3
63.7
20.3
79.7
26.8
73.2
19.2
80.8
Prompt
19.7
78.4
37.5
57.2
7.9
91.4
16.9
82.6
7.9
90.1
↑Memory
Gradient
82.7
12.2
56.4
21.2
37.1
50.4
38.0
61.1
20.5
55.9
PH3 (Ours)
95.0
0.2
87.2
1.9
70.3
16.6
47.7
49.1
43.4
51.7
OPT-1.3B
Prompt
17.0
81.4
38.5
57.7
9.3
89.5
15.9
83.8
6.3
93.3
CAD
8.1
86.5
31.6
60.0
3.2
89.6
0.1
99.5
5.1
89.1
↑Context
Gradient
22.9
73.7
35.7
63.8
12.4
82.5
16.3
82.9
17.6
80.2
PH3 (Ours)
0.2
97.0
7.9
69.6
12.8
84.1
4.0
85.4
1.6
44.1
+ Prompt
0.4
99.2
10.8
68.8
9.3
88.3
2.4
92.5
2.5
92.1
Base
40.2
59.8
46.6
53.4
8.8
91.2
26.5
73.5
4.3
95.7
Prompt
17.7
80.3
24.2
71.6
3.4
96.3
12.7
87.2
1.6
98.1
↑Memory
Gradient
75.4
19.3
10.3
79.7
13.3
57.7
42.9
56.3
5.3
94.1
PH3 (Ours)
93.4
0.5
2.8
87.6
75.6
1.5
56.3
38.9
29.3
55.5
OPT-2.7B
Prompt
4.7
94.9
11.1
86.9
3.2
96.3
13.2
86.5
0.7
99.0
CAD
10.8
72.2
28.8
46.3
2.7
87.5
9.0
89.1
0.5
99.0
↑Context
Gradient
36.1
62.9
43.3
53.5
7.3
91.6
23.7
76.3
4.1
95.9
PH3 (Ours)
1.3
97.8
3.4
81.6
4.1
94.9
9.0
90.9
0.9
98.6
+ Prompt
0.8
98.3
1.3
94.6
1.5
98.2
6.9
93.1
0.0
99.5
Base
53.3
46.6
74.8
25.2
49.7
50.3
41.3
58.7
39.1
60.9
Prompt
44.7
51.2
41.8
37.8
16.5
81.4
12.8
87.1
36.2
61.6
↑Memory
Gradient
56.5
35.8
72.8
22.4
56.7
36.3
37.9
62.1
40.1
58.9
PH3 (Ours)
90.2
6.7
88.4
10.2
71.5
11.1
41.8
57.3
66.0
31.0
Pythia-6.9B
Prompt
32.7
63.7
32.0
44.8
8.9
90.5
10.4
89.5
31.7
75.6
CAD
14.3
55.1
22.0
27.6
3.3
78.1
8.5
91.2
12.3
82.2
↑Context
Gradient
41.4
53.7
61.8
35.6
48.2
51.3
34.3
65.6
41.7
53.0
PH3 (Ours)
6.7
81.7
34.4
30.0
16.4
70.0
3.4
96.3
3.3
94.5
+ Prompt
0.6
98.6
24.4
60.4
3.3
95.6
0.3
99.5
0.7
98.8
Base
59.7
40.3
64.6
35.4
34.3
65.7
35.0
65.0
43.0
57.0
Prompt
5.8
94.1
43.7
53.3
11.1
85.8
2.4
97.5
10.1
88.5
↑Memory
Gradient
62.9
27.7
63.1
30.2
39.3
37.5
46.2
53.3
42.1
56.9
PH3 (Ours)
95.0
0.6
82.4
2.2
69.9
6.9
57.1
35.9
70.1
9.6
Pythia-12B
Prompt
6.2
93.6
34.1
62.0
21.1
77.6
1.7
98.3
6.3
92.5
CAD
3.1
65.7
13.6
42.9
2.0
89.4
3.7
94.7
11.3
80.0
↑Context
Gradient
59.3
19.6
33.3
25.0
21.3
54.6
28.5
71.3
40.2
52.1
PH3 (Ours)
18.6
76.1
56.9
33.3
10.9
80.4
16.9
76.7
26.9
67.9
+ Prompt
1.9
97.6
17.7
75.8
3.8
95.5
2.2
97.7
2.4
97.0
Base
60.6
39.4
74.6
25.0
1.6
98.4
26.2
73.7
5.6
93.3
Prompt
0.4
99.6
78.5
21.0
0.5
99.5
22.3
77.6
10.7
89.3
↑Memory
Gradient
77.7
10.0
89.9
9.5
26.1
68.3
48.1
51.4
30.7
49.0
PH3 (Ours)
86.3
12.5
91.5
8.1
71.2
11.6
47.7
51.6
11.9
45.4
LLaMA2-13B
Prompt
0.0
100.0
70.9
28.7
0.0
100.0
7.1
92.7
3.6
96.4
CAD
6.2
91.0
1.1
98.8
0.0
100.0
0.5
99.5
0.0
99.6
↑Context
Gradient
46.3
33.9
74.1
25.4
16.5
83.3
20.2
79.5
3.6
96.4
PH3 (Ours)
6.2
92.1
24.1
75.9
1.3
98.7
5.6
94.4
0.4
99.6
+ Prompt
0.0
100.0
31.0
68.0
0.0
100.0
0.0
100.0
0.0
100.0
Table 2: Experimental results of OPT-1.3B, OPT-2.7B, Pythia-6.9B, Pythia-12B and LLaMA2-7B on five datasets.
Bolds denote the best results.
Method
GPT-2 XL
GPT-J
OPT-2.7B
Base
45.6
54.8
51.4
Prompt
47.6
57.4
54.1
CAD
44.5
55.0
50.2
Gradient
45.3
55.0
50.8
PH3 (k = 1)
47.1
57.5
53.5
PH3 (k = 3)
52.2
58.6
55.4
+ Prompt
54.0
59.6
56.7
PH3 (k = 5)
49.4
56.3
54.0
Table 3: Experimental results (Recall) of GPT-2 XL,
GPT-J and OPT-2.7B on the NQ dataset. Bolds denote
the best results.
Model
#Layer L
#Head M
GPT-2 XL
48
25
GPT-J
28
16
OPT-1.3B
24
32
OPT-2.7B
32
32
Pythia-6.9B
32
32
Pythia-12B
36
40
LLaMA2-7B
32
32
LLaMA2-13B
40
40
Table 4: Model details.