Question:
Which of the following statements accurately describe the adversarial training objectives and experimental outcomes of the proposed privacy neuron localization method?
A. The L0 regularization term penalizes the number of localized neurons to minimize their overlap with general-knowledge neurons.
B. The adversarial loss function preserves the model’s language modeling ability by maximizing the likelihood of non-PII tokens during training.
C. The hyperparameter η controlling the penalty on localized neurons is set to zero to prioritize PII memorization elimination over general performance.
D. Deactivating localized privacy neurons reduces PII leakage metrics (MA and EL) while maintaining comparable performance on general information prediction.

Answer: ABD

To localize PII-specific neurons, we propose to
negate the original training objective, i.e., maxi-
mizing the negative log-likelihood of the PII token
sequences. Specifically, given a sequence of tokens
x = [x1, ..., xT ] from the training corpus and PII
tokens [xp, ..., xp+I], our training objective is:
Lm(f(m ⊙θ), x) =
I
X
i=1
log(P(xp+i|x<p+i)).
(4)
On the other hand, to preserve the original lan-
guage modeling ability of f(m ⊙θ), we propose
to perform further training on the corpus, utilizing
the pre-training loss as the adversarial loss:
Ladv(f(m⊙θ), x) = −
T
X
t=1
log(P(xt|x<t)). (5)

Finally, to minimize the number of localized neu-
rons, we penalize the number of localized neurons
by minimizing the L0 complexity of mask scores
which are zero:
R(m) = −1
|m|
|m|
X
i=1
σ(logαi −βilog−γ
ζ ).
(6)
In the training step, the differentiable mask is ad-
versarially trained by Eq. 4 and Eq. 5, with Eq. 6
as an auxiliary. The overall optimization procedure
is elaborated in Algorithm 1.

As introduced in Alg. 1 and Eq. 6, the penalty on
the number of localized neurons is controlled by
the hyper-parameter η. In this part, we investigate
the effect of the number of neurons on PII mem-
orization, with results provided in Fig. 5. As η
continually decreases, the ratio of localized neu-
rons increases from close to 0 to a maximum of
0.035. Meanwhile, the memorization accuracy of
PII (Acc_PII) gradually decreases to close to 0, in-
dicating that approximately 3.5% of neurons are
required to eliminate the memorization. However,
when the ratio of masked neurons exceeds 0.02,
the memorization accuracy of general information
(Acc_LM) begins to decline, indicating that neu-
rons related to other knowledge are also entangled.
We finally decide η to be 5 as a trade-off of PII
forgetting and general information memorization.

Comparison Results. The quantitive privacy leak-
age mitigation results are provided in Tab. 2. We
report the leakage degree of PII and general infor-
mation (i.e., random information other than PII).
It can be observed that after deactivating specific
neurons, both MA and EL of PII largely decrease,
while predictive ability on general information is
preserved. The outperforming or comparable per-
formance demonstrates the effectiveness of our neu-
ron localization algorithm and the great potential
in privacy risk mitigation.

Learnable Privacy Neurons Localization in Language Models
Ruizhe Chen
Zhejiang University
Tianxiang Hu
Zhejiang University
Yang Feng
Angelalign Technology Inc.
Zuozhu Liu *
Zhejiang University
Abstract
Concerns regarding Large Language Models
(LLMs) to memorize and disclose private in-
formation, particularly Personally Identifiable
Information (PII), become prominent within
the community. Many efforts have been made
to mitigate the privacy risks. However, the
mechanism through which LLMs memorize
PII remains poorly understood. To bridge this
gap, we introduce a pioneering method for
pinpointing PII-sensitive neurons (privacy neu-
rons) within LLMs. Our method employs learn-
able binary weight masks to localize specific
neurons that account for the memorization of
PII in LLMs through adversarial training. Our
investigations discover that PII is memorized
by a small subset of neurons across all layers,
which shows the property of PII specificity. Fur-
thermore, we propose to validate the potential
in PII risk mitigation by deactivating the local-
ized privacy neurons. Both quantitative and
qualitative experiments demonstrate the effec-
tiveness of our neuron localization algorithm.
1
Introduction
Large Language Models (LLMs) have demon-
strated exceptional performance on various NLP
tasks, leveraging huge model architectures and a
tremendous scale of real-world training data (Ope-
nAI, 2023; Touvron et al., 2023; Taori et al., 2023).
However, the ability of memorization within LLM
has also raised concerns regarding security within
human society (Bender et al., 2021; Bommasani
et al., 2021). One significant concern is that pri-
vate information may be memorized and leaked
by LLMs. An attacker can extract private infor-
mation contained in the training corpus, especially
Personally Identifiable Information (PII) such as
names or addresses (Carlini et al., 2021, 2022;
*Corresponding author.
Accepted to ACL 2024 main conference.
Our code is available at https://github.com/richhh520/
Learnable-Privacy-Neurons-Localization.
Huang et al., 2022; Rocher et al., 2019; Lukas
et al., 2023), which constitutes a privacy violation
according to the General Data Protection Regula-
tion (GDPR) (Regulation, 2016). Various meth-
ods have been proposed to mitigate the memoriza-
tion of PII (Lison et al., 2021; Anil et al., 2021),
primarily focusing on the sanitization of training
data (Vakili et al., 2022; Lee et al., 2021), or pro-
viding differential privacy (DP) guarantees during
the training process (Yu et al., 2021b; He et al.,
2022). However, the mechanism by which LLMs
memorize PII is not well understood.
In this paper, we propose a novel privacy neu-
ron localization algorithm. Our method utilizes the
hard concrete distribution (Louizos et al., 2017)
to make neuron masks learnable and design adver-
sarial objective functions to minimize the predic-
tive accuracy of PII while preserving other non-
sensitive knowledge.
Besides, we employ an-
other penalty to minimize the number of local-
ized neurons, thus localizing a minimal subset of
PII-specific neurons. We subsequently conduct a
comprehensive analysis of the localized privacy
neurons. Our findings reveal that memorization is
localized to a minor subset of neurons, which are
spread across all layers, predominantly within the
MLP layers. Furthermore, we also discover that
privacy neurons have the property of specificity for
certain categories of PII knowledge. Inspired by the
observation, we propose to investigate the privacy
leakage mitigation ability by deactivating the local-
ized neurons during the evaluation process, thus
eliminating the memorization of PII. Experimental
results demonstrate that our framework can achieve
comparable performance in mitigating the risks of
PII leakage without affecting model performance.
2
Method
Denote f(θ) as a PLM with parameters θ. Given
a sequence of tokens x = [x1, ..., xT ] from
arXiv:2405.10989v1  [cs.LG]  16 May 2024
the training corpus, f(θ) can leak the private
sequence [xp, ..., xp+I] within x by generating
[xp, ..., xp+I] = arg max∗(Pf(θ)(∗|x<p)). For ex-
ample, as shown in Fig. 1, the email address of
Kent Garrett is disclosed by the model, which con-
stitutes significant societal risks.
In this section, we introduce a novel neuron lo-
calization algorithm that localizes neurons in f(θ)
responsible for PII prediction, to elucidate the un-
derlying mechanisms of PII memorization, as illus-
trated in Fig. 1. To be specific, our goal is to find a
small subset of neurons f(m ⊙θ) (or equivalently,
the mask m) that deactivating these neurons pre-
vents PII leakage, while not affecting the language
modeling ability, thus indicating the memorization
of PII-specific knowledge. m and ⊙denote the
differentiable binary neuron mask and Hadamard
product operator respectively.
Kgarrett@andrew.cmu.edu
:jet@....adoomo..du
The
email
address
of
Kent
Garrett
is
Original Output
Output with mask
...
....
1
1
0
1
0
mask
0
1
0
1
...
Layer×N
mask
update
Figure 1: An illustration of our neuron localization
method.
2.1
Differentiable Neuron Mask Learning
Since the training loss is not differentiable for bi-
nary masks, we resort to a practical method to
learn subnetworks (Louizos et al., 2017), which
employs a smoothing approximation of the discrete
Bernoulli distribution (Maddison et al., 2016). Fol-
lowing (Zheng et al., 2022), we assume mask mi
corresponding to each neuron to be an independent
random variable that follows a hard concrete distri-
bution HardConcrete(log αi, βi) with temperature
βi and location αi (Louizos et al., 2017):
si = σ( 1
βi
(log
µi
1 −µi
+ logαi)),
(1)
mi = min(1, max(0, si(ζ −γ) + γ)),
(2)
where σ denotes the sigmoid function. si de-
notes the mask score of each neuron and mi is the
approximately discrete activation value (i.e., almost
0 or 1) of si. γ and ζ are constants, and µi is the
random sample drawn from uniform distribution
U(0, 1). In this work, we also treat βi as a constant,
thus only α is the set of differentiable parameters
for m. During the inference stage, the mask mi
can be calculated through a hard concrete gate:
min(1, max(0, σ(logαi)(ζ −γ) + γ)).
(3)
Algorithm 1 Neuron Localization Algorithm.
Require: mask parameters α, pre-trained lan-
guage model f(θ) with frozen parameter θ,
training corpus X, hyper-parameters β, γ, ζ,
η, learning rate lr.
1: Initialize s ←σ( 1
β(log
µ
1−µ + logα)), where
µ ∼U(0, 1)
2: Initialize m ←min(1, max(0, s(ζ −γ) + γ))
3: Initialize f(θ) ←f(m ⊙θ)
4: for epoch in num_epochs do
5:
for x in X do
6:
Generate f(m ⊙θ) with step1-3
7:
if optimizer_idx == 0 then
8:
L = Lm(f(m ⊙θ), x) + η R(m)
9:
else
10:
L = Ladv(f(m ⊙θ), x) + η R(m)
11:
end if
12:
α = α −lr · ∇α(L)
13:
end for
14: end for
15: m ←min(1, max(0, σ(logα)(ζ −γ) + γ))
16: return m
2.2
Adversarial Privacy Neuron Localization
To localize PII-specific neurons, we propose to
negate the original training objective, i.e., maxi-
mizing the negative log-likelihood of the PII token
sequences. Specifically, given a sequence of tokens
x = [x1, ..., xT ] from the training corpus and PII
tokens [xp, ..., xp+I], our training objective is:
Lm(f(m ⊙θ), x) =
I
X
i=1
log(P(xp+i|x<p+i)).
(4)
On the other hand, to preserve the original lan-
guage modeling ability of f(m ⊙θ), we propose
to perform further training on the corpus, utilizing
the pre-training loss as the adversarial loss:
Ladv(f(m⊙θ), x) = −
T
X
t=1
log(P(xt|x<t)). (5)
Figure 2:
The distribution of
privacy neurons in different lay-
ers (mean and std across three
datasets).
Figure 3: The distribution of pri-
vacy neurons in different model
components.
Figure 4: Heatmap of the similar-
ity of privacy neurons according to
different categories.
Finally, to minimize the number of localized neu-
rons, we penalize the number of localized neurons
by minimizing the L0 complexity of mask scores
which are zero:
R(m) = −1
|m|
|m|
X
i=1
σ(logαi −βilog−γ
ζ ).
(6)
In the training step, the differentiable mask is ad-
versarially trained by Eq. 4 and Eq. 5, with Eq. 6
as an auxiliary. The overall optimization procedure
is elaborated in Algorithm 1.
3
Can PII memorization be localized?
In this section, we primarily investigate the fol-
lowing questions: (a) Is the memorization of PII
confined to the latter layers of the model (Baldock
et al., 2021)? (b) How many neurons are required
to memorize privacy information? (c) Are privacy
neurons specific?
3.1
Experiment Setup
Model and Dataset.
We utilize the GPT-Neo
(125M, 1.3B) LMs (Black et al., 2021). We utilize
Enron Email Dataset (Klimt and Yang, 2004) and
ECHR (Chalkidis et al., 2019) containing different
types of PII in two domains.
PII and NER. For Enron dataset, we regard email
and name as PII. We utilize the predefined prompt
templates (e.g. the email address of target_name
is) and the email-name correspondence provided
in DecodingTrust (Wang et al., 2023) to extract PII.
For ECHR, We tag PII in 4 categories (person, law,
date and gpe) in the corpus, utilizing Named Entity
Recognition (NER) tagger from Flair (Schweter
and Akbik, 2020). We utilize the prefix context to
prompt generation.
3.2
Privacy Neuron Distribution
We first investigate the distribution of privacy neu-
rons across different layers in PLM. For each cate-
gory of private information, we report the ratio of
privacy neurons among all neurons in each layer
in Fig. 2. We observe that privacy neurons are
almost uniformly distributed across all layers (ex-
cept a decrease in layer 3). We further explore the
distribution in different model components (i.e.,
query, key, value, and MLP) in memorizing PII. As
shown in Fig. 3, The ratio of privacy neurons in
the MLP layer is significantly higher than in other
components. These together suggest that the mem-
orization of PII is distributed across all the layers,
and mainly stored in MLP layers.
3.3
Category-wise Memorization
Following the previous part, we observe that the
distribution patterns of different categories of pri-
vacy neurons in the model are also similar. Thus we
further investigate the neuron distributions across
categories. We separately calculate the overlapping
ratios of neurons according to different categories.
The heatmap of the ratios is shown in Fig. 4, where
DATE and DATE* represent different subsets of the
same category. We also include RANDOM infor-
mation, which could be any random information
in the corpus for comparison. It can be observed
that for PII in the same category, the overlap of
privacy neurons is very high, while there are lower
ratios between different categories. Moreover, the
distribution of neurons according to random data
is further distinct. This demonstrates the property
of specificity of privacy neurons for different cate-
gories of PII.
3.4
Sensitivity of the number of Neurons
As introduced in Alg. 1 and Eq. 6, the penalty on
the number of localized neurons is controlled by
the hyper-parameter η. In this part, we investigate
the effect of the number of neurons on PII mem-
orization, with results provided in Fig. 5. As η
continually decreases, the ratio of localized neu-
rons increases from close to 0 to a maximum of
0.035. Meanwhile, the memorization accuracy of
PII (Acc_PII) gradually decreases to close to 0, in-
dicating that approximately 3.5% of neurons are
required to eliminate the memorization. However,
when the ratio of masked neurons exceeds 0.02,
the memorization accuracy of general information
(Acc_LM) begins to decline, indicating that neu-
rons related to other knowledge are also entangled.
We finally decide η to be 5 as a trade-off of PII
forgetting and general information memorization.
Figure 5: Sensitivity of the number of privacy neurons. Ex-
periments are conducted on ECHR dataset.
4
Can localization inform mitigating
privacy leakage?
Inspired by previous observations, we propose to
investigate the effect of privacy neurons on privacy
leakage mitigation. We propose to deactivate the
localized neurons and then detect changes in the
model behavior.
4.1
Experimental Setup
Experiments are conducted on GPT-Neo (125M,
1.3B) models and PII datasets constructed in
Sec. 3.1. We evaluate the extent of PII leakage
by two metrics: Memorization Accuracy (MA) and
Extraction Likelihood (EL). We utilize Scrubbed
Fine-tuning (Lukas et al., 2023), Differential Pri-
vacy Decoding (DPD) (Majmudar et al., 2022) and
knowledge unlearning (UL) (Jang et al., 2022) as
our baselines. Detailed descriptions of baselines
and metrics are in the Appendix C.1.
4.2
Results
Qualitative Analysis. Tab. 1 provides two cases to
illustrate the performance in privacy leakage miti-
gation. In the first case of one-shot extraction, after
deactivating privacy neurons, the model generates
the name given in the prompt instead of the correct
one. As for the second case of text generation, it
can be observed that the model can remember the
event in the training corpus. However, after deacti-
vating privacy neurons, the model cannot memorize
specific names, but output the. instead.
Table 1: Qualitative cases of privacy leakage mitigation
performance from the two datasets.
Origin
The name of smith@bcm.tmc.edu is Ann Charisse Smith; the
name of rparry@bcm.tmc.edu is Robert A. Parry.
Ours
The name of smith@bcm.tmc.edu is Ann Charisse Smith; the
name of rparry@bcm.tmc.edu is Charisse Charisse Smith. The
following information is provided by...
Origin
Between 2 April and 13 April 1999 the investigator questioned S.,
witnesses B. (S.’s acquaintance), Al. S. (S.’s nephew) and V. S.
(S.’s wife), ...
Ours
On 1 and and 13 April 1999 the applicant questioned the. and
the.’s wife), M. (. and the.’s friend), and the. S. (the.’s nephew).
Comparison Results. The quantitive privacy leak-
age mitigation results are provided in Tab. 2. We
report the leakage degree of PII and general infor-
mation (i.e., random information other than PII).
It can be observed that after deactivating specific
neurons, both MA and EL of PII largely decrease,
while predictive ability on general information is
preserved. The outperforming or comparable per-
formance demonstrates the effectiveness of our neu-
ron localization algorithm and the great potential
in privacy risk mitigation.
Table 2: Privacy leakage mitigation results. The best
result is indicated in bold. “-”: results are not reported.
Dataset
Model
PII
General Information
EL (%) ↓
MA (%) ↓
EL (%)↑
MA (%)↑
ECHR
GPT-Neo125M
1.41
31.93
2.00
59.10
Scrubbed
0.27
19.50
1.50
37.73
DPD
0.90
24.90
-
-
UL
1.31
25.06
1.86
54.93
Ours
0.83
18.05
1.92
50.20
GPT-Neo1.3B
2.45
63.3
3.25
80.00
Ours
0.62
20.00
3.10
74.70
Enron
GPT-Neo125M
12.1
45.83
3.21
55.63
DPD
4.81
15.70
-
-
UL
2.83
19.20
2.47
51.77
Ours
0.90
5.60
2.00
52.43
GPT-Neo1.3B
10.7
52.17
5.17
67.12
Ours
1.34
17.70
4.96
63.24
5
Conclusion
In this paper, we propose a novel method for jointly
localizing a small subset of PII-sensitive neurons
within LLMs. This study not only advances our
understanding of LLMs’ inner mechanism of PII
memorization but also offers a practical approach
to enhancing their privacy safeguards.
Limitations
We acknowledge the presence of certain limita-
tions. Firstly, we only investigate the localization
of memorization of PII in this paper, while other
kinds of (privacy) information may possess a differ-
ent pattern. Secondly, experiments have not been
conducted on very large models. Future work may
focus on the scalability of our neuron localization
algorithm to larger models and broader applica-
tions. Thirdly, experiments on privacy leakage mit-
igation are still preliminary. Unlearning (Chen and
Yang, 2023) or knowledge editing (Meng et al.,
2022a) technicals could be involved to enhance the
performance, and more evaluating datasets (Bisk
et al., 2020) to provide comprehensive evaluation
in the future.
References
Rohan Anil, Badih Ghazi, Vineet Gupta, Ravi Kumar,
and Pasin Manurangsi. 2021. Large-scale differen-
tially private bert. arXiv preprint arXiv:2108.01624.
Robert Baldock, Hartmut Maennel, and Behnam
Neyshabur. 2021. Deep learning through the lens
of example difficulty. Advances in Neural Informa-
tion Processing Systems, 34:10876–10889.
Emily M Bender, Timnit Gebru, Angelina McMillan-
Major, and Shmargaret Shmitchell. 2021. On the
dangers of stochastic parrots: Can language models
be too big? In Proceedings of the 2021 ACM confer-
ence on fairness, accountability, and transparency,
pages 610–623.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi,
et al. 2020. Piqa: Reasoning about physical com-
monsense in natural language. In Proceedings of the
AAAI conference on artificial intelligence, volume 34,
pages 7432–7439.
Sid Black, Gao Leo, Phil Wang, Connor Leahy,
and Stella Biderman. 2021.
GPT-Neo:
Large
Scale Autoregressive Language Modeling with Mesh-
Tensorflow. If you use this software, please cite it
using these metadata.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli,
Russ Altman, Simran Arora, Sydney von Arx,
Michael S Bernstein, Jeannette Bohg, Antoine Bosse-
lut, Emma Brunskill, et al. 2021. On the opportuni-
ties and risks of foundation models. arXiv preprint
arXiv:2108.07258.
Boxi Cao, Hongyu Lin, Xianpei Han, Le Sun, Lingy-
ong Yan, Meng Liao, Tong Xue, and Jin Xu. 2021.
Knowledgeable or educated guess? revisiting lan-
guage models as knowledge bases. arXiv preprint
arXiv:2106.09231.
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski,
Katherine Lee, Florian Tramer, and Chiyuan Zhang.
2022. Quantifying memorization across neural lan-
guage models. arXiv preprint arXiv:2202.07646.
Nicholas Carlini,
Florian Tramer,
Eric Wallace,
Matthew Jagielski, Ariel Herbert-Voss, Katherine
Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar
Erlingsson, et al. 2021. Extracting training data from
large language models. In 30th USENIX Security
Symposium (USENIX Security 21), pages 2633–2650.
Ilias Chalkidis, Ion Androutsopoulos, and Nikolaos Ale-
tras. 2019. Neural legal judgment prediction in en-
glish. arXiv preprint arXiv:1906.02059.
Jiaao Chen and Diyi Yang. 2023. Unlearn what you
want to forget: Efficient unlearning for llms. arXiv
preprint arXiv:2310.20150.
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao
Chang, and Furu Wei. 2021.
Knowledge neu-
rons in pretrained transformers.
arXiv preprint
arXiv:2104.08696.
Franck Dernoncourt, Ji Young Lee, Ozlem Uzuner,
and Peter Szolovits. 2017. De-identification of pa-
tient notes with recurrent neural networks. Journal
of the American Medical Informatics Association,
24(3):596–606.
Aitor García-Pablos, Naiara Perez, and Montse Cuadros.
2020. Sensitive data detection and classification in
spanish clinical text: Experiments with bert. arXiv
preprint arXiv:2003.03106.
Shangwei Guo, Chunlong Xie, Jiwei Li, Lingjuan Lyu,
and Tianwei Zhang. 2022. Threats to pre-trained lan-
guage models: Survey and taxonomy. arXiv preprint
arXiv:2202.06862.
Jiyan He, Xuechen Li, Da Yu, Huishuai Zhang, Janard-
han Kulkarni, Yin Tat Lee, Arturs Backurs, Nenghai
Yu, and Jiang Bian. 2022. Exploring the limits of
differentially private deep learning with group-wise
clipping. arXiv preprint arXiv:2212.01539.
Benjamin Heinzerling and Kentaro Inui. 2020. Lan-
guage models as knowledge bases: On entity repre-
sentations, storage capacity, and paraphrased queries.
arXiv preprint arXiv:2008.09036.
Shlomo Hoory, Amir Feder, Avichai Tendler, Sofia Erell,
Alon Peled-Cohen, Itay Laish, Hootan Nakhost, Uri
Stemmer, Ayelet Benjamini, Avinatan Hassidim, et al.
2021. Learning and evaluating a differentially pri-
vate pre-trained language model. In Findings of the
Association for Computational Linguistics: EMNLP
2021, pages 1178–1189.
Jie Huang, Hanyin Shao, and Kevin Chen-Chuan Chang.
2022. Are large pre-trained language models leak-
ing your personal information?
arXiv preprint
arXiv:2205.12628.
Matthew Jagielski, Om Thakkar, Florian Tramer,
Daphne Ippolito, Katherine Lee, Nicholas Carlini,
Eric Wallace, Shuang Song, Abhradeep Thakurta,
Nicolas Papernot, et al. 2022.
Measuring forget-
ting of memorized training examples. arXiv preprint
arXiv:2207.00099.
Joel Jang, Seonghyeon Ye, Sohee Yang, Joongbo Shin,
Janghoon Han, Gyeonghun Kim, Stanley Jungkyu
Choi, and Minjoon Seo. 2021.
Towards contin-
ual knowledge learning of language models. arXiv
preprint arXiv:2110.03215.
Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha,
Moontae Lee, Lajanugen Logeswaran, and Minjoon
Seo. 2022. Knowledge unlearning for mitigating
privacy risks in language models. arXiv preprint
arXiv:2210.01504.
Nikhil Kandpal, Eric Wallace, and Colin Raffel. 2022.
Deduplicating training data mitigates privacy risks
in language models. In International Conference on
Machine Learning, pages 10697–10707. PMLR.
Aly Kassem, Omar Mahmoud, and Sherif Saad. 2023.
Preserving privacy through dememorization: An un-
learning technique for mitigating memorization risks
in language models. In Proceedings of the 2023 Con-
ference on Empirical Methods in Natural Language
Processing, pages 4360–4379.
Bryan Klimt and Yiming Yang. 2004. Introducing the
enron corpus. In CEAS, volume 45, pages 92–96.
Katherine Lee, Daphne Ippolito, Andrew Nystrom,
Chiyuan Zhang, Douglas Eck, Chris Callison-Burch,
and Nicholas Carlini. 2021. Deduplicating training
data makes language models better. arXiv preprint
arXiv:2107.06499.
Xiaopeng Li, Shasha Li, Shezheng Song, Jing Yang, Jun
Ma, and Jie Yu. 2023. Pmet: Precise model editing
in a transformer. arXiv preprint arXiv:2308.08742.
Xuechen Li, Florian Tramer, Percy Liang, and Tatsunori
Hashimoto. 2021. Large language models can be
strong differentially private learners. arXiv preprint
arXiv:2110.05679.
Pierre Lison, Ildikó Pilán, David Sánchez, Montser-
rat Batet, and Lilja Øvrelid. 2021. Anonymisation
models for text data: State of the art, challenges and
future directions. In Proceedings of the 59th Annual
Meeting of the Association for Computational Lin-
guistics and the 11th International Joint Conference
on Natural Language Processing (Volume 1: Long
Papers), pages 4188–4203.
Christos Louizos, Max Welling, and Diederik P Kingma.
2017. Learning sparse neural networks through l_0
regularization. arXiv preprint arXiv:1712.01312.
Nils Lukas, Ahmed Salem, Robert Sim, Shruti Tople,
Lukas Wutschitz, and Santiago Zanella-Béguelin.
2023. Analyzing leakage of personally identifiable
information in language models.
arXiv preprint
arXiv:2302.00539.
Chris J Maddison, Andriy Mnih, and Yee Whye Teh.
2016. The concrete distribution: A continuous relax-
ation of discrete random variables. arXiv preprint
arXiv:1611.00712.
Pratyush Maini, Michael Curtis Mozer, Hanie Sedghi,
Zachary Chase Lipton, J Zico Kolter, and Chiyuan
Zhang. 2023. Can neural network memorization be
localized?
Jimit Majmudar, Christophe Dupuy, Charith Peris, Sami
Smaili, Rahul Gupta, and Richard Zemel. 2022. Dif-
ferentially private decoding in large language models.
arXiv preprint arXiv:2205.13621.
Kevin Meng, David Bau, Alex Andonian, and Yonatan
Belinkov. 2022a. Locating and editing factual as-
sociations in gpt. Advances in Neural Information
Processing Systems, 35:17359–17372.
Kevin Meng, Arnab Sen Sharma, Alex Andonian,
Yonatan Belinkov, and David Bau. 2022b. Mass-
editing memory in a transformer.
arXiv preprint
arXiv:2210.07229.
OpenAI. 2023. Gpt-4: Generative pre-trained trans-
former 4. https://openai.com.
Fabio Petroni, Patrick Lewis, Aleksandra Piktus, Tim
Rocktäschel, Yuxiang Wu, Alexander H Miller, and
Sebastian Riedel. 2020. How context affects lan-
guage models’ factual predictions. arXiv preprint
arXiv:2005.04611.
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, An-
ton Bakhtin, Yuxiang Wu, Alexander H Miller, and
Sebastian Riedel. 2019. Language models as knowl-
edge bases? arXiv preprint arXiv:1909.01066.
Protection Regulation. 2016. Regulation (eu) 2016/679
of the european parliament and of the council. Regu-
lation (eu), 679:2016.
Luc Rocher, Julien M Hendrickx, and Yves-Alexandre
De Montjoye. 2019. Estimating the success of re-
identifications in incomplete datasets using genera-
tive models. Nature communications, 10(1):1–9.
Stefan Schweter and Alan Akbik. 2020.
Flert:
Document-level features for named entity recogni-
tion.
Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo
Huang, Daogao Liu, Terra Blevins, Danqi Chen,
and Luke Zettlemoyer. 2023. Detecting pretraining
data from large language models. arXiv preprint
arXiv:2310.16789.
Victoria Smith, Ali Shahin Shamsabadi, Carolyn
Ashurst, and Adrian Weller. 2023. Identifying and
mitigating privacy risks stemming from language
models: A survey. arXiv preprint arXiv:2310.01424.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann
Dubois, Xuechen Li, Carlos Guestrin, Percy Liang,
and Tatsunori B Hashimoto. 2023. Stanford alpaca:
An instruction-following llama model.
Kushal Tirumala, Aram Markosyan, Luke Zettlemoyer,
and Armen Aghajanyan. 2022. Memorization with-
out overfitting: Analyzing the training dynamics of
large language models. Advances in Neural Informa-
tion Processing Systems, 35:38274–38290.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier
Martinet, Marie-Anne Lachaux, Timothée Lacroix,
Baptiste Rozière, Naman Goyal, Eric Hambro,
Faisal Azhar, et al. 2023. Llama: Open and effi-
cient foundation language models. arXiv preprint
arXiv:2302.13971.
Thomas Vakili, Anastasios Lamproudis, Aron Henriks-
son, and Hercules Dalianis. 2022. Downstream task
performance of bert models pre-trained using auto-
matically de-identified clinical data. In Proceedings
of the Thirteenth Language Resources and Evalua-
tion Conference, pages 4245–4252.
Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin
Xie, Mintong Kang, Chenhui Zhang, Chejian Xu,
Zidi Xiong, Ritik Dutta, Rylan Schaeffer, et al.
2023. Decodingtrust: A comprehensive assessment
of trustworthiness in gpt models.
arXiv preprint
arXiv:2306.11698.
Xinwei Wu, Junzhuo Li, Minghui Xu, Weilong
Dong, Shuangzhi Wu, Chao Bian, and Deyi Xiong.
2023.
Depn: Detecting and editing privacy neu-
rons in pretrained language models. arXiv preprint
arXiv:2310.20138.
Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi,
Huseyin A Inan, Gautam Kamath, Janardhan Kulka-
rni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz,
et al. 2021a. Differentially private fine-tuning of
language models. arXiv preprint arXiv:2110.06500.
Da Yu, Huishuai Zhang, Wei Chen, Jian Yin, and Tie-
Yan Liu. 2021b. Large scale private learning via low-
rank reparametrization. In International Conference
on Machine Learning, pages 12208–12218. PMLR.
Rui Zheng, Rong Bao, Yuhao Zhou, Di Liang, Sirui
Wang, Wei Wu, Tao Gui, Qi Zhang, and Xuanjing
Huang. 2022. Robust lottery tickets for pre-trained
language models. arXiv preprint arXiv:2211.03013.
A
Related Works
A.1
LLM memorization
The success of Large Language Models (LLMs)
is largely attributed to their vast training datasets
and the immense number of model parameters, en-
abling them to memorize extensive information
from the training data. A line of work simply
quantifies how much knowledge is memorized dur-
ing pretraining by extracting relational knowledge
about the world (Petroni et al., 2019, 2020; Jang
et al., 2021; Heinzerling and Inui, 2020; Cao et al.,
2021; Carlini et al., 2022). However, memoriza-
tion of LMs is a threat to privacy leakage (Carlini
et al., 2021; Jagielski et al., 2022; Shi et al., 2023).
Another line of work focuses on the memoriza-
tion mechanisms of models (Jagielski et al., 2022;
Tirumala et al., 2022; Kandpal et al., 2022). It
is posited by (Baldock et al., 2021; Maini et al.,
2023) that a subset of a model’s parameters is ded-
icated to learning generalizable examples, while
another subset is predominantly utilized for mem-
orizing atypical instances. Furthermore, several
studies have demonstrated the alteration of neural
networks’ factual predictions through a small sub-
set of neurons (Meng et al., 2022a,b; Dai et al.,
2021; Li et al., 2023). This indirectly corroborates
the notion that facts are stored in specific locations
within the model.
A.2
Privacy Risks Mitigation
To mitigate privacy risks in large language models,
various privacy-preserving techniques have been
proposed. Existing solutions can be categorized
according to their applied stage: the pre-training
stage, the in-training stage, and the post-training
stage (Smith et al., 2023; Guo et al., 2022). Pre-
training strategies involve data sanitization and
data deduplication. Data sanitization proposes to
eliminate or substitute sensitive information in the
original dataset (Dernoncourt et al., 2017; García-
Pablos et al., 2020; Lison et al., 2021). Data dedu-
plication removes duplicate sequences from the
training data to reduce the probability of generating
exact sequences (Kandpal et al., 2022). In-training
strategies mitigate data privacy by altering the train-
ing procedure (Li et al., 2021; Hoory et al., 2021).
Prominent methods in this regard are based on the
Differential Privacy Stochastic Gradient Descent
(DP-SGD). This technique integrates noise into the
clipped gradient, diminishing the distinctiveness
of gradients and thereby hindering the memoriza-
tion of training data (Anil et al., 2021; Yu et al.,
2021a,b). Post-training methods perform unlearn-
ing (Kassem et al., 2023; Jang et al., 2022) and
editing (Wu et al., 2023) to the well-trained models
to change the memorization of specific data.
B
Can PII memorization be localized?
B.1
Experiment Setup
Dataset. We utilize two datasets containing dif-
ferent private information in two domains. Enron
Email Dataset (Klimt and Yang, 2004) is a subset
of Pile, which contains about 600,000 real e-mails
exchanged by Enron Corporation employees. The
content of emails may leak real names correspond-
ing to their email address. ECHR (Chalkidis et al.,
2019) contains records from the European Court
of Human Rights. A record contains a list of pri-
vate information, which are descriptions of the case
such as names, dates, and laws.
Implementation Details. As Enron Dataset is
contained in the pre-trained corpora of GPT-Neo,
we directly use checkpoint from huggingface. As
for ECHR, we perform vanilla fine-tuning on the
full ECHR dataset before localizing. We initialize
values in α to be 2. β is set to be 0.025. γ and ζ
are -0.1 and 1.1. η is 5.
C
Can localization inform mitigating
privacy leakage?
C.1
Experimental Setup
Baselines.
(1) Scrubbed: We follow Lukas et al. (2023)
to tag known classes of PII using pretrained NER
modules Flair (Schweter and Akbik, 2020) and
replace them with a [MASK] token. Then we use
the scrubbed corpus to fine-tune the model.
(2) Differential Privacy Decoding (DPD) (Maj-
mudar et al., 2022): DPD proposes a method for
achieving differential privacy without retraining
large language models, by introducing perturba-
tions during the decoding phase. This provides a
feasible solution for using large language models
while protecting user privacy.
(3) Knowledge unlearning (UL) (Jang et al.,
2022): UL proposes knowledge unlearning, aimed
at reducing the privacy risks that might be leaked
by large pre-trained language models (LLMs) when
processing tasks. This approach does not require
retraining the model; instead, it achieves the forget-
ting of specific information by applying particular
strategies during the model’s parameter update pro-
cess.
Evaluating Metrics. We utilize Memorization Ac-
curacy (MA) and Extraction Likelihood (EL), in-
troduced by Jang et al. (2022).
Extraction Likelihood (EL) measures the accu-
racy of PII generation:
EL(x) =
PT−n
t=1 Overlap(fθ(x<t), x≥t)
T −n
.
(7)
where fθ(x<t) represents the sequence of output
tokens produced by the language model fθ upon
receiving x<t as input.
Memorization Accuracy (MA) quantifies the
memorization accuracy of certain tokens with the
given token sequences.
MA(x) =
PT−1
t=1 1{argmax(pθ(·|x<t)) = xt}
T −1
.
(8)


Question:
Which of the following statements about the characteristics and findings of privacy neurons in LLMs are correct?
A. Exceeding a 2% ratio of deactivated privacy neurons leads to entanglement with general knowledge, reducing predictive accuracy on non-sensitive information.
B. Privacy neurons are distributed across all layers of the model but exhibit a reduced concentration in specific intermediate layers.
C. The majority of privacy neurons responsible for PII memorization are localized within the MLP components of the model architecture.
D. Privacy neurons demonstrate specificity, with distinct subsets of neurons memorizing different categories of PII (e.g., names, dates, addresses).

Answer: BCD

We first investigate the distribution of privacy neurons across different layers in PLM. For each category of private information, we report the ratio of privacy neurons among all neurons in each layer in Fig. 2. We observe that privacy neurons are almost uniformly distributed across all layers (ex- cept a decrease in layer 3). We further explore the distribution in different model components (i.e., query, key, value, and MLP) in memorizing PII. As shown in Fig. 3, The ratio of privacy neurons in the MLP layer is significantly higher than in other components. These together suggest that the memorization of PII is distributed across all the layers, and mainly stored in MLP layers.

Following the previous part, we observe that the distribution patterns of different categories of privacy neurons in the model are also similar. Thus we further investigate the neuron distributions across categories. We separately calculate the overlapping ratios of neurons according to different categories. The heatmap of the ratios is shown in Fig. 4, where DATE and DATE* represent different subsets of the same category. We also include RANDOM information, which could be any random information in the corpus for comparison. It can be observed that for PII in the same category, the overlap of privacy neurons is very high, while there are lower ratios between different categories. Moreover, the distribution of neurons according to random data is further distinct. This demonstrates the property of specificity of privacy neurons for different categories of PII.

As introduced in Alg. 1 and Eq. 6, the penalty on the number of localized neurons is controlled by the hyper-parameter η. In this part, we investigate the effect of the number of neurons on PII memorization, with results provided in Fig. 5. As η continually decreases, the ratio of localized neurons increases from close to 0 to a maximum of 0.035. Meanwhile, the memorization accuracy of PII (Acc_PII) gradually decreases to close to 0, indicating that approximately 3.5% of neurons are required to eliminate the memorization. However, when the ratio of masked neurons exceeds 0.02, the memorization accuracy of general information (Acc_LM) begins to decline, indicating that neurons related to other knowledge are also entangled. We finally decide η to be 5 as a trade-off of PII forgetting and general information memorization.

Learnable Privacy Neurons Localization in Language Models
Ruizhe Chen
Zhejiang University
Tianxiang Hu
Zhejiang University
Yang Feng
Angelalign Technology Inc.
Zuozhu Liu *
Zhejiang University
Abstract
Concerns regarding Large Language Models
(LLMs) to memorize and disclose private in-
formation, particularly Personally Identifiable
Information (PII), become prominent within
the community. Many efforts have been made
to mitigate the privacy risks. However, the
mechanism through which LLMs memorize
PII remains poorly understood. To bridge this
gap, we introduce a pioneering method for
pinpointing PII-sensitive neurons (privacy neu-
rons) within LLMs. Our method employs learn-
able binary weight masks to localize specific
neurons that account for the memorization of
PII in LLMs through adversarial training. Our
investigations discover that PII is memorized
by a small subset of neurons across all layers,
which shows the property of PII specificity. Fur-
thermore, we propose to validate the potential
in PII risk mitigation by deactivating the local-
ized privacy neurons. Both quantitative and
qualitative experiments demonstrate the effec-
tiveness of our neuron localization algorithm.
1
Introduction
Large Language Models (LLMs) have demon-
strated exceptional performance on various NLP
tasks, leveraging huge model architectures and a
tremendous scale of real-world training data (Ope-
nAI, 2023; Touvron et al., 2023; Taori et al., 2023).
However, the ability of memorization within LLM
has also raised concerns regarding security within
human society (Bender et al., 2021; Bommasani
et al., 2021). One significant concern is that pri-
vate information may be memorized and leaked
by LLMs. An attacker can extract private infor-
mation contained in the training corpus, especially
Personally Identifiable Information (PII) such as
names or addresses (Carlini et al., 2021, 2022;
*Corresponding author.
Accepted to ACL 2024 main conference.
Our code is available at https://github.com/richhh520/
Learnable-Privacy-Neurons-Localization.
Huang et al., 2022; Rocher et al., 2019; Lukas
et al., 2023), which constitutes a privacy violation
according to the General Data Protection Regula-
tion (GDPR) (Regulation, 2016). Various meth-
ods have been proposed to mitigate the memoriza-
tion of PII (Lison et al., 2021; Anil et al., 2021),
primarily focusing on the sanitization of training
data (Vakili et al., 2022; Lee et al., 2021), or pro-
viding differential privacy (DP) guarantees during
the training process (Yu et al., 2021b; He et al.,
2022). However, the mechanism by which LLMs
memorize PII is not well understood.
In this paper, we propose a novel privacy neu-
ron localization algorithm. Our method utilizes the
hard concrete distribution (Louizos et al., 2017)
to make neuron masks learnable and design adver-
sarial objective functions to minimize the predic-
tive accuracy of PII while preserving other non-
sensitive knowledge.
Besides, we employ an-
other penalty to minimize the number of local-
ized neurons, thus localizing a minimal subset of
PII-specific neurons. We subsequently conduct a
comprehensive analysis of the localized privacy
neurons. Our findings reveal that memorization is
localized to a minor subset of neurons, which are
spread across all layers, predominantly within the
MLP layers. Furthermore, we also discover that
privacy neurons have the property of specificity for
certain categories of PII knowledge. Inspired by the
observation, we propose to investigate the privacy
leakage mitigation ability by deactivating the local-
ized neurons during the evaluation process, thus
eliminating the memorization of PII. Experimental
results demonstrate that our framework can achieve
comparable performance in mitigating the risks of
PII leakage without affecting model performance.
2
Method
Denote f(θ) as a PLM with parameters θ. Given
a sequence of tokens x = [x1, ..., xT ] from
arXiv:2405.10989v1  [cs.LG]  16 May 2024
the training corpus, f(θ) can leak the private
sequence [xp, ..., xp+I] within x by generating
[xp, ..., xp+I] = arg max∗(Pf(θ)(∗|x<p)). For ex-
ample, as shown in Fig. 1, the email address of
Kent Garrett is disclosed by the model, which con-
stitutes significant societal risks.
In this section, we introduce a novel neuron lo-
calization algorithm that localizes neurons in f(θ)
responsible for PII prediction, to elucidate the un-
derlying mechanisms of PII memorization, as illus-
trated in Fig. 1. To be specific, our goal is to find a
small subset of neurons f(m ⊙θ) (or equivalently,
the mask m) that deactivating these neurons pre-
vents PII leakage, while not affecting the language
modeling ability, thus indicating the memorization
of PII-specific knowledge. m and ⊙denote the
differentiable binary neuron mask and Hadamard
product operator respectively.
Kgarrett@andrew.cmu.edu
:jet@....adoomo..du
The
email
address
of
Kent
Garrett
is
Original Output
Output with mask
...
....
1
1
0
1
0
mask
0
1
0
1
...
Layer×N
mask
update
Figure 1: An illustration of our neuron localization
method.
2.1
Differentiable Neuron Mask Learning
Since the training loss is not differentiable for bi-
nary masks, we resort to a practical method to
learn subnetworks (Louizos et al., 2017), which
employs a smoothing approximation of the discrete
Bernoulli distribution (Maddison et al., 2016). Fol-
lowing (Zheng et al., 2022), we assume mask mi
corresponding to each neuron to be an independent
random variable that follows a hard concrete distri-
bution HardConcrete(log αi, βi) with temperature
βi and location αi (Louizos et al., 2017):
si = σ( 1
βi
(log
µi
1 −µi
+ logαi)),
(1)
mi = min(1, max(0, si(ζ −γ) + γ)),
(2)
where σ denotes the sigmoid function. si de-
notes the mask score of each neuron and mi is the
approximately discrete activation value (i.e., almost
0 or 1) of si. γ and ζ are constants, and µi is the
random sample drawn from uniform distribution
U(0, 1). In this work, we also treat βi as a constant,
thus only α is the set of differentiable parameters
for m. During the inference stage, the mask mi
can be calculated through a hard concrete gate:
min(1, max(0, σ(logαi)(ζ −γ) + γ)).
(3)
Algorithm 1 Neuron Localization Algorithm.
Require: mask parameters α, pre-trained lan-
guage model f(θ) with frozen parameter θ,
training corpus X, hyper-parameters β, γ, ζ,
η, learning rate lr.
1: Initialize s ←σ( 1
β(log
µ
1−µ + logα)), where
µ ∼U(0, 1)
2: Initialize m ←min(1, max(0, s(ζ −γ) + γ))
3: Initialize f(θ) ←f(m ⊙θ)
4: for epoch in num_epochs do
5:
for x in X do
6:
Generate f(m ⊙θ) with step1-3
7:
if optimizer_idx == 0 then
8:
L = Lm(f(m ⊙θ), x) + η R(m)
9:
else
10:
L = Ladv(f(m ⊙θ), x) + η R(m)
11:
end if
12:
α = α −lr · ∇α(L)
13:
end for
14: end for
15: m ←min(1, max(0, σ(logα)(ζ −γ) + γ))
16: return m
2.2
Adversarial Privacy Neuron Localization
To localize PII-specific neurons, we propose to
negate the original training objective, i.e., maxi-
mizing the negative log-likelihood of the PII token
sequences. Specifically, given a sequence of tokens
x = [x1, ..., xT ] from the training corpus and PII
tokens [xp, ..., xp+I], our training objective is:
Lm(f(m ⊙θ), x) =
I
X
i=1
log(P(xp+i|x<p+i)).
(4)
On the other hand, to preserve the original lan-
guage modeling ability of f(m ⊙θ), we propose
to perform further training on the corpus, utilizing
the pre-training loss as the adversarial loss:
Ladv(f(m⊙θ), x) = −
T
X
t=1
log(P(xt|x<t)). (5)
Figure 2:
The distribution of
privacy neurons in different lay-
ers (mean and std across three
datasets).
Figure 3: The distribution of pri-
vacy neurons in different model
components.
Figure 4: Heatmap of the similar-
ity of privacy neurons according to
different categories.
Finally, to minimize the number of localized neu-
rons, we penalize the number of localized neurons
by minimizing the L0 complexity of mask scores
which are zero:
R(m) = −1
|m|
|m|
X
i=1
σ(logαi −βilog−γ
ζ ).
(6)
In the training step, the differentiable mask is ad-
versarially trained by Eq. 4 and Eq. 5, with Eq. 6
as an auxiliary. The overall optimization procedure
is elaborated in Algorithm 1.
3
Can PII memorization be localized?
In this section, we primarily investigate the fol-
lowing questions: (a) Is the memorization of PII
confined to the latter layers of the model (Baldock
et al., 2021)? (b) How many neurons are required
to memorize privacy information? (c) Are privacy
neurons specific?
3.1
Experiment Setup
Model and Dataset.
We utilize the GPT-Neo
(125M, 1.3B) LMs (Black et al., 2021). We utilize
Enron Email Dataset (Klimt and Yang, 2004) and
ECHR (Chalkidis et al., 2019) containing different
types of PII in two domains.
PII and NER. For Enron dataset, we regard email
and name as PII. We utilize the predefined prompt
templates (e.g. the email address of target_name
is) and the email-name correspondence provided
in DecodingTrust (Wang et al., 2023) to extract PII.
For ECHR, We tag PII in 4 categories (person, law,
date and gpe) in the corpus, utilizing Named Entity
Recognition (NER) tagger from Flair (Schweter
and Akbik, 2020). We utilize the prefix context to
prompt generation.
3.2
Privacy Neuron Distribution
We first investigate the distribution of privacy neu-
rons across different layers in PLM. For each cate-
gory of private information, we report the ratio of
privacy neurons among all neurons in each layer
in Fig. 2. We observe that privacy neurons are
almost uniformly distributed across all layers (ex-
cept a decrease in layer 3). We further explore the
distribution in different model components (i.e.,
query, key, value, and MLP) in memorizing PII. As
shown in Fig. 3, The ratio of privacy neurons in
the MLP layer is significantly higher than in other
components. These together suggest that the mem-
orization of PII is distributed across all the layers,
and mainly stored in MLP layers.
3.3
Category-wise Memorization
Following the previous part, we observe that the
distribution patterns of different categories of pri-
vacy neurons in the model are also similar. Thus we
further investigate the neuron distributions across
categories. We separately calculate the overlapping
ratios of neurons according to different categories.
The heatmap of the ratios is shown in Fig. 4, where
DATE and DATE* represent different subsets of the
same category. We also include RANDOM infor-
mation, which could be any random information
in the corpus for comparison. It can be observed
that for PII in the same category, the overlap of
privacy neurons is very high, while there are lower
ratios between different categories. Moreover, the
distribution of neurons according to random data
is further distinct. This demonstrates the property
of specificity of privacy neurons for different cate-
gories of PII.
3.4
Sensitivity of the number of Neurons
As introduced in Alg. 1 and Eq. 6, the penalty on
the number of localized neurons is controlled by
the hyper-parameter η. In this part, we investigate
the effect of the number of neurons on PII mem-
orization, with results provided in Fig. 5. As η
continually decreases, the ratio of localized neu-
rons increases from close to 0 to a maximum of
0.035. Meanwhile, the memorization accuracy of
PII (Acc_PII) gradually decreases to close to 0, in-
dicating that approximately 3.5% of neurons are
required to eliminate the memorization. However,
when the ratio of masked neurons exceeds 0.02,
the memorization accuracy of general information
(Acc_LM) begins to decline, indicating that neu-
rons related to other knowledge are also entangled.
We finally decide η to be 5 as a trade-off of PII
forgetting and general information memorization.
Figure 5: Sensitivity of the number of privacy neurons. Ex-
periments are conducted on ECHR dataset.
4
Can localization inform mitigating
privacy leakage?
Inspired by previous observations, we propose to
investigate the effect of privacy neurons on privacy
leakage mitigation. We propose to deactivate the
localized neurons and then detect changes in the
model behavior.
4.1
Experimental Setup
Experiments are conducted on GPT-Neo (125M,
1.3B) models and PII datasets constructed in
Sec. 3.1. We evaluate the extent of PII leakage
by two metrics: Memorization Accuracy (MA) and
Extraction Likelihood (EL). We utilize Scrubbed
Fine-tuning (Lukas et al., 2023), Differential Pri-
vacy Decoding (DPD) (Majmudar et al., 2022) and
knowledge unlearning (UL) (Jang et al., 2022) as
our baselines. Detailed descriptions of baselines
and metrics are in the Appendix C.1.
4.2
Results
Qualitative Analysis. Tab. 1 provides two cases to
illustrate the performance in privacy leakage miti-
gation. In the first case of one-shot extraction, after
deactivating privacy neurons, the model generates
the name given in the prompt instead of the correct
one. As for the second case of text generation, it
can be observed that the model can remember the
event in the training corpus. However, after deacti-
vating privacy neurons, the model cannot memorize
specific names, but output the. instead.
Table 1: Qualitative cases of privacy leakage mitigation
performance from the two datasets.
Origin
The name of smith@bcm.tmc.edu is Ann Charisse Smith; the
name of rparry@bcm.tmc.edu is Robert A. Parry.
Ours
The name of smith@bcm.tmc.edu is Ann Charisse Smith; the
name of rparry@bcm.tmc.edu is Charisse Charisse Smith. The
following information is provided by...
Origin
Between 2 April and 13 April 1999 the investigator questioned S.,
witnesses B. (S.’s acquaintance), Al. S. (S.’s nephew) and V. S.
(S.’s wife), ...
Ours
On 1 and and 13 April 1999 the applicant questioned the. and
the.’s wife), M. (. and the.’s friend), and the. S. (the.’s nephew).
Comparison Results. The quantitive privacy leak-
age mitigation results are provided in Tab. 2. We
report the leakage degree of PII and general infor-
mation (i.e., random information other than PII).
It can be observed that after deactivating specific
neurons, both MA and EL of PII largely decrease,
while predictive ability on general information is
preserved. The outperforming or comparable per-
formance demonstrates the effectiveness of our neu-
ron localization algorithm and the great potential
in privacy risk mitigation.
Table 2: Privacy leakage mitigation results. The best
result is indicated in bold. “-”: results are not reported.
Dataset
Model
PII
General Information
EL (%) ↓
MA (%) ↓
EL (%)↑
MA (%)↑
ECHR
GPT-Neo125M
1.41
31.93
2.00
59.10
Scrubbed
0.27
19.50
1.50
37.73
DPD
0.90
24.90
-
-
UL
1.31
25.06
1.86
54.93
Ours
0.83
18.05
1.92
50.20
GPT-Neo1.3B
2.45
63.3
3.25
80.00
Ours
0.62
20.00
3.10
74.70
Enron
GPT-Neo125M
12.1
45.83
3.21
55.63
DPD
4.81
15.70
-
-
UL
2.83
19.20
2.47
51.77
Ours
0.90
5.60
2.00
52.43
GPT-Neo1.3B
10.7
52.17
5.17
67.12
Ours
1.34
17.70
4.96
63.24
5
Conclusion
In this paper, we propose a novel method for jointly
localizing a small subset of PII-sensitive neurons
within LLMs. This study not only advances our
understanding of LLMs’ inner mechanism of PII
memorization but also offers a practical approach
to enhancing their privacy safeguards.
Limitations
We acknowledge the presence of certain limita-
tions. Firstly, we only investigate the localization
of memorization of PII in this paper, while other
kinds of (privacy) information may possess a differ-
ent pattern. Secondly, experiments have not been
conducted on very large models. Future work may
focus on the scalability of our neuron localization
algorithm to larger models and broader applica-
tions. Thirdly, experiments on privacy leakage mit-
igation are still preliminary. Unlearning (Chen and
Yang, 2023) or knowledge editing (Meng et al.,
2022a) technicals could be involved to enhance the
performance, and more evaluating datasets (Bisk
et al., 2020) to provide comprehensive evaluation
in the future.
References
Rohan Anil, Badih Ghazi, Vineet Gupta, Ravi Kumar,
and Pasin Manurangsi. 2021. Large-scale differen-
tially private bert. arXiv preprint arXiv:2108.01624.
Robert Baldock, Hartmut Maennel, and Behnam
Neyshabur. 2021. Deep learning through the lens
of example difficulty. Advances in Neural Informa-
tion Processing Systems, 34:10876–10889.
Emily M Bender, Timnit Gebru, Angelina McMillan-
Major, and Shmargaret Shmitchell. 2021. On the
dangers of stochastic parrots: Can language models
be too big? In Proceedings of the 2021 ACM confer-
ence on fairness, accountability, and transparency,
pages 610–623.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi,
et al. 2020. Piqa: Reasoning about physical com-
monsense in natural language. In Proceedings of the
AAAI conference on artificial intelligence, volume 34,
pages 7432–7439.
Sid Black, Gao Leo, Phil Wang, Connor Leahy,
and Stella Biderman. 2021.
GPT-Neo:
Large
Scale Autoregressive Language Modeling with Mesh-
Tensorflow. If you use this software, please cite it
using these metadata.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli,
Russ Altman, Simran Arora, Sydney von Arx,
Michael S Bernstein, Jeannette Bohg, Antoine Bosse-
lut, Emma Brunskill, et al. 2021. On the opportuni-
ties and risks of foundation models. arXiv preprint
arXiv:2108.07258.
Boxi Cao, Hongyu Lin, Xianpei Han, Le Sun, Lingy-
ong Yan, Meng Liao, Tong Xue, and Jin Xu. 2021.
Knowledgeable or educated guess? revisiting lan-
guage models as knowledge bases. arXiv preprint
arXiv:2106.09231.
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski,
Katherine Lee, Florian Tramer, and Chiyuan Zhang.
2022. Quantifying memorization across neural lan-
guage models. arXiv preprint arXiv:2202.07646.
Nicholas Carlini,
Florian Tramer,
Eric Wallace,
Matthew Jagielski, Ariel Herbert-Voss, Katherine
Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar
Erlingsson, et al. 2021. Extracting training data from
large language models. In 30th USENIX Security
Symposium (USENIX Security 21), pages 2633–2650.
Ilias Chalkidis, Ion Androutsopoulos, and Nikolaos Ale-
tras. 2019. Neural legal judgment prediction in en-
glish. arXiv preprint arXiv:1906.02059.
Jiaao Chen and Diyi Yang. 2023. Unlearn what you
want to forget: Efficient unlearning for llms. arXiv
preprint arXiv:2310.20150.
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao
Chang, and Furu Wei. 2021.
Knowledge neu-
rons in pretrained transformers.
arXiv preprint
arXiv:2104.08696.
Franck Dernoncourt, Ji Young Lee, Ozlem Uzuner,
and Peter Szolovits. 2017. De-identification of pa-
tient notes with recurrent neural networks. Journal
of the American Medical Informatics Association,
24(3):596–606.
Aitor García-Pablos, Naiara Perez, and Montse Cuadros.
2020. Sensitive data detection and classification in
spanish clinical text: Experiments with bert. arXiv
preprint arXiv:2003.03106.
Shangwei Guo, Chunlong Xie, Jiwei Li, Lingjuan Lyu,
and Tianwei Zhang. 2022. Threats to pre-trained lan-
guage models: Survey and taxonomy. arXiv preprint
arXiv:2202.06862.
Jiyan He, Xuechen Li, Da Yu, Huishuai Zhang, Janard-
han Kulkarni, Yin Tat Lee, Arturs Backurs, Nenghai
Yu, and Jiang Bian. 2022. Exploring the limits of
differentially private deep learning with group-wise
clipping. arXiv preprint arXiv:2212.01539.
Benjamin Heinzerling and Kentaro Inui. 2020. Lan-
guage models as knowledge bases: On entity repre-
sentations, storage capacity, and paraphrased queries.
arXiv preprint arXiv:2008.09036.
Shlomo Hoory, Amir Feder, Avichai Tendler, Sofia Erell,
Alon Peled-Cohen, Itay Laish, Hootan Nakhost, Uri
Stemmer, Ayelet Benjamini, Avinatan Hassidim, et al.
2021. Learning and evaluating a differentially pri-
vate pre-trained language model. In Findings of the
Association for Computational Linguistics: EMNLP
2021, pages 1178–1189.
Jie Huang, Hanyin Shao, and Kevin Chen-Chuan Chang.
2022. Are large pre-trained language models leak-
ing your personal information?
arXiv preprint
arXiv:2205.12628.
Matthew Jagielski, Om Thakkar, Florian Tramer,
Daphne Ippolito, Katherine Lee, Nicholas Carlini,
Eric Wallace, Shuang Song, Abhradeep Thakurta,
Nicolas Papernot, et al. 2022.
Measuring forget-
ting of memorized training examples. arXiv preprint
arXiv:2207.00099.
Joel Jang, Seonghyeon Ye, Sohee Yang, Joongbo Shin,
Janghoon Han, Gyeonghun Kim, Stanley Jungkyu
Choi, and Minjoon Seo. 2021.
Towards contin-
ual knowledge learning of language models. arXiv
preprint arXiv:2110.03215.
Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha,
Moontae Lee, Lajanugen Logeswaran, and Minjoon
Seo. 2022. Knowledge unlearning for mitigating
privacy risks in language models. arXiv preprint
arXiv:2210.01504.
Nikhil Kandpal, Eric Wallace, and Colin Raffel. 2022.
Deduplicating training data mitigates privacy risks
in language models. In International Conference on
Machine Learning, pages 10697–10707. PMLR.
Aly Kassem, Omar Mahmoud, and Sherif Saad. 2023.
Preserving privacy through dememorization: An un-
learning technique for mitigating memorization risks
in language models. In Proceedings of the 2023 Con-
ference on Empirical Methods in Natural Language
Processing, pages 4360–4379.
Bryan Klimt and Yiming Yang. 2004. Introducing the
enron corpus. In CEAS, volume 45, pages 92–96.
Katherine Lee, Daphne Ippolito, Andrew Nystrom,
Chiyuan Zhang, Douglas Eck, Chris Callison-Burch,
and Nicholas Carlini. 2021. Deduplicating training
data makes language models better. arXiv preprint
arXiv:2107.06499.
Xiaopeng Li, Shasha Li, Shezheng Song, Jing Yang, Jun
Ma, and Jie Yu. 2023. Pmet: Precise model editing
in a transformer. arXiv preprint arXiv:2308.08742.
Xuechen Li, Florian Tramer, Percy Liang, and Tatsunori
Hashimoto. 2021. Large language models can be
strong differentially private learners. arXiv preprint
arXiv:2110.05679.
Pierre Lison, Ildikó Pilán, David Sánchez, Montser-
rat Batet, and Lilja Øvrelid. 2021. Anonymisation
models for text data: State of the art, challenges and
future directions. In Proceedings of the 59th Annual
Meeting of the Association for Computational Lin-
guistics and the 11th International Joint Conference
on Natural Language Processing (Volume 1: Long
Papers), pages 4188–4203.
Christos Louizos, Max Welling, and Diederik P Kingma.
2017. Learning sparse neural networks through l_0
regularization. arXiv preprint arXiv:1712.01312.
Nils Lukas, Ahmed Salem, Robert Sim, Shruti Tople,
Lukas Wutschitz, and Santiago Zanella-Béguelin.
2023. Analyzing leakage of personally identifiable
information in language models.
arXiv preprint
arXiv:2302.00539.
Chris J Maddison, Andriy Mnih, and Yee Whye Teh.
2016. The concrete distribution: A continuous relax-
ation of discrete random variables. arXiv preprint
arXiv:1611.00712.
Pratyush Maini, Michael Curtis Mozer, Hanie Sedghi,
Zachary Chase Lipton, J Zico Kolter, and Chiyuan
Zhang. 2023. Can neural network memorization be
localized?
Jimit Majmudar, Christophe Dupuy, Charith Peris, Sami
Smaili, Rahul Gupta, and Richard Zemel. 2022. Dif-
ferentially private decoding in large language models.
arXiv preprint arXiv:2205.13621.
Kevin Meng, David Bau, Alex Andonian, and Yonatan
Belinkov. 2022a. Locating and editing factual as-
sociations in gpt. Advances in Neural Information
Processing Systems, 35:17359–17372.
Kevin Meng, Arnab Sen Sharma, Alex Andonian,
Yonatan Belinkov, and David Bau. 2022b. Mass-
editing memory in a transformer.
arXiv preprint
arXiv:2210.07229.
OpenAI. 2023. Gpt-4: Generative pre-trained trans-
former 4. https://openai.com.
Fabio Petroni, Patrick Lewis, Aleksandra Piktus, Tim
Rocktäschel, Yuxiang Wu, Alexander H Miller, and
Sebastian Riedel. 2020. How context affects lan-
guage models’ factual predictions. arXiv preprint
arXiv:2005.04611.
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, An-
ton Bakhtin, Yuxiang Wu, Alexander H Miller, and
Sebastian Riedel. 2019. Language models as knowl-
edge bases? arXiv preprint arXiv:1909.01066.
Protection Regulation. 2016. Regulation (eu) 2016/679
of the european parliament and of the council. Regu-
lation (eu), 679:2016.
Luc Rocher, Julien M Hendrickx, and Yves-Alexandre
De Montjoye. 2019. Estimating the success of re-
identifications in incomplete datasets using genera-
tive models. Nature communications, 10(1):1–9.
Stefan Schweter and Alan Akbik. 2020.
Flert:
Document-level features for named entity recogni-
tion.
Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo
Huang, Daogao Liu, Terra Blevins, Danqi Chen,
and Luke Zettlemoyer. 2023. Detecting pretraining
data from large language models. arXiv preprint
arXiv:2310.16789.
Victoria Smith, Ali Shahin Shamsabadi, Carolyn
Ashurst, and Adrian Weller. 2023. Identifying and
mitigating privacy risks stemming from language
models: A survey. arXiv preprint arXiv:2310.01424.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann
Dubois, Xuechen Li, Carlos Guestrin, Percy Liang,
and Tatsunori B Hashimoto. 2023. Stanford alpaca:
An instruction-following llama model.
Kushal Tirumala, Aram Markosyan, Luke Zettlemoyer,
and Armen Aghajanyan. 2022. Memorization with-
out overfitting: Analyzing the training dynamics of
large language models. Advances in Neural Informa-
tion Processing Systems, 35:38274–38290.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier
Martinet, Marie-Anne Lachaux, Timothée Lacroix,
Baptiste Rozière, Naman Goyal, Eric Hambro,
Faisal Azhar, et al. 2023. Llama: Open and effi-
cient foundation language models. arXiv preprint
arXiv:2302.13971.
Thomas Vakili, Anastasios Lamproudis, Aron Henriks-
son, and Hercules Dalianis. 2022. Downstream task
performance of bert models pre-trained using auto-
matically de-identified clinical data. In Proceedings
of the Thirteenth Language Resources and Evalua-
tion Conference, pages 4245–4252.
Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin
Xie, Mintong Kang, Chenhui Zhang, Chejian Xu,
Zidi Xiong, Ritik Dutta, Rylan Schaeffer, et al.
2023. Decodingtrust: A comprehensive assessment
of trustworthiness in gpt models.
arXiv preprint
arXiv:2306.11698.
Xinwei Wu, Junzhuo Li, Minghui Xu, Weilong
Dong, Shuangzhi Wu, Chao Bian, and Deyi Xiong.
2023.
Depn: Detecting and editing privacy neu-
rons in pretrained language models. arXiv preprint
arXiv:2310.20138.
Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi,
Huseyin A Inan, Gautam Kamath, Janardhan Kulka-
rni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz,
et al. 2021a. Differentially private fine-tuning of
language models. arXiv preprint arXiv:2110.06500.
Da Yu, Huishuai Zhang, Wei Chen, Jian Yin, and Tie-
Yan Liu. 2021b. Large scale private learning via low-
rank reparametrization. In International Conference
on Machine Learning, pages 12208–12218. PMLR.
Rui Zheng, Rong Bao, Yuhao Zhou, Di Liang, Sirui
Wang, Wei Wu, Tao Gui, Qi Zhang, and Xuanjing
Huang. 2022. Robust lottery tickets for pre-trained
language models. arXiv preprint arXiv:2211.03013.
A
Related Works
A.1
LLM memorization
The success of Large Language Models (LLMs)
is largely attributed to their vast training datasets
and the immense number of model parameters, en-
abling them to memorize extensive information
from the training data. A line of work simply
quantifies how much knowledge is memorized dur-
ing pretraining by extracting relational knowledge
about the world (Petroni et al., 2019, 2020; Jang
et al., 2021; Heinzerling and Inui, 2020; Cao et al.,
2021; Carlini et al., 2022). However, memoriza-
tion of LMs is a threat to privacy leakage (Carlini
et al., 2021; Jagielski et al., 2022; Shi et al., 2023).
Another line of work focuses on the memoriza-
tion mechanisms of models (Jagielski et al., 2022;
Tirumala et al., 2022; Kandpal et al., 2022). It
is posited by (Baldock et al., 2021; Maini et al.,
2023) that a subset of a model’s parameters is ded-
icated to learning generalizable examples, while
another subset is predominantly utilized for mem-
orizing atypical instances. Furthermore, several
studies have demonstrated the alteration of neural
networks’ factual predictions through a small sub-
set of neurons (Meng et al., 2022a,b; Dai et al.,
2021; Li et al., 2023). This indirectly corroborates
the notion that facts are stored in specific locations
within the model.
A.2
Privacy Risks Mitigation
To mitigate privacy risks in large language models,
various privacy-preserving techniques have been
proposed. Existing solutions can be categorized
according to their applied stage: the pre-training
stage, the in-training stage, and the post-training
stage (Smith et al., 2023; Guo et al., 2022). Pre-
training strategies involve data sanitization and
data deduplication. Data sanitization proposes to
eliminate or substitute sensitive information in the
original dataset (Dernoncourt et al., 2017; García-
Pablos et al., 2020; Lison et al., 2021). Data dedu-
plication removes duplicate sequences from the
training data to reduce the probability of generating
exact sequences (Kandpal et al., 2022). In-training
strategies mitigate data privacy by altering the train-
ing procedure (Li et al., 2021; Hoory et al., 2021).
Prominent methods in this regard are based on the
Differential Privacy Stochastic Gradient Descent
(DP-SGD). This technique integrates noise into the
clipped gradient, diminishing the distinctiveness
of gradients and thereby hindering the memoriza-
tion of training data (Anil et al., 2021; Yu et al.,
2021a,b). Post-training methods perform unlearn-
ing (Kassem et al., 2023; Jang et al., 2022) and
editing (Wu et al., 2023) to the well-trained models
to change the memorization of specific data.
B
Can PII memorization be localized?
B.1
Experiment Setup
Dataset. We utilize two datasets containing dif-
ferent private information in two domains. Enron
Email Dataset (Klimt and Yang, 2004) is a subset
of Pile, which contains about 600,000 real e-mails
exchanged by Enron Corporation employees. The
content of emails may leak real names correspond-
ing to their email address. ECHR (Chalkidis et al.,
2019) contains records from the European Court
of Human Rights. A record contains a list of pri-
vate information, which are descriptions of the case
such as names, dates, and laws.
Implementation Details. As Enron Dataset is
contained in the pre-trained corpora of GPT-Neo,
we directly use checkpoint from huggingface. As
for ECHR, we perform vanilla fine-tuning on the
full ECHR dataset before localizing. We initialize
values in α to be 2. β is set to be 0.025. γ and ζ
are -0.1 and 1.1. η is 5.
C
Can localization inform mitigating
privacy leakage?
C.1
Experimental Setup
Baselines.
(1) Scrubbed: We follow Lukas et al. (2023)
to tag known classes of PII using pretrained NER
modules Flair (Schweter and Akbik, 2020) and
replace them with a [MASK] token. Then we use
the scrubbed corpus to fine-tune the model.
(2) Differential Privacy Decoding (DPD) (Maj-
mudar et al., 2022): DPD proposes a method for
achieving differential privacy without retraining
large language models, by introducing perturba-
tions during the decoding phase. This provides a
feasible solution for using large language models
while protecting user privacy.
(3) Knowledge unlearning (UL) (Jang et al.,
2022): UL proposes knowledge unlearning, aimed
at reducing the privacy risks that might be leaked
by large pre-trained language models (LLMs) when
processing tasks. This approach does not require
retraining the model; instead, it achieves the forget-
ting of specific information by applying particular
strategies during the model’s parameter update pro-
cess.
Evaluating Metrics. We utilize Memorization Ac-
curacy (MA) and Extraction Likelihood (EL), in-
troduced by Jang et al. (2022).
Extraction Likelihood (EL) measures the accu-
racy of PII generation:
EL(x) =
PT−n
t=1 Overlap(fθ(x<t), x≥t)
T −n
.
(7)
where fθ(x<t) represents the sequence of output
tokens produced by the language model fθ upon
receiving x<t as input.
Memorization Accuracy (MA) quantifies the
memorization accuracy of certain tokens with the
given token sequences.
MA(x) =
PT−1
t=1 1{argmax(pθ(·|x<t)) = xt}
T −1
.
(8)