Question:
Which of the following design choices reflect the trade-off between computational efficiency and forgetting accuracy in the paper? (Select all that apply)
A. Online selection uses the original model's log-prob to avoid an additional model
B. Offline annotation uses a two-stage LLM process
C. Merges adjacent low-probability tokens into spans
D. Main results are reported only for the d=32 setting in experiments

Answer: AC

Moreover, we introduce automatic online and offline methods for sensitive span annotation to facilitate the training and evaluation of SEUL. Online selection relies on language probability scores of tokens, ensuring efficiency in calculation and supporting selective unlearning without additional dependencies. For offline annotation, we employ a two-stage process with a large language model (LLM, e.g., ChatGPT), ensuring thorough annotation and verification. The offline annotated spans serve to evaluate unlearned language models, accompanied by two novel unlearning evaluation metrics—sensitive extraction likelihood (S-EL) and sensitive memorization accuracy (S-MA)—specifically designed to address sensitive information unlearning.

Online Selection
When conducting unlearning based on our proposed learning objective (i.e., Eq. 1) without providing forget spans but for general goal of forgetting (i.e., reducing the privacy leakage risk of Df), we propose an automatic and efficient target span online selection method. We assume that content spans with privacy leakage risks are most likely not common knowledge (i.e. the trained model should not have seen them frequently during training). This means that they are probably with low language probability in language models (a higher perplexity score accordingly). Based on the assumption, we thereby define the following online selection process:
Select(x, α) = {xt | log(p′θ(xt|x<t)) < α}
where p′θ(·) indicates the language probability of original model (before unlearning) and α should be a predefined language probability threshold or the average log-probability of the tokens in x (i.e. α = Σxt∈x log(p′θ(xt|x<t)). We also include tokens that are located between two closely positioned tokens (for any xi and xj, we consider them close if |i−j| ≤2). This decision is based on our preliminary observation that, in some cases, a complete word or phrase may exhibit a higher language probability in the middle tokens. Then we aggregate all the selectively adjacent tokens into spans, forming sx = (s1, s2, ..., sm). The efficient selection of forget spans described above can be conducted during the training of the language model, thereby facilitating our proposed selective unlearning. As this selection is based on our aforementioned assumption, we will empirically discuss its rationality in the Experimental Results Section.

Offline Annotation
Online selection provides a rapid and cost-effective solution, but it may lack the precision required for identifying genuinely privacy-sensitive tokens. To address this limitation, we propose an offline forget spans annotation method that ensures greater accuracy. The annotation workflow and the original forget set Df together with the annotated forget span sets could serve as a valuable resource to evaluate unlearning methods’ efficiency in forgetting target knowledge. In the following, we describe our bidirectional verified LLM-based annotation process.
(i) Forward Span Annotation. We leverage the strong in-context learning capability of large language models to facilitate forget span annotation. As previously mentioned, this work primarily focuses on the general purpose of unlearning, i.e., forgetting information with high risks of leaking private information. To achieve this, we instruct large language models (e.g., ChatGPT) to annotate sensitive spans in a given text using a few-shot setting. The detailed prompt is provided in the Appendix. Following this prompt, the large language models generate sensitive spans1 for each instance x in the forget set Df. The entire process can be summarized as follows:
ˇsx ←F(x, D)
where D denotes the examples used in few-show learning.
(ii) Backward Verification. Previous research (Ling et al. 2023; Shinn et al. 2023) has demonstrated double-checking the reasoning results generated by LLMs can produce significantly more reliable content. In this work, we employ a backward verification mechanism. Instead of merely prompting LLMs to validate the accuracy of previously generated sensitive spans, we instruct them to independently assess and score the generated spans on a scale of {0, 1, 2} without providing the entire sequence (i.e., x). Subsequently, we filter out spans with low scores (i.e., score 0).

Selective Forgetting: Advancing Machine Unlearning Techniques and
Evaluation in Language Models
Lingzhi Wang 1, Xingshan Zeng2*, Jinsong Guo3, Kam-Fai Wong4,5, Georg Gottlob6
1Harbin Institute of Technology, Shenzhen, China
2Huawei Noah’s Ark Lab, China 3Unlimidata Ltd, United Kingdom 4The Chinese University of Hong Kong, China
5MoE Key Laboratory of High Confidence Software Technologies, China 6University of Calabria, Italy
wanglingzhi@hit.edu.cn, zeng.xingshan@huawei.com, jinsong.guo@unlimidata.com,
kfwong@se.cuhk.edu.hk, georg.gottlob@unical.it
Abstract
This paper explores Machine Unlearning (MU), an emerging
field that is gaining increased attention due to concerns about
neural models unintentionally remembering personal or sen-
sitive information. We present SEUL, a novel method that en-
ables selective and fine-grained unlearning for language mod-
els. Unlike previous work that employs a fully reversed train-
ing objective in unlearning, SEUL minimizes the negative
impact on the capability of language models, particularly in
terms of generation. Furthermore, we introduce two innova-
tive evaluation metrics, sensitive extraction likelihood (S-EL)
and sensitive memorization accuracy (S-MA), specifically
designed to assess the effectiveness of forgetting sensitive in-
formation. In support of the unlearning framework, we pro-
pose efficient automatic online and offline sensitive span an-
notation methods. The online selection method, based on lan-
guage probability scores, ensures computational efficiency,
while the offline annotation involves a two-stage LLM-based
process for robust verification. In summary, this paper con-
tributes a novel selective unlearning method (SEUL), intro-
duces specialized evaluation metrics (S-EL and S-MA) for
assessing sensitive information forgetting, and proposes au-
tomatic online and offline sensitive span annotation methods
to support the overall unlearning framework and evaluation
process.
Introduction
Machine Unlearning (MU) (Romero, Barrio, and Belanche
2007; Karasuyama and Takeuchi 2009; Cao and Yang 2015)
has increasingly attracted the attention of researchers. The
focus on MU stems primarily from the fact that neural mod-
els are trained on data mainly sourced from the Internet,
and the trained model may permanently “remember” per-
sonal or sensitive information contained in the training data.
Concerns about the leakage of personal sensitive data from
neural networks have intensified, especially following the
breakthrough of language models, which exhibit incredible
capabilities in data generation. Meanwhile, the “right to be
forgotten” has been legislated in many countries, such as the
General Data Protection Regulation (GDPR) in the Euro-
pean Union and the PIPEDA privacy legislation in Canada.
* Xingshan Zeng is the corresponding author.
Copyright © 2025, Association for the Advancement of Artificial
Intelligence (www.aaai.org). All rights reserved.
… Please email Bob at Bob@xxx.com ...
…Bob is currently a third-year student at
xxx University Junior High School ...
Bob
… Please email Eve at Eve@xxx.com.
Earth is round ...
…Eve is currently a third-year student at
xxx University Junior High School.
Trump is the 45th President of
the United States...
Eve
(Attacker)
Normal Deletion Query (containing only
personal or sensitive information)
𝑞!:
𝑞":
Deletion Query in Attack (injecting common
sense or knowledge into the query)
𝑞!
#:
𝑞"
# :
Figure 1: Illustration of knowledge injection attack.
This right mandates that companies erase personal data upon
user request. Furthermore, while removing data from back-
end databases is straightforward, it poses a challenge for
neural models as the relationship between the model weights
and the data is unclear.
Given the identified necessities and challenges in unlearn-
ing, particularly in the context of large language models,
researchers have focused on machine unlearning to make
trained models forget specific data. While many prior works
(Golatkar, Achille, and Soatto 2020a,b; Mehta et al. 2022)
in machine unlearning address computer vision classifica-
tion tasks, fewer target generation tasks in NLP. Wang et al.
(2023) propose a general machine unlearning framework,
but it relies on additional model training, which is costly
for language models. Meanwhile, Jang et al. (2023) intro-
duces sequential knowledge unlearning for language mod-
els, utilizing a reversed language modeling learning objec-
tive. However, employing a fully reversed training objective
in unlearning can significantly impact the language model’s
generation capability.
In contrast to Jang et al. (2023), which fully reverses the
training loss of instances for forgetting, we propose a selec-
tive unlearning method, SEUL. SEUL achieves knowledge
forgetting in a fine-grained manner, focusing on specific se-
quence spans rather than entire instances, as illustrated in
Fig. 1. In the scenarios where an attacker (e.g., Eve in Fig. 1)
injects general knowledge into regular personal data and re-
arXiv:2402.05813v2  [cs.CL]  16 Dec 2024
quests data forgetting, fully reversed unlearning can com-
promise the model. Our selective unlearning method, how-
ever, better preserves the generation capability of language
models, especially when most parts of the sequence to be
deleted consist of general expressions. Beyond extreme ad-
versarial cases, fully reversed unlearning can also detrimen-
tally impact the capability of language models as the number
of deletion queries increases. This is due to its inherent na-
ture of affecting model’s training on all tokens in deletion
queries without selection.
Moreover, we introduce automatic online and offline
methods for sensitive span annotation to facilitate the train-
ing and evaluation of SEUL. Online selection relies on lan-
guage probability scores of tokens, ensuring efficiency in
calculation and supporting selective unlearning without ad-
ditional dependencies. For offline annotation, we employ a
two-stage process with a large language model (LLM, e.g.,
ChatGPT), ensuring thorough annotation and verification.
The offline annotated spans serve to evaluate unlearned lan-
guage models, accompanied by two novel unlearning eval-
uation metrics—sensitive extraction likelihood (S-EL) and
sensitive memorization accuracy (S-MA)—specifically de-
signed to address sensitive information unlearning.
In brief, the main contributions of this paper are:
• We propose a novel unlearning method called SEUL,
which facilitates selective, fine-grained, and effective un-
learning in language models. It shows a comparable ability
to maintain classification performance and a significantly
enhanced capability in preserving generation performance
after unlearning, as compared to the SOTA.
• We propose two new evaluation metrics, namely sensi-
tive extraction likelihood (S-EL) and sensitive memoriza-
tion accuracy (S-MA), to emphasize the assessment of un-
learning methods based on their ability to forget sensitive
information rather than general information.
• We propose both online and offline automatic sensitive in-
formation annotation methods to assist the effective selec-
tive unlearning framework and facilitate efficient unlearn-
ing evaluation.
Related Work
We provide a comprehensive overview of machine unlearn-
ing, focusing on three key aspects: methodology, datasets,
and evaluation metrics.
Methodology.
Machine unlearning falls into two cate-
gories: Exact Unlearning and Approximate Unlearning. Ex-
act Unlearning (Cao and Yang 2015; Ginart et al. 2019;
Bourtoule et al. 2021) aims to completely eliminate the im-
pact of deleted data but struggles with scalability in deep
neural networks (Cao and Yang 2015; Ginart et al. 2019)
and efficiency (Bourtoule et al. 2021). In contrast, Approxi-
mate Unlearning methods like Golatkar, Achille, and Soatto
(2020a), Guo et al. (2019), Koh and Liang (2017), and
Mehta et al. (2022) prioritize efficiency at the cost of ex-
actness. Though most methods are developed in computer
vision, there are some recent unlearning works in natural
language (Wang et al. 2023; Jang et al. 2023). Wang et al.
(2023) proposes a general framework that relies on extra
model training to maintain knowledge gap. However, train-
ing extra models is expensive and impractical for large lan-
guage models. Jang et al. (2023) simply negates the training
objective.
Datasets.
Currently, no dedicated datasets exist for the ex-
plicit examination and evaluation of unlearning methods.
Researchers typically assess the efficacy and efficiency of
unlearning approaches across diverse datasets based on indi-
vidual considerations. For instance, Lu et al. (2022) investi-
gate toxicity unlearning using the RealToxicityPrompts (Liu
et al. 2021) and WritingPrompts (Fan, Lewis, and Dauphin
2018) datasets. In another study, Wang et al. (2023) perform
experiments on three representative datasets: LEDGAR
(Tuggener et al. 2020), IWSLT14 German-English (Cettolo
et al. 2014), and PersonaChat (Zhang et al. 2018). This eval-
uation encompasses classification, translation, and genera-
tion tasks. Additionally, Jang et al. (2023) conduct exper-
iments on reasoning datasets, such as Hellaswag (Zellers
et al. 2019) and MathQA (Amini et al. 2019), to assess lan-
guage models’ capabilities before and after the unlearning
process.
Evaluation Metrics.
We mainly discuss the unlearning
evaluation in natural language field here. Patil, Hase, and
Bansal (2023) assess unlearning performance using ∆-Acc,
comparing accuracy scores before and after data point dele-
tion on CounterFact (Meng et al. 2022) and zsRE (Levy
et al. 2017). Jang et al. (2023) introduce Extraction Likeli-
hood (EL) and leverage Memorization Accuracy (MA) from
(Tirumala et al. 2022) to quantitatively measure informa-
tion leakage during extraction attacks. Additionally, Wang
et al. (2023) evaluate unlearning through Jensen–Shannon
Divergence (JSD), Language Model Probability Distance
(LPD), and Proportion of instances with Decreased Lan-
guage Model Probability (PDLP). It is noteworthy that
this method heavily relies on retraining models to assess
changes, which can be prohibitively costly, especially in the
case of large language models.
Selective Unlearning For LMs
Our Unlearning Methodology
Problem Formulation
By following Wang et al. (2023),
we formulate machine unlearning as follows. Given a train-
ing set D ∈Z∗, The process of training a model on data
set D is denoted by a function A : Z∗→H, where Z is
the space of data instances and H is the hypothesis space
of models. Then the trained model can be denoted as A(D).
For general machine unlearning, the unlearning mechanism
is denoted as a function U, which takes a training dataset
D ∈Z∗(optional), a forget set Df ⊂D (containing data
to be unlearned) and a model A(D) as input, and returns an
unlearned model U(D, Df, A(D)) ∈H.
For our selective unlearning, for each instance in the for-
get set Df, we define a forget span set that explicitly refers
to the sequence spans that need to be unlearned. Formally,
for any x = (x1, x2, . . . , xT ) ∈Df, the forget span set is
denoted as sx = (s1, s2, . . . , sm), where m is the number of
the forget spans and si represents a continuous sub-sequence
of x, i.e., si = (xji, xji+1, . . . , xji+|si|−1), where ji indi-
cates the original index in x of the first token of si.
Our SEUL Unlearning Method
Different from Jang
et al. (2023) which simply negates the original training ob-
jective of minimizing the negative log-likelihood of token
sequences as an unlearning method, we propose a selec-
tive unlearning method called SEUL. SEUL forgets knowl-
edge in a fine-grained manner, aiming to forget specific se-
quence spans instead of instance-level forgetting. The moti-
vation mainly comes from the fact that instance-level forget-
ting cannot handle the adversarial attack mentioned in Fig. 1
and selective unlearning can have less impact on the perfor-
mance of the original language model. The learning objec-
tive of SEUL is to minimize the following loss function for
all x ∈Df:
LUL(A(D), x) =
X
si∈sx
ji+|si|−1
X
t=ji
log(pθ(xt|x<t))
(1)
where x<t denotes the sequence preceding the index t, and
ji is the start index of subsequence si in original sequence x.
pθ(xt|x<t) denotes the conditional probability of predicting
the next token when given prefix xt to an LM A(D) with pa-
rameters θ. With the learning objective, the model is pushed
to only unlearn the forget spans in forget span set sx without
affecting other general knowledge.
How to Determine the Forgetting Span?
It’s essential to note that, the forgetting span could be tai-
lored based on practical needs (e.g., for spans with toxic
words) in some scenarios. However, a more common sce-
nario is that the forget span set is not predefined, and
we must develop a span selection/annotation mechanism to
achieve the goal of selective unlearning. We propose a com-
prehensive span selection mechanism designed to adapt to
various stages of machine unlearning, specifically address-
ing concerns related to privacy leakage. In the following sec-
tions, we provide details about our forget span selection/an-
notation design.
Online Selection
When conducting unlearning based on
our proposed learning objective (i.e., Eq. 1) without pro-
viding forget spans but for general goal of forgetting (i.e.,
reducing the privacy leakage risk of Df), we propose an
automatic and efficient target span online selection method.
We assume that content spans with privacy leakage risks are
most likely not common knowledge (i.e. the trained model
should not have seen them frequently during training). This
means that they are probably with low language probability
in language models (a higher perplexity score accordingly).
Based on the assumption, we thereby define the following
online selection process:
Select(x, α) =
n
xt | log(p
′
θ(xt|x<t) < α
o
(2)
where p
′
θ(·) indicates the language probability of original
model (before unlearning) and α should be a predefined lan-
guage probability threshold or the average log-probability of
the tokens in x (i.e. α = P
xt∈x log(p
′
θ(xt|x<t)). We also
include tokens that are located between two closely posi-
tioned tokens (for any xi and xj, we consider them close if
|i−j| ≤2). This decision is based on our preliminary obser-
vation that, in some cases, a complete word or phrase may
exhibit a higher language probability in the middle tokens.
Then we aggregate all the selectively adjacent tokens into
spans, forming sx = (s1, s2, ..., sm). The efficient selection
of forget spans described above can be conducted during the
training of the language model, thereby facilitating our pro-
posed selective unlearning. As this selection is based on our
aforementioned assumption, we will empirically discuss its
rationality in the Experimental Results Section.
Offline Annotation
Online selection provides a rapid and
cost-effective solution, but it may lack the precision required
for identifying genuinely privacy-sensitive tokens. To ad-
dress this limitation, we propose an offline forget spans an-
notation method that ensures greater accuracy. The annota-
tion workflow and the original forget set Df together with
the annotated forget span sets could serve as a valuable re-
source to evaluate unlearning methods’ efficiency in forget-
ting target knowledge. In the following, we describe our bi-
directional verified LLM-based annotation process.
(i) Forward Span Annotation. We leverage the strong in-
context learning capability of large language models to facil-
itate forget span annotation. As previously mentioned, this
work primarily focuses on the general purpose of unlearn-
ing, i.e., forgetting information with high risks of leaking
private information. To achieve this, we instruct large lan-
guage models (e.g., ChatGPT) to annotate sensitive spans in
a given text using a few-shot setting. The detailed prompt is
provided in the Appendix. Following this prompt, the large
language models generate sensitive spans1 for each instance
x in the forget set Df. The entire process can be summarized
as follows:
ˇsx ←F(x, D)
(3)
where D denotes the examples used in few-show learning.
(ii) Backward Verification. Previous research (Ling et al.
2023; Shinn et al. 2023) has demonstrated double-checking
the reasoning results generated by LLMs can produce sig-
nificantly more reliable content. In this work, we em-
ploy a backward verification mechanism. Instead of merely
prompting LLMs to validate the accuracy of previously gen-
erated sensitive spans, we instruct them to independently
assess and score the generated spans on a scale of {0, 1,
2} without providing the entire sequence (i.e., x). Subse-
quently, we filter out spans with low scores (i.e., score 0).
Evaluation of Unlearning
In this subsection, we present two newly proposed quantita-
tive evaluation metrics designed for assessing machine un-
learning with a specific focus on handling sensitive informa-
tion: Sensitive Extraction Likelihood (S-EL) and Sensitive
Memorization Accuracy (S-MA).
1An exact token matching-based post-processing is applied to
ensure that the generated spans match the original text input.
Query
Input
… Please email Eve at
Eve@xxx.com. Earth is
round ...
… The concentration of carbon
dioxide in the atmosphere is
currently around 415 parts per
million (ppm). …
Query with predefined spans
Query w/o predefined spans
Online Selection … Please email Eve at
Eve@xxx.com. Earth is
round ...
Our
Unlearning
Method
LM
LM
Unlearned
Offline Annotation
… Please email Eve at
Eve@xxx.com. Earth is
round ...
Unlearned LM
Evaluation
(S-EL, S-MA)
Figure 2: Workflow of SEUL: Queries with predefined spans (either sensitive or within other definitions) can be inputted
directly into SEUL. For queries without predefined spans, we conduct online selection before feeding them to SEUL.
Sensitive Extraction Likelihood
Before introducing S-
EL, we provide a succinct definition of string overlap (OVL)
(following Jang et al. (2023)) and our proposed overlap with
consideration for sensitive information (S-OVL) given two
token sequences a and b:
OVLn(a, b) =
P
c∈n-grams(a) 1{c ⊆b}
|n-grams(a)|
(4)
S-OVLn(a, b) =
P
c∈n-grams(a) 1{(c ∩sb) ̸= ∅∧(c ∩sb) ⊆b}
P
c∈n-grams(a) 1{(c ∩sb) ̸= ∅}
(5)
where 1{·} denotes an indicator function that returns 1 if the
evaluated condition is true; otherwise 0. n-grams() returns
the list of n-grams in the given token sequence, and sb rep-
resents the sensitive span set in b that needs to be unlearned.
c ⊆b indicates that c is the subsequence of b. c ∩sb returns
the common subsequence of c and any span in sb. Notably,
the distinction between S-OVL and OVL lies in the fact that
S-OVL exclusively considers n-grams containing sensitive
spans.
Building upon S-OVL, we propose S-EL, which assesses
the likelihood of a language model fθ generating sensitive
information when prompted with a prefix in x, in the basis
of n-gram.
S-ELn(x) =
PT −n
t=1 S-OVLn(fθ(x<t), x≥t)
T −n
(6)
where fθ(x<t) denotes the suffix generated by the language
model fθ given the prefix x<t.
Sensitive Memorization Accuracy
S-MA quantifies the
model’s accuracy in predicting the next token given a prefix
following Tirumala et al. (2022), but only considering the
sensitive information. We define it as follows.
S-MA(x) =
PT −1
t=1 1{ argmax(pθ(·|x<t)) = xt ∧xt ∈sx}
PT −1
t=1 1{xt ∈sx}
(7)
where argmax(pθ(·|x<t)) denotes the most probable next
token predicted by the language model. T is the total number
of tokens in the sequence x and xt ∈sx indicates that token
xt is contained in any sensitive span of span set sx.
Both S-EL and S-MA provide a robust framework for
evaluating the efficacy of machine unlearning, particularly
in contexts where the handling of sensitive information is a
critical concern. For comparison, we also denote the extrac-
tion likelihood and memorization accuracy without consid-
ering whether the tokens are sensitive or not as ELn (Jang
et al. 2023) and MA (Tirumala et al. 2022) and display their
results together with the two proposed metrics.
Experimental Setup
Datasets and Evaluation Metrics
As we focus on unlearning of language models, we start
from pretrained language models, then we do unlearning on
forget dataset Df and test the models (original LMs and un-
learned LMs) on test set Dt.
Forget Dataset Df.
In order to ensure a fair comparison
with Jang et al. (2023) and to assess the privacy risk asso-
ciated with language models, we employ the identical sam-
ples as those disclosed by Jang et al. (2023). The forget set
is sourced from the Training Data Extraction Challenge2,
comprising 15,000 examples, each consisting of 200-token
sequences from various domains of Pile corpora.
Evaluation Datasets Dt.
To evaluate general language
modeling capabilities, we employ 8 classification tasks (i.e.,
2https://github.com/google-research/lm-extraction-benchmark
Models
Forget Evaluation
Avg. 8 Cla.
Avg. 4 Dia.
Epoch
EL10
MA
S-EL10(↓)
S-MA(↓)
Acc(↑)
F1(↑)
PPL(↓)
#
GPT-Neo
125M
Original
31.5
76.9
22.0
62.4
43.7
7.4
43.6
-
KUL
0.2
26.8
1.3
22.9
40.9
1.1
539.1
15.6
SEUL
1.0
29.5
0.3
16.1
41.0
7.8
179.8
16.2
GPT-Neo
1.3B
Original
60.4
91.1
44.6
81.6
48.6
12.3
26.0
-
KUL
0.3
27.2
1.4
23.9
48.2
10.4
32.4
9.2
SEUL
0.6
29.4
1.9
23.6
48.0
11.4
28.3
8.8
GPT-Neo
2.7B
Original
66.6
93.1
49.5
82.5
50.7
12.3
23.3
-
KUL
0.5
29.8
1.7
23.0
49.9
10.1
30.1
8.6
SEUL
0.3
18.6
1.0
14.6
50.2
10.9
26.7
8.8
LlaMa2
7B
Original
25.7
73.2
19.9
58.2
56.9
13.2
9.0
-
KUL
0.7
29.1
1.5
22.3
52.1
9.7
21.0
7.1
SEUL
0.6
13.9
1.2
12.3
53.6
10.4
16.3
6.8
Mistral
7B
Original
22.9
75.9
19.0
63.9
62.5
14.1
9.1
-
KUL
0.5
27.9
2.1
23.6
53.3
13.3
19.8
7.0
SEUL
0.4
19.1
1.3
20.9
55.7
12.8
17.0
7.4
Table 1: Comparison results (in %) on forget set (d=32), 8 classification datasets and 4 dialogue datasets. The best performance
in comparable columns are highlighted in bold. “Avg.” denotes average scores.
Hellaswag (Zellers et al. 2019) to measure linguistic rea-
soning abilities, Winogrande (Sakaguchi et al. 2021), and
COPA (Gordon, Kozareva, and Roemmele 2012) to measure
commonsense reasoning abilities, and ARC-Easy (Clark
et al. 2018), ARC-Challenge (Clark et al. 2018), Piqa (Bisk
et al. 2020), MathQA (Amini et al. 2019), PubmedQA (Jin
et al. 2019) benchmarks to measure scientific reasoning abil-
ities) and 4 dialogue tasks (Wizard of Wikipedia (Dinan
et al. 2019), Empathetic Dialogues (Rashkin et al. 2019),
Blended Skill Talk (Smith et al. 2020), and Wizard of Inter-
net (Komeili, Shuster, and Weston 2022)). These test sets are
used to assess whether the general capabilities of language
models are affected after unlearning.
Evaluation Metrics.
We assess the performance of the
methods from two perspectives: the effectiveness of unlearn-
ing and the maintenance of general performance. To evaluate
the unlearning effectiveness on the forget set, we employ our
proposed metrics S-ELn and S-MA, as well as ELn and MA,
introduced in Section . As for the evaluation of maintaining
performance, we use accuracy for the classification test sets
and F1 and Perplexity (PPL) scores for dialogue tasks.
Baselines and Implementation Details
Baselines and Forgetting Threshold.
Our primary com-
parative analysis involves contrasting our SEUL with the ap-
proach proposed by Jang et al. (2023) (referred to as KUL).
We follow their specified forgetting thresholds, based on
EL10 and MA representing the extraction likelihood and
memorization accuracy of regular data. Importantly, these
thresholds serve as empirical indicators of the targeted for-
getting objectives for unlearning methods. For fair compar-
ison, the results of SEUL reported are all achieved with our
online selection method. We denote our results with offline
annotation as Oracle results and are compared in Table 4.
Models and Training Details.
The unlearning experi-
ments are conducted on the pre-trained language model
GPT-Neo series (125M, 1.3B, and 2.7B), Llama2-7B and
Mistral-7B. The learning rate for training is set to 5 × 10−5,
based on the selection from [2 × 10−5, 5 × 10−5, 1 ×
10−4]. The variable denoting the number of forgetting in-
stances, represented as d, is examined across the values
d = 1, 2, 4, 8, 16, 32, 64, 128. Unless otherwise specified,
the reported results in this paper are based on the d = 32
setting. We adapt the global batch size during training to be
the same as d, the number of forgetting instances, follow-
ing Jang et al. (2023). Each setting is run 5 times and the
reported results are the average of 5 different runs. All the
models are trained with a single Nvidia GeForce RTX 3090.
Experimental Results
Main Results
The main comparative results of unlearning methods are pre-
sented in Table 1, we have the following observations:
• Our SEUL generally exhibits superior effectiveness in
unlearning sensitive information compared to the baseline.
As explained in Experimental Setup Section, we set the for-
getting threshold consistent for both SEUL and KUL. This
can be reflected by the results reported in Table 1 that they
exhibit the same levels of EL10 and MA. In this context,
our SEUL demonstrates better scores in S-EL10 and S-MA
compared to KUL. Given that EL10 and MA reflect general
information leakage risk, and S-EL10 and S-MA scores em-
phasize the risk of sensitive information leakage, these re-
sults indicate that our SEUL unlearning method offers better
protection against sensitive information leakage.
0%
20%
40%
60%
80%
100%
0
1
2
3
4
5
6
7
8
9
S-MA
S-EL
MA
EL
Epoch
(a) Scores over Epoch
0%
20%
40%
60%
80%
100%
0
1
2
3
4
5
6
7
8
9
S-MA
S-EL
Acc
F1
Epoch
(b) Scores over Epoch
Figure 3: Unlearning GPT-Neo 1.3B: (a) S-MA, S-EL, MA,
EL, and (b) Accuracy, F1 Scores over epochs.
• Our SEUL demonstrates comparable results on classifi-
cation datasets when compared to the KUL method, but it ex-
hibits significantly better performance on dialogue datasets.
When examining the average accuracy scores on classifica-
tion tasks, we observe that SEUL maintains an accuracy dif-
ference of 0.1 −0.3% compared to the KUL method across
three different GPT-Neo language model backbones (125M,
1.3B, and 2.7B). In contrast, when considering the average
F1 and PPL scores on dialogue datasets, SEUL significantly
outperforms KUL. This improvement may stem from that
our fine-grained unlearning approach minimally impacts the
generation performance of the language model, unlike KUL
unlearning method that completely negate the loss function.
• Our unlearning target does not result in longer training
epochs. Upon examining the average number of epochs re-
quired to reach the forgetting threshold, we observe that the
average epochs for SEUL and KUL are similar. This sug-
gests that our approach, which involves partially negating
the loss function of pretraining, does not prolong the un-
learning process compared to the fully reversed loss function
employed by KUL.
• Smaller language models exhibit less stability in main-
taining performance. When examining the unlearning re-
sults across different language models, ranging from small
to large (125M to 2.7B), it becomes apparent that the 125M
GPT-Neo model experiences the most significant perfor-
mance drop when forgetting d = 32 instances.
Effectiveness Analysis of SEUL
Performance Over Epochs.
We illustrate the evolution
of scores across training epochs, focusing on Sensitive Ex-
traction Likelihood (S-EL), Sensitive Memorization Accu-
racy (S-MA), EL, and MA scores in Fig. 3(a). Additionally,
Fig. 3(b) displays the trends in S-EL, S-MA, average accu-
racy for classification datasets, and average F1 for dialogue
datasets. In Fig. 3(a), the evaluation results of unlearning,
with and without consideration of sensitive spans, demon-
strate general consistency (i.e., S-MA and S-EL compared to
MA and EL). Observing the results in Fig. 3(b), we note that
as unlearning progresses (i.e., S-MA and S-EL degrade), the
impact on both classification (Accuracy score) and dialogue
(F1 score) performance is minimal. This observation under-
scores the effectiveness of our unlearning methods.
Performance with Varying d.
The main results presented
in Table 1 are derived from a forgetting set with a size of
1
2
4
8
16
32
64
128
Forget Instance Numbers
40
41
42
43
44
Accuracy (in %) on Test Set
Accuracy
(a) Accuracy over d
1
2
4
8
16
32
64
128
Forget Instance Numbers
6
7
8
9
10
11
F1 (in %) on Test Set
F1
(b) F1 over d
Figure 4: Unlearning GPT-Neo 125M: (a) Accuracy and (b)
F1 Scores with varying d values.
Prefix
Trump is the 45th
Original president of the United States. He is the first
president to be elected in ...
KUL
Nation to Target Black Voters Ahead of Elec-
tion Day ...
SEUL
president of the United States. The current pres-
ident of the United States ...
Table 2: Given a prefix, the suffixes generated by the origi-
nal GPT-Neo 1.3B model, KUL and SEUL unlearned.
32. Here, we investigate the impact of unlearning based on
different numbers of forgetting instances. The results for un-
learning GPT-Neo 125M are shown in Fig. 4. Analyzing the
trend in classification accuracy shown in Fig. 4(a), we ob-
serve that, while the overall performance drop is not sub-
stantial, there is a noticeable degrading trend as the num-
ber of forgetting instances increases. However, when assess-
ing generation performance using the F1 score (Fig. 4(b)),
we note that as n increases, the F1 scores remain more sta-
ble than the classification performance. Fig. 4(b) highlights
the superiority of SEUL in maintaining generation perfor-
mance, consistent with the observations discussed in Main
Results Subsection based on Table 1.
Stability under Adversarial Attacks.
As illustrated in
Fig. 1, attackers may inject common sense or general knowl-
edge into their original information and subsequently re-
quest unlearning. Under such attacks, full loss negation-
based method (i.e., KUL (Jang et al. 2023)), may inadver-
tently forget the certain knowledge during unlearning. In
contrast, our method demonstrates a capacity to withstand
such attacks due to our selective unlearning. To simulate this
adversarial scenario, we add additional common knowledge
(e.g., “Trump is the 45th president of the United States.”) —
information easily generated by the original language model
— into every instances in forget dataset Df and then conduct
unlearning. We showcase the continuation results of mod-
els (with greedy decoding) when given the prefix “Trump
is the 45th” in Table 2. The results in Table 2 demonstrate
that KUL is unable to continue the commonsense sentence
after unlearning, while our SEUL method can handle such
Method
Span Annotation
NER
XXXX@XXXmail.com¿.* * Permission is hereby granted, free of charge, to any person obtaining a copy* of this software
and associated documentation files (the Software ), to deal* in the Software without restriction, including without limi-
tation the rights* to use, copy, modify, merge, publish, distribute, sublicense, and/or sell* copies of the Software , ...
Online
Se-
lection
XXXX@XXXmail.com¿.* * Permission is hereby granted, free of charge, to any person obtaining a copy* of this
software and associated documentation files (the Software), to deal* in the Software without restriction, including without
limitation the rights* to use, copy, modify, merge, publish, distribute, sublicense, and/or sell* copies of the Software, ...
Offline
An-
notation
XXXX@XXXmail.com ¿.* * Permission is hereby granted, free of charge, to any person obtaining a copy* of this software
and associated documentation files (the Software), to deal* in the Software without restriction, including without limitation
the rights* to use, copy, modify, merge, publish, distribute, sublicense, and/or sell* copies of the Software, ...
Table 3: Comparing annotated spans of NER, Online Selection, and LLM-based Offline Annotation.
Models
Forget Evaluation
Avg. 8 Cla.
Avg. 4 Dia.
Epoch
S-EL10
S-MA
Acc(↑)
F1(↑)
PPL(↓)
#
SEUL
0.3
16.1
41.0
7.8
179.8
16.2
SEUL O
0.3
14.6
41.5
7.8
102.4
13.0
Table 4: Results (in %) for SEUL (trained on online selected
data) and SEUL Oracle (trained on offline annotated data).
Method
Avg. Span #
Avg. Prop.
Cover (Human, ·)
Human
1.8
3.1%
-
NER
19.2
8.9%
5.3%
Online
8.6
13.7%
72.0%
Offline
4.5
9.5%
75.1%
Table 5: Statistics of sensitive spans annotated by various
methods. Avg. Span # refers to the average spans per in-
stance, Avg. Prop. refers to the proportion of sensitive spans
in the sequence. Cover (Human, ·) indicates the percentage
of human-annotated spans covered by other annotations.
attacks. This underscores the superiority and robustness of
our unlearning methods.
Oracle Sensitive Span Based Unlearning.
It is notewor-
thy that our approach based on online selection emphasizes
avoiding reliance on pre-determined sensitive or forgetting
spans to produce a more general and cost-efficient solution.
However, this inevitably introduces some inaccurately de-
tected spans when evaluating with S-EL and S-MA based on
ChatGPT-annotated spans. Therefore, we further explore un-
learning based on offline annotated sensitive spans (denoted
as SEULOracle) to assess the magnitude of this discrepancy.
We stop the training of SEULOracle when its S-EL10 S-MA
scores are less or equal to SEUL, and then report the corre-
sponding accuracy and F1 scores in Table 4. From the results
in Table 4, we can see that SEULOracle shows a better ca-
pability in maintaining the performance of language model,
thanks to its accurate annotation and less training epochs.
Reliability of Sensitive Span Annotation
SEUL’s training and evaluation rely on automatic online
or offline annotated sensitive spans. To ascertain the relia-
bility of the annotation, we compare our automatic annota-
tion with human annotation and Named Entity Recognition
(NER) based annotation. Specifically, we invite two annota-
tors to do sensitivity annotation on 50 instances to be forgot-
ten and then get 50 annotated instances regarded as ground
truth. We also employ NER toolkit to annotate the same 50
instances where extracted entities are regarded as sensitive
information. We have the following quantitative and quali-
tative analyses.
Quantitative Comparison.
We conduct a statistical anal-
ysis of annotated sensitive spans derived by these methods
and display results in Table 5. The results reveal that hu-
man annotators apply a more stringent criterion for sensitive
annotation, leading to a smaller average number of sensitive
spans per instance. In contrast, NER-based annotation labels
more spans, but these spans exhibit a lower overlap with the
spans annotated by human annotators. Our online sensitive
selection method tends to label more spans while offline an-
notation achieves the highest matching with human results.
Case Study.
We present a case study using the example in
Table 3 to illustrate the annotation results of NER, Online
Selection, and Offline Annotation. As observed, NER pri-
marily extracts potential entities from the sequence, without
specifically identifying sensitive information. Both Online
Selection and Offline Annotation successfully annotate sen-
sitive information, e.g. the email address (masked to avoid
privacy issues). Nevertheless, it’s important to note that On-
line Selection may introduce additional spans that do not ac-
curately represent sensitive information.
Conclusion
In summary, this paper presents SEUL, a novel selective un-
learning method for language models that minimizes neg-
ative impacts on model capabilities by focusing on specific
sequence spans. It introduces specialized evaluation metrics,
S-EL and S-MA, designed to assess the forgetting of sensi-
tive information. The paper also proposes efficient automatic
online and offline sensitive span annotation methods to sup-
port the overall unlearning framework. Overall, these contri-
butions address concerns regarding the inadvertent retention
of personal or sensitive information by neural models.
Acknowledgments
The research described in this paper is partially sup-
ported by HK RGC GRF #14206324, The National Nat-
ural Science Foundation of China (Grant No. 62227808)
and Shenzhen Science and Technology Program (Grant
No.ZDSYS20210623091809029).
References
Amini, A.; Gabriel, S.; Lin, S.; Koncel-Kedziorski, R.; Choi,
Y.; and Hajishirzi, H. 2019. MathQA: Towards Interpretable
Math Word Problem Solving with Operation-Based For-
malisms.
In Proceedings of the 2019 Conference of the
North American Chapter of the Association for Computa-
tional Linguistics: Human Language Technologies, Volume
1 (Long and Short Papers), 2357–2367. Minneapolis, Min-
nesota: Association for Computational Linguistics.
Bisk, Y.; Zellers, R.; Gao, J.; Choi, Y.; et al. 2020. Piqa: Rea-
soning about physical commonsense in natural language.
In Proceedings of the AAAI conference on artificial intel-
ligence, volume 34, 7432–7439.
Bourtoule, L.; Chandrasekaran, V.; Choquette-Choo, C. A.;
Jia, H.; Travers, A.; Zhang, B.; Lie, D.; and Papernot, N.
2021. Machine unlearning. In 2021 IEEE Symposium on
Security and Privacy (SP), 141–159. IEEE.
Cao, Y.; and Yang, J. 2015. Towards making systems for-
get with machine unlearning. In 2015 IEEE Symposium on
Security and Privacy, 463–480. IEEE.
Cettolo, M.; Niehues, J.; St¨uker, S.; Bentivogli, L.; and Fed-
erico, M. 2014. Report on the 11th IWSLT evaluation cam-
paign. In Proceedings of the 11th International Workshop
on Spoken Language Translation: Evaluation Campaign.
Clark, P.; Cowhey, I.; Etzioni, O.; Khot, T.; Sabharwal, A.;
Schoenick, C.; and Tafjord, O. 2018. Think you have Solved
Question Answering? Try ARC, the AI2 Reasoning Chal-
lenge. ArXiv, abs/1803.05457.
Dinan, E.; Roller, S.; Shuster, K.; Fan, A.; Auli, M.; and We-
ston, J. 2019. Wizard of Wikipedia: Knowledge-Powered
Conversational Agents.
In International Conference on
Learning Representations.
Fan, A.; Lewis, M.; and Dauphin, Y. 2018. Hierarchical neu-
ral story generation. arXiv preprint arXiv:1805.04833.
Ginart, A.; Guan, M.; Valiant, G.; and Zou, J. Y. 2019. Mak-
ing ai forget you: Data deletion in machine learning. Ad-
vances in neural information processing systems, 32.
Golatkar, A.; Achille, A.; and Soatto, S. 2020a. Eternal sun-
shine of the spotless net: Selective forgetting in deep net-
works.
In Proceedings of the IEEE/CVF Conference on
Computer Vision and Pattern Recognition, 9304–9312.
Golatkar, A.; Achille, A.; and Soatto, S. 2020b. Forgetting
outside the box: Scrubbing deep networks of information ac-
cessible from input-output observations. In European Con-
ference on Computer Vision, 383–398. Springer.
Gordon, A.; Kozareva, Z.; and Roemmele, M. 2012.
SemEval-2012 Task 7: Choice of Plausible Alternatives: An
Evaluation of Commonsense Causal Reasoning. In *SEM
2012: The First Joint Conference on Lexical and Computa-
tional Semantics – Volume 1: Proceedings of the main con-
ference and the shared task, and Volume 2: Proceedings of
the Sixth International Workshop on Semantic Evaluation
(SemEval 2012), 394–398. Montr´eal, Canada: Association
for Computational Linguistics.
Guo, C.; Goldstein, T.; Hannun, A.; and Van Der Maaten, L.
2019. Certified data removal from machine learning models.
arXiv preprint arXiv:1911.03030.
Jang, J.; Yoon, D.; Yang, S.; Cha, S.; Lee, M.; Logeswaran,
L.; and Seo, M. 2023. Knowledge Unlearning for Mitigat-
ing Privacy Risks in Language Models. In Proceedings of
the 61st Annual Meeting of the Association for Computa-
tional Linguistics (Volume 1: Long Papers), 14389–14408.
Association for Computational Linguistics.
Jin, Q.; Dhingra, B.; Liu, Z.; Cohen, W.; and Lu, X. 2019.
PubMedQA: A Dataset for Biomedical Research Question
Answering. In Proceedings of the 2019 Conference on Em-
pirical Methods in Natural Language Processing and the 9th
International Joint Conference on Natural Language Pro-
cessing (EMNLP-IJCNLP), 2567–2577.
Karasuyama, M.; and Takeuchi, I. 2009. Multiple incremen-
tal decremental learning of support vector machines. Ad-
vances in neural information processing systems, 22.
Koh, P. W.; and Liang, P. 2017. Understanding black-box
predictions via influence functions. In International confer-
ence on machine learning, 1885–1894. PMLR.
Komeili, M.; Shuster, K.; and Weston, J. 2022.
Internet-
Augmented Dialogue Generation.
In Proceedings of the
60th Annual Meeting of the Association for Computational
Linguistics (Volume 1: Long Papers), 8460–8478. Dublin,
Ireland: Association for Computational Linguistics.
Levy, O.; Seo, M.; Choi, E.; and Zettlemoyer, L. 2017. Zero-
shot relation extraction via reading comprehension. arXiv
preprint arXiv:1706.04115.
Ling, Z.; Fang, Y.; Li, X.; Huang, Z.; Lee, M.; Memisevic,
R.; and Su, H. 2023. Deductive Verification of Chain-of-
Thought Reasoning. arXiv preprint arXiv:2306.03872.
Liu, A.; Sap, M.; Lu, X.; Swayamdipta, S.; Bhagavatula,
C.; Smith, N. A.; and Choi, Y. 2021. DExperts: Decoding-
time controlled text generation with experts and anti-experts.
arXiv preprint arXiv:2105.03023.
Lu, X.; Welleck, S.; Hessel, J.; Jiang, L.; Qin, L.; West, P.;
Ammanabrolu, P.; and Choi, Y. 2022. Quark: Controllable
text generation with reinforced unlearning. Advances in neu-
ral information processing systems, 35: 27591–27609.
Mehta, R.; Pal, S.; Singh, V.; and Ravi, S. N. 2022. Deep Un-
learning via Randomized Conditionally Independent Hes-
sians. In Proceedings of the IEEE/CVF Conference on Com-
puter Vision and Pattern Recognition, 10422–10431.
Meng, K.; Bau, D.; Andonian, A.; and Belinkov, Y. 2022.
Locating and editing factual knowledge in gpt.
arXiv
preprint arXiv:2202.05262, 1.
Patil, V.; Hase, P.; and Bansal, M. 2023.
Can Sensi-
tive Information Be Deleted From LLMs? Objectives for
Defending Against Extraction Attacks.
arXiv preprint
arXiv:2309.17410.
Rashkin, H.; Smith, E. M.; Li, M.; and Boureau, Y.-L. 2019.
Towards Empathetic Open-domain Conversation Models: A
New Benchmark and Dataset. In Proceedings of the 57th
Annual Meeting of the Association for Computational Lin-
guistics, 5370–5381. Florence, Italy: Association for Com-
putational Linguistics.
Romero, E.; Barrio, I.; and Belanche, L. 2007. Incremental
and decremental learning for linear support vector machines.
In International Conference on Artificial Neural Networks,
209–218. Springer.
Sakaguchi, K.; Bras, R. L.; Bhagavatula, C.; and Choi, Y.
2021. Winogrande: An adversarial winograd schema chal-
lenge at scale. Communications of the ACM, 64(9): 99–106.
Shinn, N.; Cassano, F.; Gopinath, A.; Narasimhan, K. R.;
and Yao, S. 2023.
Reflexion: Language agents with ver-
bal reinforcement learning. In Thirty-seventh Conference on
Neural Information Processing Systems.
Smith, E. M.; Williamson, M.; Shuster, K.; Weston, J.; and
Boureau, Y.-L. 2020. Can You Put it All Together: Eval-
uating Conversational Agents’ Ability to Blend Skills. In
Proceedings of the 58th Annual Meeting of the Association
for Computational Linguistics, 2021–2030. Online: Associ-
ation for Computational Linguistics.
Tirumala, K.; Markosyan, A.; Zettlemoyer, L.; and Agha-
janyan, A. 2022.
Memorization without overfitting: An-
alyzing the training dynamics of large language models.
Advances in Neural Information Processing Systems, 35:
38274–38290.
Tuggener, D.; Von D¨aniken, P.; Peetz, T.; and Cieliebak, M.
2020. LEDGAR: A large-scale multi-label corpus for text
classification of legal provisions in contracts. In Proceedings
of the Twelfth Language Resources and Evaluation Confer-
ence, 1235–1241.
Wang, L.; Chen, T.; Yuan, W.; Zeng, X.; Wong, K.-F.; and
Yin, H. 2023. KGA: A General Machine Unlearning Frame-
work Based on Knowledge Gap Alignment. In Proceedings
of the 61st Annual Meeting of the Association for Computa-
tional Linguistics (Volume 1: Long Papers), 13264–13276.
Toronto, Canada: Association for Computational Linguis-
tics.
Zellers, R.; Holtzman, A.; Bisk, Y.; Farhadi, A.; and Choi,
Y. 2019. Hellaswag: Can a machine really finish your sen-
tence? arXiv preprint arXiv:1905.07830.
Zhang, S.; Dinan, E.; Urbanek, J.; Szlam, A.; Kiela, D.; and
Weston, J. 2018. Personalizing Dialogue Agents: I have a
dog, do you have pets too? In Proceedings of the 56th An-
nual Meeting of the Association for Computational Linguis-
tics (Volume 1: Long Papers), 2204–2213.