Question:
Which steps are included in the NoiserBench data construction pipeline?
A. QA Instance Generation
B. Adversarial Training
C. Entailment Verification
D. Noise Introduction

Answer: ACD

As shown in Figure 3 (A), our framework comprises four essential steps, including QA Instance Generation, Entailment Verification, Noise Introduction and Testbeds Construction.
Step 1: QA Instance Generation
For prior noise,
we collect article snippets from mainstream media and
Wikipedia, covering various time periods and domains such
as sports, politics, and finance. We then design prompts for
ChatGPT to generate relevant events, questions, and answers
for each snippet. Note that the generated questions contain
prior noise (factual errors), which we manually review to en-
sure that they are reasonably answerable by LLMs. For the
remaining six types of noise (SeN, DN, ISN, CN, SuN, ON,
PN), we obtain question-answering (QA) pairs from exist-
ing datasets, following previous work (Fang et al. 2024; Cu-
conasu et al. 2024; Yoran et al. 2024). After obtaining can-
didate QA pairs, we employ ChatGPT to remove ambiguous
or difficult-to-assess pairs, followed by a manual review. For
example, questions like “How many companies have a mar-
ket capitalization of over $25 billion and pledged to reduce
greenhouse gas emissions?” should be excluded due to their
broad potential answers and the dynamic market values of
companies. Similar criteria are applied to other instances.
Step 2: Entailment Verification As illustrated in Xie
et al. (2024); Yoran et al. (2024), effective evidence should
strongly support its answer. For example, golden evidence
about David Beckham should support the answer that he
played for Real Madrid before joining LA Galaxy. There-
fore, we use the natural language inference model bart-large-
mnli-407M (Lewis et al. 2019) to ensure evidence properly
entails the answer. Note that, we only keep those examples
with an entailment probability ≥0.8.
Step 3: Noise Introduction We construct diverse re-
trieval documents for noise testbeds. For counterfactual
noise, we extract related entities and relations from Google
search results to create counterfactual answers. ChatGPT is
then employed to construct corresponding supportive evi-
dence, followed by entailment verification. We present the
prompts in Figure 4. For Supportive and semantic noise, we
utilize the 2018 English Wikipedia dump (Karpukhin et al.
2020) as source documents, with off-the-shelf Contriever-
MS MARCO model (Izacard et al. 2022) for retrieval
and the lightweight text embedding model all-MiniLM-
L6-v2 (Wang et al. 2021) for semantic relevance filtering.
To simulate illegal sentence noise, we construct meaning-
less sentences by randomly combining words from model
vocabulary, mimicking real-world garbled text. Datatype
noise is created by prompting ChatGPT to insert URLs or
code snippets while preserving key answer information. Fi-
nally, orthographic noise is generated using the open-source
textnoisr package (Preligens Lab 2023), which enables con-
venient noise introduction. Four types of “action” are im-
plemented: insert, delete, substitute, and swap. In summary,
this pipeline enables a comprehensive assessment of model
performance across a range of noise scenarios.
Step 4: Testbeds Construction After obtaining high-
quality QA instances and diverse retrieval documents, we
build testbeds to evaluate model performance under various
noise conditions. Given the challenges in automatically as-
sessing LLM responses to open-ended QA tasks (Xie et al.
2024), we convert free-form QA into a multiple-choice for-
mat. This constrains the response space and facilitates more
accurate evaluation. Specifically, for each QA pair, LLMs
choose from 4 options: the correct answer, two counterfac-
tual alternatives, and “Uncertain”. The order of the golden
option remains entirely random to avoid LLM sensitivity to
option order (Wu et al. 2024).

Pandora’s Box or Aladdin’s Lamp: A Comprehensive Analysis Revealing the Role
of RAG Noise in Large Language Models
Jinyang Wu1, Feihu Che1, Chuyuan Zhang1, Jianhua Tao1,2,*, Shuai Zhang1, Pengpeng Shao1
1Department of Automation
2Beijing National Research Center for Information Science and Technology
Tsinghua University, Beijing, China
{wu-jy23,cyzhang24}@mails.tinghua.edu.cn, {qkr, zhang shuai, ppshao, jhtao}@tsinghua.edu.cn
Abstract
Retrieval-Augmented Generation (RAG) has emerged as a
crucial method for addressing hallucinations in large lan-
guage models (LLMs). While recent research has extended
RAG models to complex noisy scenarios, these explorations
often confine themselves to limited noise types and presup-
pose that noise is inherently detrimental to LLMs, potentially
deviating from real-world retrieval environments and restrict-
ing practical applicability. In this paper, we define seven dis-
tinct noise types from a linguistic perspective and establish
a Noise RAG Benchmark (NoiserBench), a comprehensive
evaluation framework encompassing multiple datasets and
reasoning tasks. Through empirical evaluation of eight repre-
sentative LLMs with diverse architectures and scales, we re-
veal that these noises can be further categorized into two prac-
tical groups: noise that is beneficial to LLMs (aka beneficial
noise) and noise that is harmful to LLMs (aka harmful noise).
While harmful noise generally impairs performance, benefi-
cial noise may enhance several aspects of model capabilities
and overall performance. Our analysis offers insights for de-
veloping more robust, adaptable RAG solutions and mitigat-
ing hallucinations across diverse retrieval scenarios.
Introduction
Large language models (LLMs) (OpenAI 2023; Meta, AI
2024) have demonstrated remarkable proficiency across var-
ious tasks (Bubeck et al. 2023). Despite these impres-
sive capabilities, LLMs face challenges such as reliance
on outdated knowledge and hallucination (Huang et al.
2023; Kandpal et al. 2023). Retrieval-Augmented Genera-
tion (RAG) has recently emerged as a promising approach to
mitigate these limitations (Gao et al. 2024). RAG enhances
LLM performance by augmenting inputs with additional in-
formation retrieved from external sources during inference.
However, the internet is filled with various non-standard
noises, including AI-generated fake news, outdated content,
spelling errors, and data contamination, which may poten-
tially influence model performance (Shi et al. 2023; Xie
et al. 2024). It is crucial to explore how noise affects RAG
systems and understand the underlying mechanisms.
Recently, several studies (Chen et al. 2024; Xiang et al.
2024) have attempted to extend RAG systems to complex
*Corresponding author
Copyright © 2025, Association for the Advancement of Artificial
Intelligence (www.aaai.org). All rights reserved.
Figure 1: An example from NoiserBench illustrating effects
of different RAG noises. Initially, the model is misled by
counterfactual noise. Interestingly, upon introducing benefi-
cial noise, it successfully discriminates between correct and
incorrect information and produces the accurate answer ‘D’.
real-world scenarios, investigating the impact of noisy docu-
ments and strategies to enhance the system’s robustness. For
example, Cuconasu et al. (2024) defines three types of noise
in retrieved documents and examines their impacts on LLM.
Despite highlighting one noise’s positive effect, the study
lacks a comprehensive noise definition and in-depth inves-
tigation of underlying principles. Fang et al. (2024) applies
adversarial training to dynamically adjust the model’s train-
ing process in response to retrieval noises. RobustRAG (Xi-
ang et al. 2024) proposes a defense framework to improve
the robustness of RAG models against retrieval corruption
attacks. Nevertheless, these investigations typically focus on
a limited number of noise types (usually no more than three)
and lack clear classification, which fails to fully capture the
complexity of real-world noise environments. Additionally,
these studies often assume that noise is harmful, neglecting
its potential positive effects and lacking systematic evalua-
tion datasets. As shown in Figure 1, introducing beneficial
arXiv:2408.13533v1  [cs.CL]  24 Aug 2024
noise allows the LLM to avoid the harmful effects of coun-
terfactual noise, concentrate on the golden context, and pro-
duce accurate responses. Thus, there’s an urgent need to re-
define and describe noise scenarios in RAG, and systemati-
cally explore the specific impacts of retrieval noises.
In this paper, we conduct a comprehensive analysis to re-
veal the role of RAG noises in LLMs. We first define seven
types of noise from a linguistic perspective. Based on this
definition, we propose a systematic framework to create di-
verse noisy documents and establish NoiserBench, a novel
noise RAG benchmark. Then, we evaluate eight represen-
tative LLMs with different architectures and scales. Exten-
sive results show that RAG noises can be categorized into
two practical groups: beneficial noise (semantic, datatype,
illegal sentence) and harmful noise (counterfactual, support-
ive, orthographic, prior). While harmful noise impairs per-
formance, beneficial noise surprisingly enhances model ca-
pabilities and leads to improved performance. Further analy-
sis reveals that beneficial noise facilitates more standardized
answer formats, clearer reasoning paths, and increased con-
fidence in responses with golden context. These contrasting
effects are analogous to opening Pandora’s Box (harmful
noise) versus unlocking Aladdin’s Lamp (beneficial noise).
We hope this study will advance efforts to mitigate harmful
noise and leverage the positive effects of beneficial noise in
future research. The main contributions are:
• We define seven types of noise and categorize them into
two groups: beneficial and harmful. This is the first com-
prehensive study to define and assess RAG noises from
both linguistic and practical perspectives.
• We propose a novel framework for constructing diverse
retrieval documents and create the noise RAG benchmark
(NoiserBench), which effectively simulates the impact of
real-world noise on RAG models.
• Evaluated on eight datasets and representative LLMs, our
results reveal that while some RAG noises (e.g. counter-
factual) can open Pandora’s Box and cause errors, bene-
ficial noise (e.g. datatype) has the potential to unlock the
power of Aladdin’s Lamp and deliver positive effects.
• Our findings redefine retrieval noise and encourage re-
searchers to explore methods that harness its beneficial
properties while addressing its harmful effects.
Related Work
Retrieval-Augmented Generation
By integrating external information, RAG methods enhance
reasoning and generation process (Gao et al. 2024; Zhao
et al. 2024). Early work primarily focuses on improving
retrieval model performance to obtain relevant documents
for subsequent generation (Qu et al. 2021; Wang et al. 2023;
Zheng et al. 2024). Recent research has expanded RAG
framework to real-world noisy scenarios, aiming to build
robust RAG systems by enhancing the generator (Fang
et al. 2024; Xiang et al. 2024). For instance, Self-RAG
(Asai et al. 2024) employs four specialized tokens and
GPT-4-generated instruction-tuning data to fine-tune the
Llama2 model. RetRobust (Yoran et al. 2024) introduces an
automated data generation method to fine-tune the generator
to utilize retrieved passages against noise effectively. Ro-
bustRAG (Xiang et al. 2024) proposes a defense framework
that enhances RAG model robustness against retrieval cor-
ruption attacks through an isolate-then-aggregate strategy,
achieving certifiable robustness via secure text aggregation
techniques. However, these investigations are constrained
by their narrow focus on specific noise types and the
inherent assumption that noise is harmful, potentially
hindering method generalization. This paper aims to present
a systematic analysis of RAG noise and reveal its role.
Noise Injection in LLMs
Noise injection (Grandvalet, Canu, and Boucheron 1997) in
LLMs involves adding noise to inputs during training or in-
ference, such as data augmentation (Ye et al. 2024), adver-
sarial training (Fang et al. 2024), and prompt perturbation
(Zhu et al. 2024). Recently, researchers have focused on
noise injection in RAG systems (Chen et al. 2024). For ex-
ample, Cuconasu et al. (2024) classifies three retrieval noises
and explores their effects on LLMs. Fang et al. (2024) lever-
ages adversarial training to dynamically adjust LLM’s train-
ing process in response to retrieval noises. However, these
noise types are limited and may not reflect complex real-
world scenarios. A comprehensive framework that simulates
real-world noise is necessary.
A Taxonomy of RAG Noise
As shown in Figure 2, we categorize RAG noise into seven
types from a linguistic perspective. They are further divided
into beneficial (semantic, datatype, and illegal sentence) and
harmful noise (counterfactual, supportive, orthographic, and
prior) for practical applications. We will explain the reason
behind this classification in the Experiments section.
Semantic Noise (SeN) Retrieval documents may con-
tain content with low semantic relevance to the query, of-
ten being off-topic or deviating from the intended meaning.
Given that Warren Weaver originally defined semantic noise
as ”the perturbations or distortions of sentence meaning”
(Shannon, Weaver, and Hockett 1961), we classify off-topic,
low-semantic-relevance documents as semantic noise.
Datatype Noise (DN) This type of noise refers to the
mixing of different data types on the web, such as the blend-
ing of links and text on Wikipedia. In this paper, we consider
three types of data: text, URLs, and code.
Illegal Sentence Noise (ISN) Web content may include
fragments that do not form grammatically correct sentences,
such as “history transform cover managed that hand black”.
We define this type of noise as illegal sentence noise.
Counterfactual Noise (CN) The internet contains abun-
dant false information, including fake news and outdated
knowledge (Tumarkin and Whitelaw 2001; Olan et al. 2024),
which poses significant challenges to RAG systems. Draw-
ing from linguistics, where “counterfactual” denotes state-
ments contrary to fact (Feng and Yi 2006), we introduce the
term “counterfactual noise” to characterize factual errors.
This concept aligns with prior research (Fang et al. 2024).
Figure 2: (A) Seven types of noise in RAG comprehensively reflect real-world scenarios. (B) This detailed illustration of diverse
RAG noise intuitively showcases various types. Note that significant noise injections are highlighted in red.
Supportive Noise (SuN) Supportive evidence, known as
positive evidence, is highly semantically relevant to a hy-
pothesis and provides necessary information to support it
(Kert´esz and R´akosi 2012). We introduce the term “support-
ive noise” to describe documents that exhibit high semantic
relevance but lack corresponding answer information.
Orthographic Noise (ON)
The word “orthography”
originates from the Greek orth´os (meaning “correct”) and
gr´aphein (meaning “to write”), and refers to the way words
are written in linguistics (Skeat 1993; Aloufi 2021). Ortho-
graphic noise, on the other hand, can refer to writing errors
such as spelling mistakes and word lengthening.
Prior Noise (PN) In linguistics, prior knowledge refers
to what a learner already knows before solving a problem
(Chafe 1971). Our study defines prior noise as questions
based on false assumptions or premises. For example, the
question “Who was the CEO of Google when they were re-
structured into Alphabet in 2017?” contains prior noise be-
cause the restructuring occurred in 2015, not 2017.
Noise RAG Benchmark Construction
We discuss the data construction and evaluation metrics. The
overall framework is illustrated in Figure 3.
Data Construction
As shown in Figure 3 (A), our framework comprises four es-
sential steps, including QA Instance Generation, Entailment
Verification, Noise Introduction and Testbeds Construction.
Step 1: QA Instance Generation
For prior noise,
we collect article snippets from mainstream media and
Wikipedia, covering various time periods and domains such
as sports, politics, and finance. We then design prompts for
ChatGPT to generate relevant events, questions, and answers
for each snippet. Note that the generated questions contain
prior noise (factual errors), which we manually review to en-
sure that they are reasonably answerable by LLMs. For the
remaining six types of noise (SeN, DN, ISN, CN, SuN, ON,
PN), we obtain question-answering (QA) pairs from exist-
ing datasets, following previous work (Fang et al. 2024; Cu-
conasu et al. 2024; Yoran et al. 2024). After obtaining can-
didate QA pairs, we employ ChatGPT to remove ambiguous
or difficult-to-assess pairs, followed by a manual review. For
example, questions like “How many companies have a mar-
ket capitalization of over $25 billion and pledged to reduce
greenhouse gas emissions?” should be excluded due to their
broad potential answers and the dynamic market values of
companies. Similar criteria are applied to other instances.
Step 2: Entailment Verification As illustrated in Xie
et al. (2024); Yoran et al. (2024), effective evidence should
strongly support its answer. For example, golden evidence
about David Beckham should support the answer that he
played for Real Madrid before joining LA Galaxy. There-
fore, we use the natural language inference model bart-large-
mnli-407M (Lewis et al. 2019) to ensure evidence properly
entails the answer. Note that, we only keep those examples
with an entailment probability ≥0.8.
Step 3: Noise Introduction We construct diverse re-
trieval documents for noise testbeds. For counterfactual
noise, we extract related entities and relations from Google
search results to create counterfactual answers. ChatGPT is
Figure 3: The overall framework for simulating the impact of real-world noise on RAG models. Initially, we generate and obtain
QA instances, utilizing ChatGPT to filter out ambiguous examples (Step 1). Then, we perform entailment verification using NLI
models to maintain evidence quality (Step 2). After that, we use tools like search engines to create noisy documents (Step 3).
Finally, we transform the free-form QA into a multiple-choice QA format by providing several answer options for convenient
automatic evaluation (Step 4). All experiments are conducted in a zero-shot setting to avoid bias from demonstrations.
Figure 4: Example LLM input for counterfactual evidence
generation. The context of the prompt is composed of in-
struction, examples, and candidate counterfactual QA.
then employed to construct corresponding supportive evi-
dence, followed by entailment verification. We present the
prompts in Figure 4. For Supportive and semantic noise, we
utilize the 2018 English Wikipedia dump (Karpukhin et al.
2020) as source documents, with off-the-shelf Contriever-
MS MARCO model (Izacard et al. 2022) for retrieval
and the lightweight text embedding model all-MiniLM-
L6-v2 (Wang et al. 2021) for semantic relevance filtering.
To simulate illegal sentence noise, we construct meaning-
less sentences by randomly combining words from model
vocabulary, mimicking real-world garbled text. Datatype
noise is created by prompting ChatGPT to insert URLs or
code snippets while preserving key answer information. Fi-
nally, orthographic noise is generated using the open-source
textnoisr package (Preligens Lab 2023), which enables con-
venient noise introduction. Four types of “action” are im-
plemented: insert, delete, substitute, and swap. In summary,
this pipeline enables a comprehensive assessment of model
performance across a range of noise scenarios.
Step 4: Testbeds Construction After obtaining high-
quality QA instances and diverse retrieval documents, we
build testbeds to evaluate model performance under various
noise conditions. Given the challenges in automatically as-
sessing LLM responses to open-ended QA tasks (Xie et al.
2024), we convert free-form QA into a multiple-choice for-
mat. This constrains the response space and facilitates more
accurate evaluation. Specifically, for each QA pair, LLMs
choose from 4 options: the correct answer, two counterfac-
tual alternatives, and “Uncertain”. The order of the golden
option remains entirely random to avoid LLM sensitivity to
option order (Wu et al. 2024).
Finally, eight datasets are obtained for NoiserBench. Fol-
lowing (Yoran et al. 2024; Wang et al. 2024), we randomly
select 500 samples from each dataset as test cases or use all
samples if the dataset contains fewer than 500.
Evaluation Metrics
This benchmark aims to reveal the role that RAG noise plays
on LLMs. We use accuracy as the primary metric and report
the weighted average accuracy across datasets, calculated by
aggregating accuracy for each dataset.
Table 1: Impact of diverse noise types on accuracy (%) for Llama3-8B-Instruct and Qwen2-7b-Instruct across seven datasets. We
assess performance across various retrieval scenarios: “Base” (no retrieval), “Golden Only” (only golden retrieval context), and
“Golden & XXX” (golden context + specific retrieval noises, including Counterfactual, Supportive, Orthographic, Semantic,
Datatype, Illegal Sentence Noise). The green and red values indicate the performance gap from ”Golden Only”. We also provide
the weighted average accuracy for each noise type. The best two results are shown in bold and underlined.
Llama3-8B-Instruct
Scenario
Single-hop
Multi-hop (Explicit)
Multi-hop (Implicit)
Average
NQ
RGB
HotpotQA
2WikiMQA Bamboogle
StrategyQA TempQA
Base
61.34
47.00
53.80
34.40
32.00
58.80
50.54
51.58
Golden Only
93.06
80.00
97.80
79.80
87.20
73.40
91.94
86.57
Golden & CN
58.86
36.33
44.20
21.20
61.60
43.20
67.74
45.58−40.99
Golden & SuN
90.58
80.00
95.60
81.00
93.60
69.40
93.01
85.37−1.20
Golden & ON
93.31
75.00
96.20
78.60
89.60
63.60
90.86
83.99−2.58
Golden & SeN
96.53+0.47
81.33+1.33
98.40+0.60
87.20+7.40
93.60+6.40
68.40
96.24+4.30
88.73+2.16
Golden & DN
93.19+0.13
81.67+1.67
95.00
82.00+2.20
88.00+0.80
73.60+0.20
94.62+2.68
86.91+0.34
Golden & ISN
96.65+0.65
83.00+1.33
98.80+1.00
87.40+7.60
94.40+7.20
72.60
97.85+5.91
89.89+3.32
Qwen2-7B-Instruct
Base
58.24
31.33
50.20
22.60
31.20
42.40
40.86
43.01
Golden Only
97.03
76.33
98.40
78.00
94.40
67.00
94.62
86.46
Golden & CN
41.88
26.00
38.40
12.40
39.20
37.60
45.16
33.96−52.50
Golden & SuN
90.46
74.00
96.40
80.40
92.00
64.00
90.32
83.65−2.81
Golden & ON
95.66
74.00
97.80
80.00
91.20
54.60
94.62
83.82−2.64
Golden & SeN
96.53
77.67+1.34
98.80+0.40
77.00
96.80+2.40
66.80
97.31+2.69
86.60+0.14
Golden & DN
96.03
84.33+9.00
98.20
79.60+1.60
93.60
71.80+4.80
95.70+1.08
88.11+1.65
Golden & ISN
96.65
80.00+3.67
99.00+0.60
83.80+5.80
96.80+2.40
66.80
97.85+1.23
88.11+1.65
Experiments
Experiment Setup
Datasets
We experiment with multiple QA datasets,
which are categorized into four types based on the required
reasoning skills:
• Single-hop: Questions requiring one-step reasoning. We
evaluate using the Natural Questions (NQ) (Kwiatkowski
et al. 2019) and RGB (Chen et al. 2024) datasets.
• Explicit Multi-hop: Questions where multiple reason-
ing steps are explicitly expressed. We utilize HotpotQA
(Yang et al. 2018), 2WIKIMQA (Welbl, Stenetorp, and
Riedel 2018) and Bamboogle dataset (Press et al. 2022).
• Implicit Multi-hop: Questions where intermediate steps
are not explicitly stated, often requiring commonsense
knowledge for implicit reasoning. We employ Strate-
gyQA (Geva et al. 2021) and TempQA (Jia et al. 2018).
• Mixed-Hop: Questions requiring single- or multi-hop
reasoning. We use our constructed dataset, PriorQA.
Baseline Models
We evaluate LLMs of different archi-
tectures and scales: Llama3-Instruct (8B, 70B) (Meta, AI
2024), Qwen2-7B-Instruct (Yang et al. 2024), Mistral (7B,
8x7B) (Jiang et al. 2023, 2024), Vicuna-13B-v1.5 (Chi-
ang et al. 2023), Llama2-13B (Touvron et al. 2023), and
Baichuan2-13B (Yang et al. 2023). This enables a compre-
hensive assessment of noise across various dimensions. De-
tailed descriptions of each model are provided in the official
websites or the corresponding Huggingface repository1.
Implementation Details
We execute the experiments us-
ing the following compute specifications.
• NVIDIA A100 80 GB GPU × 2
• 256 GB RAM
We use Python 3.10.0 and speed up inference using vllm2,
a fast and easy-to-use library.
Main Results
Firstly, we discuss the role of diverse RAG noises. While
prior work has studied the harmful effects of RAG noise,
we focus on beneficial noise. Specifically, after revealing
the role of noises, we evaluate the effectiveness of beneficial
noise across multiple dimensions, including model architec-
tures, scales, and RAG system designs. Then, we investigate
whether beneficial noise improves performance amidst other
noise types and verify its effectiveness statistically.
The Role of Diverse RAG Noises
Table 1 illustrates
the impact of diverse noise types (the first six) on two
state-of-the-art open-source models: Llama3-8B-Instruct
and Qwen2-7B-Instruct. We observe consistent performance
trends across multiple datasets and retrieval noises. Based
on these trends, we can categorize retrieval noises into two
1https://huggingface.co/models
2https://github.com/vllm-project/vllm
Table 2: Effects of prior noise measured by accuracy (%).
‘Base’ indicates the scenario with no retrieval. ‘Misleading’
refers to counterfactual content associated with prior noise.
‘Background’ denotes multiple retrieval results obtained af-
ter decomposing the query into its constituent entities.
Models
Base
Misleading
Background
Llama3-8B
93.40
47.80
90.00
Qwen2-7B
94.20
28.20
98.20
Mistral-7B
96.60
28.60
99.20
Llama2-13B
21.00
5.60
61.60
Vicuna-13b
91.00
25.80
99.20
Baichuan2-13b
90.00
45.20
96.40
Llama3-70b
99.00
78.40
99.80
Mixtral-8x7b
91.20
39.00
99.60
Average
79.93
34.20
88.47
Figure 5: Impact of illegal sentence noise (ISN) on the
average accuracy of eight representative LLMs on RGB.
‘Golden’, ‘ON’, ‘CN’, and ‘DN’ represent golden context
only, golden context with orthographic, counterfactual, and
datatype noise, respectively. The mean is marked by a red
solid line and the median by a purple dashed line.
types: harmful noise (counterfactual, supportive, and ortho-
graphic) and beneficial noise (semantic, datatype, and illegal
sentence). We find that:
(1) For harmful noise, counterfactual noise impacts model
performance most significantly by disrupting accurate fact
discernment and answer generation. As shown in Figure 1,
the false statement “Beckham was a prominent player for
Manchester United” leads the model to disregard correct in-
formation and respond erroneously.
(2) For beneficial noise, illegal sentence noise exhibits
the most notable improvement in model performance. It im-
proves accuracy by an average of 3.32% and 1.65% for
two models, respectively, and consistently achieves power-
ful performance across diverse datasets.
For prior noise, we assess eight LLMs on our dataset, Pri-
orQA. Questions in PriorQA contain factual errors, such as
“Which country hosted 1980 FIFA World Cup?” (1980 FIFA
World Cup was not held). Accuracy is measured by whether
LLMs correctly identify and respond with “The question is
Figure 6: Impact of three noise types on accuracy (%)
on RGB. We assess performance across various retrieval
scenarios: “Golden Only” (only golden retrieval context),
“Golden & ON” (golden context + orthographic noise), and
“Golden & SeN” (golden context + semantic noise).
factually incorrect”. As shown in Table 2, results show an
average accuracy of 79.93% across eight LLMs when han-
dling prior noise. However, when models fail to identify
prior errors and continue retrieval, performance drops signif-
icantly to 34.20%. This underscores the need to detect prior
errors in user queries before answering.
Beneficial Noise Enhances Performance Across Models
We consider both model architectures (Figure 5) and RAG
system designs (Table 3) to demonstrate the positive effects
of beneficial noise across various models. We present results
for illegal sentence noise here. Additionally, since prior re-
search has highlighted the positive effect of semantic noise
(Cuconasu et al. 2024), our subsequent discussion will focus
on two types: datatype noise and illegal sentence noise.
(1) Results across various architectures and scales As
shown in Figure 5, we evaluate the impact of illegal sen-
tence noise (ISN) on eight LLMs (different architectures
and scales) by calculating average accuracy across scenarios
with no noise, harmful noise (e.g. CN, ON), and beneficial
noise (e.g. DN). We apply proportional scaling to CN data
to make a clearer illustration within one figure while main-
taining consistent conclusions. The results indicate that ISN
significantly enhances model performance in all scenarios,
with the most substantial improvement under harmful noise.
To better illustrate the impacts of certain noise types, which
may not be immediately apparent in tabular form, we plot
their performance across multiple models using line graphs
(Figure 6) under three conditions: golden only, golden & or-
thographic noise, and golden & semantic noise. These visu-
alizations clearly demonstrate the negative effect of ortho-
graphic noise and the slight performance boost provided by
semantic noise.
(2) Noise effects on specialized RAG models As illus-
trated in Table 3, introducing illegal sentence noise to the
specialized RAG model Self-RAG (Asai et al. 2024) consis-
/ODPD%LQVWUXFW
4ZHQ%LQVWUXFW
Figure 7: Results for the impact of illegal sentence noise on the Llama3-8B-instruct and Qwen2-7B-instruct models when
exposed to five typical noise categories across four datasets, including both single-hop (S) and multi-hop (explicit: EM, implicit:
IM) reasoning tasks. The bar charts show performance differences upon introducing illegal sentence noise. The line graphs
illustrate the average accuracy improvement across noise types per dataset.
Table 3: Effects of beneficial noise on Self-RAG (13B). We
assess performance through enhanced accuracy ratios (%),
and the weighted average values (WA, %) are also provided.
Scenario
NQ
RGB
StrategyQA
WA
Golden only
+3.12
+1.74
+18.88
+7.77
Golden & DN
+1.84
+1.96
+13.50
+5.49
Golden & ON
+1.76
+3.63
+10.00
+4.67
tently enhances model performance across various datasets
(NQ, RGB, and StrategyQA) and scenarios (without noise,
with harmful and beneficial noise). This further validates the
positive effects of beneficial noise.
In conclusion, based on our comprehensive analysis, we
can classify illegal sentence noise, datatype noise, and se-
mantic noise as beneficial, while counterfactual, supportive,
and orthographic noises are categorized as harmful.
Beneficial Noise Remains Effective Under Other Noise
Disturbances
To illustrate the impact of beneficial noise
under other noise disturbances, we analyze the effect of il-
legal sentence noise (ISN) in five scenarios: no noise (i.e.,
Golden only), harmful noise (i.e., Golden & Counterfactual,
Counterfactual only and Golden & Orthographic), and ben-
eficial noise (i.e., Golden & Datatype). Figure 7 shows the
model’s accuracy gains after introducing ISN in these sce-
narios. We find that ISN generally enhances performance
across all datasets, particularly when combined with harmful
noise like counterfactual noise, with average accuracy im-
provements exceeding 10 percentage points. The consistent
positive effects of ISN in various real-world scenarios un-
derscore its potential significance for future RAG research.
Table 4: Statistical significance of differences between sce-
narios with and without beneficial noises.
Noise
Llama3-8B-Instruct
Qwen2-7B-Instruct
ISN
4.10e-5
4.88e-3
DN
1.71e-4
9.59e-4
Beneficial Noise Is Statistically Significant
To statis-
tically evaluate the differences between scenarios with
and without beneficial noise, we apply the nonparametric
Wilcoxon signed-rank test (Kotz and Johnson 1992). This
method effectively measures the magnitudes of differences
and detects statistical significance between two conditions.
We test the null hypothesis of no significant difference (H0 :
difference = 0) against the alternative hypothesis of a sig-
nificant difference (H1 : difference ̸= 0). Following Seth
et al. (2023); Wu et al. (2023), we use a significance level
of 0.05. As shown in Table 4, all p-values are below 0.05,
leading us to reject the null hypothesis (H0). These results
provide strong statistical evidence that beneficial noise im-
proves model performance.
Exploring the Mechanisms Behind Beneficial Noise
We investigate why beneficial noise positively impacts RAG
systems. We propose 3 hypotheses that beneficial noise con-
tributes to:
• H1: Clearer and more explicit reasoning process
• H2: More standardized response formats
• H3: Increased confidence with golden context
We confirm them through case study and statistical analysis.
Table 5: Examples of LLM outputs without and with beneficial noise (BN). The blue and green colors denote the correct and
incorrect responses, respectively. Upon introducing BN, LLMs exhibit clearer reasoning processes, more standardized response
formats, and increased focus on golden context.
Retrieved
Context
wo BN
[1] Paul R. Ehrlich, author of ”The Population Bomb,” was born on June 14, 1931. Ehrlich, a renowned
biologist, published his influential book on population control in 1968, which sparked . . . [Counterfactual]
[2] The author of The Population Bomb, Paul Ehrlich, was born on May 29, 1932. He is an American biologist
and educator, known for his warnings about the consequences of overpopulation . . .
[Golden Context]
. . .
Question
When was the author of The Population Bomb born?
‘A’: June 14, 1931
‘B’: Uncertain
‘C’: May 29, 1932 (Ground Truth)
‘D’: April 22, 1934
Answer
wo BN
A ###\n\n### Explanation:\n The correct answer is ‘A’: June 14, 1931. According to the information
provided, Paul R. Ehrlich, the author of “The Population Bomb” was born on June 14, 1931. The other
options are incorrect, as they provide different birth dates for the author.
é
Answer
w BN
C\n### Explanation: According to information [2], Paul Ehrlich, the author of The Population Bomb, was
born on May 29, 1932. Therefore, the correct answer is ‘C’. Information [1] provides a different birthdate,
but it is not the correct answer. The other options are not mentioned in the given information.
Ë
Figure 8: The effects of beneficial noise on LLM output un-
certainty (anti-confidence). ‘ISN’ and ‘DN’ denote Illegal
Sentence Noise and Datatype Noise, respectively. The red
star ⋆represents the mean uncertainty rate (µ).
Case Study
Table 5 presents the complete reasoning and
generation process of Llama3-8B-instruct on the multi-hop
dataset Bamboogle. When exposed to harmful noise without
any beneficial noise, the model ignores correct information
and exhibits logical flaws under the influence of counter-
factual noise influence. This is exemplified by its erroneous
statement: “The other options are incorrect, as they provide
different birth dates for the author.” However, upon intro-
ducing beneficial noise, the model exhibits heightened at-
tention to the golden context and successfully distinguishes
between correct and incorrect information (H1). We hypoth-
esize that beneficial noise enhances the LLM’s ability to in-
tegrate its parameterized knowledge with retrieved informa-
tion, thus improving its capacity to discern truth from false-
hood. Furthermore, by comparing model outputs under two
conditions, we observe that beneficial noise contributes to
more standardized answer formats (H2).
Statistical Analysis
To verify three hypotheses statisti-
cally, we use a two-step process. We first gather model out-
puts from multiple datasets before and after introducing ben-
eficial noise. Then, we randomly sample 100 examples per
dataset to manually assess which condition produces more
standardized output formats and clearer reasoning processes.
Outputs are deemed similar if no significant difference exists
between conditions with and without beneficial noise. Re-
sults across seven datasets show that, on average, 37 samples
with beneficial noise exhibit clearer reasoning compared to
31 without (H1), while 26 samples with beneficial noise
demonstrate better output formats versus 23 without (H2).
Second, as shown in Figure 8, we analyze the impact of
beneficial noise on LLM output uncertainty across four pow-
erful LLMs. The results indicate indicate that when com-
bined with beneficial noise (ISN or DN), LLMs generally
exhibit lower uncertainty and increased confidence in their
outputs. This suggests that LLMs pay more attention to the
provided golden context and respond with greater confi-
dence (H3).
Conclusion
In this paper, we provide clear definitions for seven types
of RAG noise and categorize them into two groups: benefi-
cial and harmful noise. This is the first comprehensive study
to explore retrieval noise from both linguistic and practi-
cal perspectives. To conduct this evaluation, we propose a
systematic framework for generating various retrieval docu-
ments and establish a novel noise benchmark, NoiserBench.
Evaluated on eight representative LLMs, extensive experi-
mental results reveal the role that different noise plays in
RAG systems. The most surprising finding is that beneficial
noise can act like the power of Aladdin’s Lamp and enhance
model performance by leading to clearer reasoning paths,
more standardized answers, and increased confidence. We
anticipate that future research will propose methods to fully
leverage beneficial mechanisms of noise while avoiding the
negative effects of harmful noise.
References
Aloufi, A. 2021. Language and Linguistic Orthography. En-
glish Language and Literature Studies, 11(3).
Asai, A.; Wu, Z.; Wang, Y.; Sil, A.; and Hajishirzi, H.
2024. Self-RAG: Learning to Retrieve, Generate, and Cri-
tique through Self-Reflection. In The Twelfth International
Conference on Learning Representations.
Bubeck, S.; Chandrasekaran, V.; Eldan, R.; Gehrke, J.;
Horvitz, E.; Kamar, E.; Lee, P.; Lee, Y. T.; Li, Y.; Lundberg,
S.; et al. 2023. Sparks of artificial general intelligence: Early
experiments with gpt-4. arXiv:2303.12712.
Chafe, W. L. 1971.
Linguistics and human knowledge.
Monograph series on languages and linguistics, (24): 57.
Chen, J.; Lin, H.; Han, X.; and Sun, L. 2024. Benchmark-
ing large language models in retrieval-augmented genera-
tion. In Proceedings of the AAAI Conference on Artificial
Intelligence, 17754–17762. AAAI Press.
Chiang, W.-L.; Li, Z.; Lin, Z.; Sheng, Y.; Wu, Z.; Zhang,
H.; Zheng, L.; Zhuang, S.; Zhuang, Y.; Gonzalez, J. E.; Sto-
ica, I.; and Xing, E. P. 2023.
Vicuna: An Open-Source
Chatbot Impressing GPT-4 with 90%* ChatGPT Quality.
https://lmsys.org/blog/2023-03-30-vicuna/.
Cuconasu, F.; Trappolini, G.; Siciliano, F.; Filice, S.; Cam-
pagnano, C.; Maarek, Y.; et al. 2024. The power of noise:
Redefining retrieval for rag systems. In Proceedings of the
47th International ACM SIGIR Conference on Research and
Development in Information Retrieval, 719–729. New York,
NY, USA: Association for Computing Machinery.
Fang, F.; Bai, Y.; Ni, S.; Yang, M.; Chen, X.; and Xu,
R. 2024.
Enhancing Noise Robustness of Retrieval-
Augmented Language Models with Adaptive Adversarial
Training. arXiv:2405.20978.
Feng, G.; and Yi, L. 2006. What if Chinese had linguis-
tic markers for counterfactual conditionals? Language and
thought revisited. In Proceedings of the Annual Meeting of
the Cognitive Science Society, volume 28.
Gao, Y.; Xiong, Y.; Gao, X.; Jia, K.; Pan, J.; Bi, Y.; Dai,
Y.; Sun, J.; Wang, M.; and Wang, H. 2024.
Retrieval-
Augmented Generation for Large Language Models: A Sur-
vey. arXiv:2312.10997.
Geva, M.; Khashabi, D.; Segal, E.; Khot, T.; Roth, D.;
and Berant, J. 2021.
Did aristotle use a laptop? a ques-
tion answering benchmark with implicit reasoning strate-
gies. Transactions of the Association for Computational Lin-
guistics, 9: 346–361.
Grandvalet, Y.; Canu, S.; and Boucheron, S. 1997. Noise
injection: Theoretical prospects. Neural Computation, 9(5):
1093–1108.
Huang, L.; Yu, W.; Ma, W.; Zhong, W.; Feng, Z.; et al.
2023.
A survey on hallucination in large language mod-
els: Principles, taxonomy, challenges, and open questions.
arXiv:2311.05232.
Izacard, G.; Caron, M.; Hosseini, L.; Riedel, S.; Bo-
janowski, P.; Joulin, A.; and Grave, E. 2022.
Unsuper-
vised Dense Information Retrieval with Contrastive Learn-
ing. arXiv:2112.09118.
Jia, Z.; Abujabal, A.; Saha Roy, R.; Str¨otgen, J.; and
Weikum, G. 2018. Tempquestions: A benchmark for tem-
poral question answering. In Companion Proceedings of the
The Web Conference 2018, 1057–1062. Republic and Can-
ton of Geneva, CHE: International World Wide Web Confer-
ences Steering Committee.
Jiang, A. Q.; Sablayrolles, A.; Mensch, A.; Bamford, C.;
Chaplot, D. S.; et al. 2023. Mistral 7B. arXiv:2310.06825.
Jiang, A. Q.; Sablayrolles, A.; Roux, A.; Mensch, A.; et al.
2024. Mixtral of Experts. arXiv:2401.04088.
Kandpal, N.; Deng, H.; Roberts, A.; Wallace, E.; and Raffel,
C. 2023. Large language models struggle to learn long-tail
knowledge. In International Conference on Machine Learn-
ing, 15696–15707. PMLR.
Karpukhin, V.; O˘guz, B.; Min, S.; Lewis, P.; Wu, L.; Edunov,
S.; Chen, D.; and tau Yih, W. 2020. Dense Passage Retrieval
for Open-Domain Question Answering. arXiv:2004.04906.
Kert´esz, A.; and R´akosi, C. 2012.
Data and evidence in
linguistics: A plausible argumentation model. Cambridge
University Press.
Kotz, S.; and Johnson, N. L., eds. 1992. Breakthroughs in
Statistics: Methodology and Distribution. New York, NY:
Springer New York.
Kwiatkowski, T.; Palomaki, J.; Redfield, O.; Collins, M.;
Parikh, A.; Alberti, C.; Epstein, D.; Polosukhin, I.; Devlin,
J.; Lee, K.; et al. 2019. Natural questions: a benchmark for
question answering research. Transactions of the Associa-
tion for Computational Linguistics, 7: 453–466.
Lewis, M.; Liu, Y.; Goyal, N.; Ghazvininejad, M.; Mo-
hamed, A.; et al. 2019.
BART: Denoising Sequence-to-
Sequence Pre-training for Natural Language Generation,
Translation, and Comprehension. arXiv:1910.13461.
Meta, AI. 2024. Introducing meta llama 3: The most capable
openly available llm to date. https://ai.meta.com/blog/meta-
llama-3/.
Olan, F.; Jayawickrama, U.; Arakpogun, E. O.; Suklan, J.;
and Liu, S. 2024. Fake news on social media: the impact on
society. Information Systems Frontiers, 26(2): 443–458.
OpenAI. 2023. Introducing ChatGPT. https://openai.com/
index/chatgpt/.
Preligens Lab. 2023. Textnoisr: Adding random noise to a
dataset. https://github.com/preligens-lab/textnoisr.
Press, O.; Zhang, M.; Min, S.; Schmidt, L.; Smith, N. A.;
and Lewis, M. 2022. Measuring and narrowing the com-
positionality gap in language models.
arXiv preprint
arXiv:2210.03350.
Qu, Y.; Ding, Y.; Liu, J.; Liu, K.; Ren, R.; et al. 2021. Rock-
etQA: An Optimized Training Approach to Dense Passage
Retrieval for Open-Domain Question Answering. In Pro-
ceedings of the 2021 Conference of the North American
Chapter of the Association for Computational Linguistics:
Human Language Technologies, 5835–5847. Online: Asso-
ciation for Computational Linguistics.
Seth, I.; Lim, B.; Xie, Y.; Cevik, J.; Rozen, W. M.; Ross,
R. J.; and Lee, M. 2023. Comparing the efficacy of large
language models ChatGPT, BARD, and Bing AI in provid-
ing information on rhinoplasty: an observational study. Aes-
thetic Surgery Journal Open Forum, 5: ojad084.
Shannon, C.; Weaver, W.; and Hockett, C. 1961. The math-
ematical theory of communication. Urbana: University of
Illinois.
Shi, F.; Chen, X.; Misra, K.; Scales, N.; Dohan, D.; Chi,
E. H.; Sch¨arli, N.; et al. 2023. Large language models can
be easily distracted by irrelevant context. In International
Conference on Machine Learning, 31210–31227. PMLR.
Skeat, W. W. 1993. The concise dictionary of English ety-
mology. Wordsworth Editions.
Touvron, H.; Martin, L.; Stone, K.; Albert, P.; Almahairi,
A.; Babaei, Y.; et al. 2023. Llama 2: Open Foundation and
Fine-Tuned Chat Models. arXiv:2307.09288.
Tumarkin, R.; and Whitelaw, R. F. 2001. News or noise? In-
ternet postings and stock prices. Financial Analysts Journal,
57(3): 41–51.
Wang, W.; Bao, H.; Huang, S.; Dong, L.; and Wei,
F. 2021.
MiniLMv2: Multi-Head Self-Attention Rela-
tion Distillation for Compressing Pretrained Transformers.
arXiv:2012.15828.
Wang, Y.; Li, P.; Sun, M.; and Liu, Y. 2023. Self-Knowledge
Guided Retrieval Augmentation for Large Language Mod-
els. arXiv:2310.05002.
Wang, Z.; Liu, A.; Lin, H.; Li, J.; Ma, X.; and Liang,
Y. 2024.
RAT: Retrieval Augmented Thoughts Elicit
Context-Aware Reasoning in Long-Horizon Generation.
arXiv:2403.05313.
Welbl, J.; Stenetorp, P.; and Riedel, S. 2018. Constructing
datasets for multi-hop reading comprehension across docu-
ments. Transactions of the Association for Computational
Linguistics, 6: 287–302.
Wu, J.; Che, F.; Zheng, X.; Zhang, S.; Jin, R.; Nie, S.;
Shao, P.; and Tao, J. 2024.
Can large language mod-
els understand uncommon meanings of common words?
arXiv:2405.05741.
Wu, J.; Ning, Z.; Ding, Y.; Wang, Y.; Peng, Q.; and Fu,
L. 2023. KGETCDA: an efficient representation learning
framework based on knowledge graph encoder from trans-
former for predicting circRNA-disease associations. Brief-
ings in Bioinformatics, 24(5): bbad292.
Xiang, C.; Wu, T.; Zhong, Z.; Wagner, D.; Chen, D.; and
Mittal, P. 2024. Certifiably Robust RAG against Retrieval
Corruption. arXiv:2405.15556.
Xie, J.; Zhang, K.; Chen, J.; Lou, R.; and Su, Y. 2024.
Adaptive Chameleon or Stubborn Sloth: Revealing the Be-
havior of Large Language Models in Knowledge Conflicts.
arXiv:2305.13300.
Yang, A.; Xiao, B.; Wang, B.; Zhang, B.; Bian, C.; Yin, C.;
Lv, C.; Pan, D.; et al. 2023. Baichuan 2: Open Large-scale
Language Models. arXiv:2309.10305.
Yang, A.; Yang, B.; Hui, B.; Zheng, B.; Yu, B.; Zhou, C.;
et al. 2024. Qwen2 Technical Report. arXiv:2407.10671.
Yang, Z.; Qi, P.; Zhang, S.; Bengio, Y.; Cohen, W. W.;
Salakhutdinov, R.; and Manning, C. D. 2018. HotpotQA: A
dataset for diverse, explainable multi-hop question answer-
ing. arXiv:1809.09600.
Ye, J.; Xu, N.; Wang, Y.; Zhou, J.; Zhang, Q.; Gui, T.; and
Huang, X. 2024. LLM-DA: Data Augmentation via Large
Language Models for Few-Shot Named Entity Recognition.
arXiv:2402.14568.
Yoran, O.; Wolfson, T.; Ram, O.; and Berant, J. 2024. Mak-
ing Retrieval-Augmented Language Models Robust to Irrel-
evant Context. In The Twelfth International Conference on
Learning Representations.
Zhao, P.; Zhang, H.; Yu, Q.; Wang, Z.; Geng, Y.; Fu, F.;
Yang, L.; Zhang, W.; Jiang, J.; and Cui, B. 2024. Retrieval-
Augmented Generation for AI-Generated Content: A Sur-
vey. arXiv:2402.19473.
Zheng, X.; Che, F.; Wu, J.; Zhang, S.; Nie, S.; Liu, K.; and
Tao, J. 2024. KS-LLM: Knowledge Selection of Large Lan-
guage Models with Evidence Document for Question An-
swering. arXiv:2404.15660.
Zhu, K.; Wang, J.; Zhou, J.; Wang, Z.; Chen, H.; Wang, Y.;
Yang, L.; Ye, W.; et al. 2024. PromptRobust: Towards Eval-
uating the Robustness of Large Language Models on Adver-
sarial Prompts. arXiv:2306.04528.


Question:
According to the analysis in the paper,which of the following statements accurately describe the principles and mechanisms underlying the proposed CoBa method?
A. CoBa identifies potential hallucinations during decoding by monitoring token-level conditional probabilities and embedding distances between generated and context tokens.
B. Once a hallucination is detected, CoBa performs local editing through backtracking to regenerate alternative tokens that avoid low-probability continuations.
C. CoBa requires additional model training and fine-tuning to adjust the parameters of the summarization model toward more faithful text generation.
D. CoBa operates entirely at inference time and can be combined with existing decoding strategies such as greedy decoding or nucleus sampling to improve factual consistency.

Answer: ABD

Correction with Backtracking Reduces Hallucination in Summarization
Zhenzhen Liu∗
Chao Wan
Varsha Kishore
Jin Peng Zhou
Cornell University
{zl535, cw862, vk352, jz563}@cornell.edu
Minmin Chen
Google DeepMind
minminc@google.com
Kilian Q. Weinberger
Cornell University
kqw4@cornell.edu
Abstract
Abstractive summarization aims at generating
natural language summaries of a source doc-
ument that are succinct while preserving the
important elements. Despite recent advances,
neural text summarization models are known
to be susceptible to hallucinating (or more cor-
rectly confabulating), that is to produce sum-
maries with details that are not grounded in
the source document. In this paper, we intro-
duce a simple yet efficient technique, CoBa, to
reduce hallucination in abstractive summariza-
tion. The approach is based on two steps: hal-
lucination detection and mitigation. We show
that the former can be achieved through mea-
suring simple statistics about conditional word
probabilities and distance to context words.
Further, we demonstrate that straight-forward
backtracking is surprisingly effective at miti-
gation. We thoroughly evaluate the proposed
method with prior art on three benchmark
datasets for text summarization. The results
show that CoBa is effective and efficient in
reducing hallucination, and offers great adapt-
ability and flexibility. Code can be found at
https://github.com/zhenzhel/CoBa.
1
Introduction
Recent summarization methods, based on neural
sequence-to-sequence and language models (LM),
are able to produce high-quality summaries (Zhang
et al., 2020; Chung et al., 2022; Touvron et al.,
2023a). However, despite their impressive capa-
bilities these summarization models are prone to
hallucinations, a phenomenon where models make
statements that seem plausible but are not grounded
in the source document (Pagnoni et al., 2021a;
Maynez et al., 2020a; Zhao et al., 2020). Hallucina-
tions compromise the accuracy and trustworthiness
of the generated summaries.
∗Part of the work is conducted during an internship at
Google.
We hypothesize that one reason for hallucination
is that sometimes after a LM generates partial
text, there is no completion that is grounded in
the source text. An illustration of this situation is
shown in Figure 1. Although the partial sentence
I live in is highly plausible, it forces the LM to
specify where the person lives, even though this is
not specified in the source document. Such situa-
tions can often be detected by intrinsic properties
of hallucinated text: (1) the first word of a hal-
lucinated sequence tends to have low conditional
probability, (2) hallucinations are not supported by
words in the context, and therefore have a large
distance to context words. Returning to our previ-
ous example, if the language model continues the
sentence I live in without any support from the con-
text, Munich might be just as plausible as New York,
or Penn State. None of the locations would have
particularly high probability, therefore triggering
condition (1). Further, if none of the cities are men-
tioned in the context, all would have a large word
distances to the context words, triggering condition
(2). Once the beginning of a hallucination is de-
tected, we backtrack and re-generate the preceding
words that “cornered” the LM into a position with-
out a faithful continuation. In our example, we re-
place the token in by the token with; consequently,
based on the context, the generated sentence can
be completed with my dog.
Our method Correction with Backtracking (CoBa),
is a simple inference-time method that requires
no additional model training and is compatible
with most of the decoding methods. We evalu-
ate CoBa on three established document summa-
rization datasets and measure the faithfulness of
generated summaries. We show that it is highly
effective and efficient for detecting and mitigating
hallucinations. CoBa is also orthogonal to many
existing hallucination reduction techniques and can
be used in conjunction to those.
1
arXiv:2310.16176v3  [cs.CL]  3 Sep 2024
I share my home with a loyal and
affectionate companion - my dog. Living
together has brought joy, companionship,
and a unique bond into my life. Each day
is marked by our shared adventures,
whether it’s going for long walks,
playing fetch in the park, or simply
enjoying quiet moments at home. Their
unwavering presence brings comfort and
a sense of connection, making every day
brighter and more fulfilling.
Context Document
Greedy Decoding: I live in New York . ✗
Greedy+Backtrack: I live with my dog. ✓
Summary
I
live
in
New
Penn
with
my
dog
0.70
0.55
0.15
0.08
0.35
0.40
0.13
…
 All next tokens are
low probability
➞ Backtrack
cat
a
0.72
0.25
York
State
Port
0.80
0.15
0.50
…
Figure 1: Schematic illustration of CoBa (using only
token probability as the detection metric with threshold
0.2). After the partial summary “I live”, the token “in”
has a higher probability than “with". However, “I live in”
will pressure the model into hallucinating a place. We
detect this because all the next tokens have a probability
lower than our threshold 0.2. Backtracking enables the
model to find an alternative continuation that avoids
hallucination down the line.
2
Background and Related Work
We adopt the definition of hallucination for abstrac-
tive summarization from Maynez et al. (2020a):
The summary S for a context document C contains
hallucinations if there exists a span in S which is
not supported by C.
Hallucinations exhibit task-specific characteristics
in various Natural Language Generation (NLG)
tasks. For instance, in Machine Translation, hallu-
cination is often observed in the output when the
input source undergoes specific perturbation (Lee
et al., 2018). In Question Answering (QA), one
common manifestation is semantic drift, where the
generated answers deviate from the topic of the
question (Li et al., 2021). Additionally, in retrieval-
based QA, the retrieval model may introduce addi-
tional sources of hallucination (Ji et al., 2023).
Various existing works seek to understand how hal-
lucination happens, and have identified several fac-
tors. In various datasets, human generated ground
truth summaries can contain additional information
not present in the corresponding input texts (Dhin-
gra et al., 2019; Wang et al., 2020). Training on
such data may increase a model’s tendency to hallu-
cinate. During generation, hallucination may occur
when the model attends to irrelevant parts of the
input context (Tian et al., 2019), or utilizes knowl-
edge acquired during training that is not grounded
in the context (Longpre et al., 2021). Addition-
ally, the decoding method also impacts the faith-
fulness of generation. Past work has observed that
sampling-based decoding can lead to increased hal-
lucination (Dziri et al., 2021; Lee et al., 2022; Wan
et al., 2023).
2.1
Methods for Reducing Hallucination
Depending on the task and problem setup, various
methods have been developed to detect and mit-
igate hallucinations. Existing approaches can be
broadly categorized into training time mitigation
and generation time mitigation.
Training Time Mitigation.
Noise in the pre-
training corpus is shown to be a significant source
of hallucination for language models (Zhou et al.,
2023). Some past work has focused on applying
simple mechanisms to filter training data, many of
which are already used in training large language
models (Touvron et al., 2023b; Penedo et al., 2023;
Li et al., 2023b). Data curation is not only done
in the pre-training stage but also can happen dur-
ing supervised finetuning (SFT). Researches in this
area focus on using high-quality, human curated,
or domain-specific data (Elaraby et al., 2023) for
SFT and have shown that this can lead to improved
faithfulness (Zhou et al., 2023; Chen et al., 2023;
Lee et al., 2023; Cao et al., 2023).
Generation Time Mitigation.
Recent publica-
tions have also explored how to enhance the faith-
fulness of generation during inference time (Zhang
et al., 2023). One line of work performs post-
2
editing by training specialized models (Cao et al.,
2020; Chen et al., 2021; Dong et al., 2020) or by di-
rectly prompting the models (Varshney et al., 2023;
Mündler et al., 2023). Others modify the decoding
algorithm. Lee et al. (2022) proposes to gradu-
ally decrease the value of p in top-p sampling (i.e.
nucleus sampling), to reduce hallucinations intro-
duced by randomness. Li et al. (2023a) modifies
attention to encourage more factual generations.
Shi et al. (2023) proposes Context-Aware Decoding
(CAD) to suppress hallucinations arising from the
model’s prior knowledge; they adjust the context-
conditional token logits with the unconditional log-
its. Wan et al. (2023) proposes Lookahead: At each
decoding step, it rolls out future summaries for the
top k tokens with the highest probabilities, adjusts
their probabilities with BS-Fact, and picks the to-
ken with the highest adjusted probability. They also
show that the performance can be further improved
by ranking multiple candidates with a composite
faithfulness score, or by distilling student models
with the generated summaries. In contrast to these
methods, CoBa does not tamper with token proba-
bilities. Instead, it detects hallucinated tokens and
fixes them through backtracking and local edits
(see Figure 1).
Most similar to our work is arguably King et al.
(2022), a publication that we were not aware of
until after the completion of this paper. While we
do have distinct design choices and evaluations,
we acknowledge that the two methods are rather
similar and expect them to perform similarly under
our setting.
3
Problem Setup
Let Mθ be an autoregressive summarization model
with parameters θ, and let Σ be its vocabulary.
Given a context document C = (c1, · · · cm) as in-
put, Mθ produces a summary S = (s1, · · · , sn):
Mθ(C) = S
where c1, · · · cm, s1, · · · , sn ∈Σ; m and n are the
lengths of the context and the summary respec-
tively. In practice, Mθ can either be a special-
ized summarization model like PEGASUS (Zhang
et al., 2020), or a general language model capable
of zero-shot summarization like Flan-T5 (Chung
et al., 2022). If Mθ requires prompting, we add a
prompt like “summarize: " to the context as input.
4
3
2
1
0
1
2
3
4
Token Offset
0.0
0.2
0.4
0.6
0.8
1.0
Token Probability
Start of Hallucination
Token Probability Around Hallucination Span
4
3
2
1
0
1
2
3
4
Token Offset
0.0
0.2
0.4
0.6
0.8
1.0
Token Distance
Start of Hallucination
Token-to-Context Distance Around Hallucination Span
Figure 2: Average token probability (top) and token-to-
context distance (bottom) around the hallucination span.
Token offset 0 stands for the token where hallucination starts,
negative offsets stand for the tokens before hallucination and
positive ones are for the hallucinated tokens. On average, the
token which starts the hallucination has the lowest probability
and is the furthest away from the context tokens compared to
surrounding ones.
Model Mθ generates the summary autoregres-
sively. At each step, given a partially generated
summary S<t up to token st−1, it outputs a distri-
bution pθ(st|C, S<t) for the next token st over the
vocabulary Σ. The probability of generating the
summary S is thus
p(S) =
|S|
Y
t=1
pθ(st|C, S<t)
4
Reducing Hallucination at Inference
We present a detection-correction approach for re-
ducing hallucination at decoding time. The main
idea is illustrated in Figure 1: If a hallucination
occurs, the problem typically originates already
with its preceding tokens. The partially decoded
summary can “corner" the model such that there is
no faithful next token. For example, in Figure 1,
the natural continuation for the partial summary “I
3
live in" is a name of a place. The source context
however does not mention any places. We design
strategies to detect such occurrences, and use back-
tracking (Tarjan, 1972) to find alternative phrases
that prevent hallucinations down the line.
4.1
Hallucination Detection
We investigate different properties of hallucinated
text and devise two strategies for detecting text that
is not grounded in the context.
4.1.1
Uncertainty-based Detection
The intuition behind uncertainty-based detection is
that hallucination is likely to occur if the model is
unsure about what it should generate next condi-
tioning on the input. The conditional probability of
a token is one way of measuring uncertainty and
prior work has shown that the token-wise proba-
bility of autoregressive language models is well-
calibrated (Kadavath et al., 2022). Petryk et al.
(2023) also use a similar technique for evaluating
and ranking the correctness of image captions.
We validate that token probabilities are effective
for identifying hallucinated tokens in summaries
by computing probabilities on an annotated halluci-
nation dataset from Maynez et al. (2020b). The
dataset contains generated summaries from dif-
ferent summarization models, such as finetuned
BERT (Devlin et al., 2018), Pointer-Generator
Model (See et al., 2017) and several more, with
human annotations for hallucination spans. Fig-
ure 2 presents the conditional token probabilities
of Flan-T5 XL around the hallucination span. Off-
set 0 represents where the hallucination starts, the
negative offsets represent preceding tokens and the
positive offsets represent successive tokens. In the
figure, we observe a significant drop in token con-
fidence at the start of hallucination. The average
probability is only 0.2 in contrast with 0.5-0.6 for
non-hallucinated tokens. The distribution of the
probabilities is noisy shown as wide standard devi-
ation in the figure, because of annotation noise and
because some generated summaries can contain
unnatural segments.
Therefore, measuring conditional token probability
is one way of detecting the beginning of hallucina-
tions during the decoding process, when all possi-
ble next tokens have low probability, it suggests the
absence of a suitable candidate, and potentially sig-
nals the onset of hallucination. Formally, at step t,
we flag the token if the following condition holds:
pθ(st|C, S<t) < δ
where C is the context document, S<t is the par-
tially generated summary, and δ is the token level
conditional probability threshold for hallucination.
4.1.2
Similarity-based Detection
Another intuitive way of detecting hallucination is
to find tokens in the generated summary that are not
supported by the context, i.e., tokens that are not
“close" to any part of the context document. One
method of measuring closeness is by computing co-
sine distance in the embedding space of a language
model. More concretely, given a proposed token,
we compute the distance between its embedding
and the embeddings of all tokens in the context
and flag the token as a potential hallucination if
the minimum distance is above a certain threshold.
The detection criterion in this case is:
d(v, C) = min
ci∈C cos_dist
Emb(v), Emb(ci)

> φ
where v is the proposed token, C is the context doc-
ument and φ is the distance threshold. Figure 2
presents the minimum token-to-context distance
computed over the annotated dataset from Maynez
et al. (2020b)’s with embeddings from Flan-t5 XL
(the results are averaged over 5000 samples). The
average token distance at the first word in a hallu-
cination span is significantly higher than words at
other positions, as expected.
4.2
Hallucination Mitigation
After detecting potential hallucination during de-
coding using the techniques described in subsec-
tion 4.1, we perform a local intervention to prevent
the generation of hallucinated phrases. Specifically,
we introduce a process similar to depth first search.
We eliminate the last generated token st and try to
propose an alternative token s′
t that does not sat-
isfy the hallucination criteria. We keep track of
the eliminations given a partial sequence S<t and
context C to avoid repetitive proposals. If s′
t can be
found we add it to the generation and continue the
forward decoding. We also continue if the partial
sequence S<t only contains the start-of-sequence
token [SOS]. Otherwise, we backtrack again, i.e.
eliminate the current last token st−1 and repeat the
process (see Figure 1 for a pictorial description).
4
Admittedly, sometimes the model is unable to find a
good solution, and this is signaled by backtracking
too many times. We therefore introduce an upper
bound L, for the number of decoding steps (both
forward and backtracking) that can be performed.
We pick L = 10T where T is the maximum gen-
eration length for our model Mθ. If an acceptable
summary cannot be generated in L steps, we turn
off the backtracking mechanism and adopt greedy
decoding to generate the summary. We empirically
observe that with reasonable threshold choices, less
than 3% of the generations exceed the upper bound
L when using moderate threshold values in general.
5
Experiments
5.1
Datasets and Models
We consider two models: Flan-T5 XL (Chung et al.,
2022) and LLaMA (Touvron et al., 2023a). We use
the pretrained models without any further finetun-
ing on individual datasets.
We consider three datasets: Newsroom (Grusky
et al., 2018), CNN/Dailymail (Nallapati et al.,
2016), and XSUM (Narayan et al., 2018). We re-
port numbers on the full test set of CNN/Dailymail
and XSUM, and randomly sample a subset of size
5000 from the Newsroom test set. The XSUM
dataset uses the first sentence of the original article
as the ground truth summary, and the rest of the ar-
ticle as the context document. Consequently, core
information is sometimes missing from the context.
To improve the completeness of the context and en-
able more meaningful comparison with the ground
truth, we adopt a similar approach as Wang et al.
(2020) and prepend the ground truth summary back
to the articles before performing summarization.
5.2
Baselines and Implementation Details
We examine four baseline decoding methods:
greedy decoding,
nucleus sampling,
Looka-
head (Wan et al., 2023) (see section 2), and
CAD (Shi et al., 2023).
Note that Lookahead
takes a long time to roll out future summaries and
compute BS-Fact for each of the rollouts (for in-
stance, generating 5000 Newsroom samples takes
108 hours). One natural way of increasing the
speeding of this method is to perform “lookahead"
once every l tokens instead of after every token.
Thus, we consider four choices of l for Lookahead:
l = 1 (the original version), l = 2, l = 4 and l = 8.
Additional implementation details can be found in
subsection A.1 in the Appendix.
We consider two versions of CoBa: (1) CoBa that
only uses the conditional word probabilities for de-
tection, which we refer as CoBa in the tables; (2)
CoBa that uses both the conditional word probabil-
ity and the token-context distance, which we refer
as CoBa-d. We use probability threshold δ = 0.2
and distance threshold φ = 0.5 for Flan-T5, and
δ = 0.3 and φ = 0.9 for LLaMA.
We evaluate CoBa’s performance with greedy de-
coding and nucleus sampling. Since CoBa is com-
plementary to most decoding methods, we can also
use CoBa in conjunction with some of the baselines.
We report results of using CoBa and CAD together.
We do not evaluate using CoBa with Lookahead
due to the high computational cost.
5.3
Metrics
To evaluate faithfulness, we compare the gen-
erated summaries with their source documents.
We use the following metrics: AlignScore (Zha
et al., 2023) and FactCC (Kry´sci´nski et al.,
2019), both of which employ learned models to
score faithfulness; BS-Fact, which measures the
BERTScore (Zhang et al., 2019) precision of a gen-
erated summary with respect to its context docu-
ment; ROUGE-L (Lin, 2004), which measures the
longest common subsequence between the genera-
tion and reference. These metrics align relatively
well with human judgement (Pagnoni et al., 2021b)
and have reasonable runtime.
We also report standard summarization met-
rics, including ROUGE-L, BERTScore F1 and
Bleurt (Sellam et al., 2020), computed between
the generated summaries and the datasets’ ground
truth summaries. It should be noted that the mod-
els are used in a zero-shot manner. The quality of
the generated summaries depends on the model’s
capabilities, and they may have different styles
compared to the ground truth. Therefore, this com-
parison may not always yield informative results.
5.4
Results
We report the faithfulness performance of Flan-
T5 on the different datasets in Table 1, and the
performance of LLaMA in Table 2. Note that all
metrics are computed between the source docu-
ment and the generated summary. We report the
5
Table 1: Faithfulness of the summaries generated with various decoding methods using Flan-T5. All the
metrics are computed between the context document and the generated summary; higher is better.
Method
AlignScore↑
FactCC↑
BS-Fact↑
Rouge-L↑
Newsroom
Greedy
0.765
0.604
0.919
0.131
+ Lookahead (every 8 tok.)
0.768
0.607
0.920
0.133
+ Lookahead (every 4 tok.)
0.774
0.607
0.922
0.136
+ Lookahead (every 2 tok.)
0.811
0.662
0.931
0.153
+ Lookahead (every tok.)
0.816
0.662
0.933
0.159
+ CAD
0.746
0.490
0.916
0.145
+ CoBa
0.821
0.674
0.923
0.138
+ CoBa-d
0.865
0.709
0.926
0.145
+ CoBa + CAD
0.773
0.515
0.919
0.149
+ CoBa-d + CAD
0.820
0.560
0.922
0.161
Nucleus
0.636
0.482
0.902
0.101
+ CAD
0.694
0.430
0.907
0.117
+ CoBa
0.800
0.645
0.920
0.128
+ CoBa-d
0.857
0.692
0.923
0.139
+ CoBa + CAD
0.767
0.505
0.917
0.139
+ CoBa-d + CAD
0.817
0.552
0.921
0.154
XSUM
Greedy
0.723
0.485
0.919
0.096
+ Lookahead (every 8 tok.)
0.727
0.486
0.919
0.096
+ Lookahead (every 4 tok.)
0.733
0.487
0.920
0.097
+ Lookahead (every 2 tok.)
0.756
0.514
0.925
0.101
+ Lookahead (every tok.)
0.767
0.524
0.926
0.102
+ CAD
0.694
0.383
0.919
0.094
+ CoBa
0.752
0.504
0.920
0.096
+ CoBa-d
0.791
0.523
0.921
0.104
+ CoBa + CAD
0.707
0.398
0.919
0.094
+ CoBa-d + CAD
0.735
0.414
0.923
0.103
Nucleus
0.545
0.364
0.902
0.082
+ CAD
0.621
0.317
0.911
0.088
+ CoBa
0.730
0.489
0.917
0.093
+ CoBa-d
0.772
0.499
0.920
0.101
+ CoBa + CAD
0.695
0.373
0.918
0.093
+ CoBa-d + CAD
0.728
0.392
0.922
0.102
CNN/DM
Greedy
0.840
0.506
0.922
0.146
+ Lookahead (every 8 tok.)
0.843
0.511
0.923
0.147
+ Lookahead (every 4 tok.)
0.848
0.514
0.925
0.149
+ Lookahead (every 2 tok.)
0.866
0.546
0.930
0.157
+ Lookahead (every tok.)
0.874
0.561
0.932
0.162
+ CAD
0.828
0.301
0.917
0.173
+ CoBa
0.869
0.554
0.924
0.149
+ CoBa-d
0.884
0.570
0.925
0.151
+ CoBa + CAD
0.836
0.312
0.918
0.174
+ CoBa-d + CAD
0.849
0.330
0.919
0.178
Nucleus
0.706
0.310
0.907
0.122
+ CAD
0.777
0.232
0.911
0.157
+ CoBa
0.857
0.521
0.922
0.142
+ CoBa-d
0.872
0.533
0.923
0.145
+ CoBa + CAD
0.828
0.291
0.916
0.169
+ CoBa-d + CAD
0.841
0.313
0.918
0.174
metrics between the generated and ground truth
summaries in Table 4 and Table 5 in the Appendix.
For Flan-T5, both Greedy with CoBa and Looka-
head at every token are competitive across datasets
and metrics. Lookahead is slightly better accord-
ing to BS-Fact and ROUGE-L, but is significantly
slower as seen in Figure 3. Greedy with CoBa is
comparable to Lookahead every 4 tokens and is
still much faster. For LLaMA, CoBa also attains
performance gain. The improvement is smaller as
LLaMA produces more faithful summaries than
Flan-T5. It is important to note that the absolute
6
Table 2: Faithfulness of the summaries generated with various decoding methods using LLaMA. All the
metrics are computed between the context document and the generated summary; higher is better.
Method
AlignScore↑
FactCC↑
BS-Fact↑
Rouge-L↑
Newsroom
Greedy
0.701
0.321
0.897
0.161
+ CAD
0.706
0.247
0.910
0.170
+ CoBa
0.715
0.328
0.906
0.162
+ CoBa-d
0.729
0.335
0.906
0.164
XSUM
Greedy
0.798
0.406
0.931
0.221
+ CAD
0.783
0.335
0.931
0.237
+ CoBa
0.800
0.410
0.932
0.221
+ CoBa-d
0.805
0.418
0.933
0.223
CNN/DM
Greedy
0.750
0.316
0.900
0.152
+ CAD
0.740
0.251
0.919
0.176
+ CoBa
0.753
0.323
0.902
0.153
+ CoBa-d
0.759
0.327
0.902
0.154
values of FactCC is smaller for LLaMA, because
LLaMA produces much longer summaries than
Flan-T5, while FactCC has negative correlation
with the summary length. We report the distribu-
tion of generated summary length in Figure 6 in
the Appendix, to show that the performance gain is
not caused by producing shorter summaries.
In Figure 5, we present two qualitative examples
comparing greedy decoding vs. CoBa and CoBa-
d. In the first example, the greedy decoding pro-
duces the summary "The Boston Globe’s review of
"Looper" by John Sutter." with a name that does
not appear in the source document. Backtracking
successfully replaces it with the correct name. In
the second example, although the extended name of
the soccer club can include "United" based on real
world knowledge, the document itself only refers
the soccer club as "Scunthorpe". CoBa-d is able to
detect this and remove "United".
5.5
Analysis
Token Probability Threshold. We examine the
effects of using different values for the token confi-
dence threshold, and present the results in Figure 4.
We use the newsroom dataset and the Flan-T5 XL
model. To better capture faithfulness, all the met-
rics are computed between source document and
the generated summary. High value is better for all
metrics. For AlignScore and BS-Fact, the improve-
ment saturates at threshold 0.2-0.25, while FactCC
continue to improve.
Embedding Distance Threshold. We perform
ablation studies on the choice of embedding dis-
tance threshold. Intuitively, the smaller the distance
threshold is, the more similar the generated sum-
maries are to their original documents. Results
are presented in Table 3. "N/A" represents not
applying the embedding distance threshold. We
use token probability threshold 0.2, the Newsroom
dataset, and the Flan-T5 XL model for the ablation
experiments. Decreasing the threshold improves
the performance, and the improvement saturates
around threshold 0.5.
2
4
8
16
32
64
128
Average Time in s per Summary
0.7
0.75
0.8
0.85
0.9
AlignScore
Greedy
Greedy + Lookahead
Greedy + CoBa
Figure 3: AlignScore vs. Generation Time. Note that the
the x-axis is in log scale. The curve for Lookahead represents
doing lookahead every k tokens for k from 200 to 1. CoBa
attains the highest AlignScore with more than 10x speedup.
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.35
0.40
Token Confidence Threshold
0.5
0.6
0.7
0.8
0.9
1.0
Performance
AlignScore
FactCC
BS-Fact
Figure 4: Ablation on the token confidence threshold for
CoBa. High is better for all metrics. Most metrics saturate
around threshold 0.2-0.25.
7
Document: Dear God, what have they done to Joseph Gordon-Levitt's face? In Rian Johnson's time-travel
action drama, "Looper," the star has been prosthetically altered with a fresh set of eyebrows, a snubbed nose,
green contact lenses, and what appear to be new lips. He's supposed to look like the young Bruce Willis. What
he resembles, mostly, is mid-period Devo. The imposture is hardly convincing, but "Looper" is fast enough,
weird enough, and just about smart enough to make you forget about that. The movie wants to mess with your
head, depositing us in a decrepit, overcrowded 2044 and then sprinkling on time-loop … This is an article
preview. The full story is available to BostonGlobe.com subscribers.
Greedy Decoding: The Boston Globe's review of "Looper" by John Sutter.
CoBa: The Boston Globe reviews "Looper" by Rian Johnson.
Document: Scunthorpe midfielder Neal Bishop has signed a one-year contract extension. The 35-year-old
joined the Iron from Blackpool in 2013 and has made 119 league appearances for the League One side. He
helped them to a third-placed finish this season, before they were beaten by Millwall in the play-off semi-finals.
Bishop told the club website: "With the way the season finished, it's a sense of unfinished business and it was
disappointing for all of us."
Greedy Decoding: Scunthorpe United midfielder Neal Bishop has signed a new one-year contract with the Iron.
CoBa-d: Scunthorpe have signed midfielder Neal Bishop on a one-year contract extension.
Figure 5: Qualitative examples of greedy decoding vs. CoBa and CoBa-d. The hallucinated content is marked in
red and the corrected details are marked in green. CoBa and CoBa-d correctly remove the hallucinated content by
triggering backtracking at corresponding positions and generate summaries with more and faithful details.
Table 3: Ablation on the threshold on token em-
bedding distance. We use token confidence threshold
δ = 0.2 while varying the distance threshold φ for all
the experiments in this table.
Dist. Thresh
AlignScore↑
FactCC↑
BS-Fact↑
Rouge-L↑
N/A
0.821
0.674
0.923
0.138
0.9
0.825
0.677
0.924
0.139
0.7
0.859
0.699
0.925
0.143
0.5
0.865
0.709
0.926
0.145
0.3
0.867
0.718
0.920
0.146
0.1
0.867
0.720
0.920
0.146
6
Limitations and Future Work
In this study, we propose a method for reducing hal-
lucinations in text summarization by backtracking.
Our method consists of two steps: detection and
backtracking. We employ two token level condi-
tional probabilities and distance between generated
tokens and context tokens to detect hallucinations.
Both of these are effective ways of detecting hal-
lucinated text, but there could be other comple-
mentary metrics that could improve detection. We
defer the exploration of alternative metrics to future
research endeavors.
While our primary focus in this paper is summa-
rization models, our method can easily be extended
to other applications where generating factual text
is paramount. For instance, in question-answering
systems which first retrieve relevant documents and
then generate an answer, we can define the retrieved
documents to be the context and employ CoBa to
produce factually correct answers.
7
Conclusion
Current decoding methods don’t explicitly allow
a model to re-generate some part of the generated
text when there is no highly probable completion
to the partial text. Such a scenario would lead to
hallucinations because the model is uncertain about
how to complete the sentence and will sample a low
probability word. We show that there is a relatively
simple solution to mitigate hallucination, which we
refer to as Correction with Backtracking (CoBa).
CoBa is an inference-time method that requires no
additional models, is computationally efficient, and
can be directly applied to diverse summarization
models without retraining. CoBa detects halluci-
nations by using conditional probabilities of the
generated tokens and measuring the distance be-
tween the generated text and the context. To cor-
rect the hallucinated text, it applies backtracking
to before the hallucination and re-generates text to
avoid ending up in positions with only low scoring
token options. We empirically verify that CoBa
is able to identify and rectify hallucinated tokens
8
during autoregressive decoding, and we show that
CoBa produces more factual summaries for various
datasets. Our future work includes exploring other
detection strategies as well as extending CoBa to
more diverse tasks.
Acknowledgement
This research is supported by a gift from the Si-
mons Foundation, grants from the National Science
Foundation NSF (IIS-2107161, III1526012, IIS-
1149882, OAC-2118310), Natural Sciences and
Engineering Research Council of Canada (NSERC
567916) and IIS-1724282), the Cornell Center for
Materials Research with funding from the NSF
MRSEC program (DMR1719875), LinkedIn and
NewYork-Presbyterian Hospital.
References
Meng Cao, Yue Dong, Jiapeng Wu, and Jackie
Chi Kit Cheung. 2020. Factual error correction for
abstractive summarization models.
arXiv preprint
arXiv:2010.08712.
Yihan Cao, Yanbin Kang, and Lichao Sun. 2023. In-
struction mining:
High-quality instruction data se-
lection for large language models.
arXiv preprint
arXiv:2307.06290.
Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa
Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan,
Tianyi Zhou, Heng Huang, et al. 2023.
Alpagasus:
Training a better alpaca with fewer data. arXiv preprint
arXiv:2307.08701.
Sihao Chen, Fan Zhang, Kazoo Sone, and Dan Roth.
2021. Improving faithfulness in abstractive summariza-
tion with contrast candidate generation and selection.
arXiv preprint arXiv:2104.09061.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret
Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang,
Mostafa Dehghani, Siddhartha Brahma, et al. 2022.
Scaling instruction-finetuned language models. arXiv
preprint arXiv:2210.11416.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and
Kristina Toutanova. 2018. Bert: Pre-training of deep
bidirectional transformers for language understanding.
arXiv preprint arXiv:1810.04805.
Bhuwan Dhingra, Manaal Faruqui, Ankur Parikh,
Ming-Wei Chang, Dipanjan Das, and William W Co-
hen. 2019. Handling divergent reference texts when
evaluating table-to-text generation.
arXiv preprint
arXiv:1906.01081.
Yue Dong, Shuohang Wang, Zhe Gan, Yu Cheng, Jackie
Chi Kit Cheung, and Jingjing Liu. 2020. Multi-fact cor-
rection in abstractive text summarization. arXiv preprint
arXiv:2010.02443.
Nouha Dziri, Andrea Madotto, Osmar Zaiane, and
Avishek Joey Bose. 2021. Neural path hunter: Reducing
hallucination in dialogue systems via path grounding.
arXiv preprint arXiv:2104.08455.
Mohamed Elaraby, Mengyin Lu, Jacob Dunn, Xuey-
ing Zhang, Yu Wang, and Shizhu Liu. 2023. Halo:
Estimation and reduction of hallucinations in open-
source weak large language models. arXiv preprint
arXiv:2308.11764.
Max Grusky, Mor Naaman, and Yoav Artzi. 2018.
Newsroom: A dataset of 1.3 million summaries with
diverse extractive strategies.
arXiv preprint arXi
v:1804.11283.
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan
Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto,
and Pascale Fung. 2023. Survey of hallucination in
natural language generation. ACM Computing Surveys,
55(12):1–38.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom
Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer,
Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson,
et al. 2022. Language models (mostly) know what they
know. arXiv preprint arXiv:2207.05221.
Daniel King,
Zejiang Shen,
Nishant Subramani,
Daniel S Weld, Iz Beltagy, and Doug Downey. 2022.
Don’t say what you don’t know: Improving the con-
sistency of abstractive summarization by constraining
beam search. arXiv preprint arXiv:2203.08436.
Wojciech Kry´sci´nski, Bryan McCann, Caiming Xiong,
and Richard Socher. 2019. Evaluating the factual consis-
tency of abstractive text summarization. arXiv preprint
arXiv:1910.12840.
Ariel N Lee, Cole J Hunter, and Nataniel Ruiz. 2023.
Platypus: Quick, cheap, and powerful refinement of
llms. arXiv preprint arXiv:2308.07317.
Katherine Lee, Orhan Firat, Ashish Agarwal, Clara Fan-
njiang, and David Sussillo. 2018. Hallucinations in
neural machine translation.
Nayeon Lee, Wei Ping, Peng Xu, Mostofa Patwary,
Pascale N Fung, Mohammad Shoeybi, and Bryan Catan-
zaro. 2022. Factuality enhanced language models for
open-ended text generation. Advances in Neural Infor-
mation Processing Systems, 35:34586–34599.
Chenliang Li, Bin Bi, Ming Yan, Wei Wang, and Song-
fang Huang. 2021. Addressing semantic drift in gen-
erative question answering with auxiliary extraction.
In Proceedings of the 59th Annual Meeting of the As-
sociation for Computational Linguistics and the 11th
International Joint Conference on Natural Language
Processing (Volume 2: Short Papers), pages 942–947.
9
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter
Pfister, and Martin Wattenberg. 2023a. Inference-time
intervention: Eliciting truthful answers from a language
model. arXiv preprint arXiv:2306.03341.
Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie
Del Giorno, Suriya Gunasekar, and Yin Tat Lee. 2023b.
Textbooks are all you need ii: phi-1.5 technical report.
arXiv preprint arXiv:2309.05463.
Chin-Yew Lin. 2004.
Rouge: A package for auto-
matic evaluation of summaries. In Text summarization
branches out, pages 74–81.
Shayne Longpre, Kartik Perisetla, Anthony Chen,
Nikhil Ramesh, Chris DuBois, and Sameer Singh. 2021.
Entity-based knowledge conflicts in question answering.
arXiv preprint arXiv:2109.05052.
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and
Ryan McDonald. 2020a.
On faithfulness and fac-
tuality in abstractive summarization. arXiv preprint
arXiv:2005.00661.
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and
Ryan McDonald. 2020b.
On faithfulness and fac-
tuality in abstractive summarization. arXiv preprint
arXiv:2005.00661.
Niels Mündler, Jingxuan He, Slobodan Jenko, and Mar-
tin Vechev. 2023. Self-contradictory hallucinations of
large language models: Evaluation, detection and miti-
gation. arXiv preprint arXiv:2305.15852.
Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing
Xiang, et al. 2016. Abstractive text summarization using
sequence-to-sequence rnns and beyond. arXiv preprint
arXiv:1602.06023.
Shashi Narayan, Shay B Cohen, and Mirella Lapata.
2018. Don’t give me the details, just the summary!
topic-aware convolutional neural networks for extreme
summarization. arXiv preprint arXiv:1808.08745.
Artidoro Pagnoni, Vidhisha Balachandran, and Yulia
Tsvetkov. 2021a. Understanding factuality in abstrac-
tive summarization with frank: A benchmark for factu-
ality metrics. arXiv preprint arXiv:2104.13346.
Artidoro Pagnoni, Vidhisha Balachandran, and Yulia
Tsvetkov. 2021b. Understanding factuality in abstrac-
tive summarization with frank: A benchmark for factu-
ality metrics. arXiv preprint arXiv:2104.13346.
Guilherme Penedo, Quentin Malartic, Daniel Hess-
low, Ruxandra Cojocaru, Alessandro Cappelli, Hamza
Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and
Julien Launay. 2023. The refinedweb dataset for falcon
llm: outperforming curated corpora with web data, and
web data only. arXiv preprint arXiv:2306.01116.
Suzanne Petryk, Spencer Whitehead, Joseph E Gon-
zalez, Trevor Darrell, Anna Rohrbach, and Mar-
cus Rohrbach. 2023.
Simple token-level confi-
dence improves caption correctness. arXiv preprint
arXiv:2305.07021.
Abigail See, Peter J Liu, and Christopher D Manning.
2017. Get to the point: Summarization with pointer-
generator networks. arXiv preprint arXiv:1704.04368.
Thibault Sellam, Dipanjan Das, and Ankur P Parikh.
2020. Bleurt: Learning robust metrics for text genera-
tion. arXiv preprint arXiv:2004.04696.
Weijia Shi,
Xiaochuang Han,
Mike Lewis,
Yu-
lia Tsvetkov, Luke Zettlemoyer, and Scott Wen-tau
Yih. 2023.
Trusting your evidence:
Hallucinate
less with context-aware decoding.
arXiv preprint
arXiv:2305.14739.
Robert Tarjan. 1972. Depth-first search and linear graph
algorithms. SIAM journal on computing, 1(2):146–160.
Ran Tian, Shashi Narayan, Thibault Sellam, and
Ankur P Parikh. 2019. Sticking to the facts: Confi-
dent decoding for faithful data-to-text generation. arXiv
preprint arXiv:1910.08684.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier
Martinet, Marie-Anne Lachaux, Timothée Lacroix, Bap-
tiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar,
et al. 2023a. Llama: Open and efficient foundation
language models. arXiv preprint arXiv:2302.13971.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert,
Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov,
Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.
2023b. Llama 2: Open foundation and fine-tuned chat
models. arXiv preprint arXiv:2307.09288.
Neeraj Varshney, Wenlin Yao, Hongming Zhang, Jian-
shu Chen, and Dong Yu. 2023. A stitch in time saves
nine: Detecting and mitigating hallucinations of llms by
validating low-confidence generation. arXiv preprint
arXiv:2307.03987.
David Wan, Mengwen Liu, Kathleen McKeown,
Markus Dreyer, and Mohit Bansal. 2023. Faithfulness-
aware decoding strategies for abstractive summarization.
arXiv preprint arXiv:2303.03278.
Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020.
Asking and answering questions to evaluate the
factual consistency of summaries.
arXiv preprint
arXiv:2004.04228.
Yuheng Zha, Yichi Yang, Ruichen Li, and Zhiting
Hu. 2023. Alignscore: Evaluating factual consistency
with a unified alignment function.
arXiv preprint
arXiv:2305.16739.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Pe-
ter Liu. 2020. Pegasus: Pre-training with extracted
gap-sentences for abstractive summarization.
In In-
ternational Conference on Machine Learning, pages
11328–11339. PMLR.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Wein-
berger, and Yoav Artzi. 2019. Bertscore: Evaluating text
generation with bert. arXiv preprint arXiv:1904.09675.
10
Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu,
Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang,
Yulong Chen, et al. 2023. Siren’s song in the ai ocean: A
survey on hallucination in large language models. arXiv
preprint arXiv:2309.01219.
Zheng Zhao, Shay B Cohen, and Bonnie Webber. 2020.
Reducing quantity hallucinations in abstractive summa-
rization. arXiv preprint arXiv:2009.13312.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao
Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu,
Lili Yu, et al. 2023. Lima: Less is more for alignment.
arXiv preprint arXiv:2305.11206.
11
A
Appendix
A.1
Additional Implementation Details
For the baselines, we use the Hugging Face implementation1 for greedy decoding and nucleus sampling,
and the official code of lookahead2. We use our own implementation for CAD as we did not find existing
publicly available implementation. We use top-p = 0.9 for nucleus sampling. Lookahead performs rollout
for the top-k tokens with the highest probabilities; we use k = 5 following the original paper. CAD uses a
scaling factor α when adjusting the conditional probabilities with the unconditional probabilities; we use
α = 0.5 following the original paper. During generation, we set the minimum generation length to be 2
and maximum generation length to be 200 for all decoding methods.
A.2
Summarization Metrics between Ground Truth and Generated Summaries
We report the summarization metrics between the ground truth summaries and the generated summaries
from Flan-T5 in Table 4 and LLaMA in Table 5. All decoding methods with both models demonstrate
reasonable performance. It is important to note that the ground truth summaries for each dataset are
collected by distinct criteria: CNN/Dailymail (Nallapati et al., 2016) uses the human-written story
highlights in bullet points, XSUM takes the first sentence of a document (Narayan et al., 2018; Wang
et al., 2020), and Newsroom uses the HTML metadata (Grusky et al., 2018). As the models are not further
finetuned on the individual datasets, their summaries often exhibit different styles from the ground truth
summaries. Consequently, the summarization metrics only provide limited insights into the quality of the
generated summaries.
A.3
Generated Summary Lengths
Figure 6 shows the lengths of generated summaries from Flan-T5 on the Newsroom dataset. In general,
the length distribution is similar across different decoding methods.
Greedy
Lookahead
CAD
CoBa
CoBa-d
0
25
50
75
100
125
150
175
200
Number of Tokens
Number of Tokens in Generated Summaries
Greedy
Lookahead
CAD
CoBa
CoBa-d
0
200
400
600
800
1000
Number of Characters
Number of Characters in Generated Summaries
Figure 6: Number of tokens (left) and number of characters (right) in the generated summaries from Flan-T5 on the
Newsroom dataset. The lengths have similar distributions across generation methods.
1https://huggingface.co
2https://github.com/amazon-science/faithful-summarization-generation
12
Table 4: Summarization metrics between the ground truth summaries from the dataset and the generated
summaries using Flan-T5. Higher is better.
Method
ROUGE-L↑
BERTScore F1↑
Bleurt↑
Newsroom
Greedy
0.312
0.890
0.441
+ Lookahead (every 8 tok.)
0.313
0.890
0.441
+ Lookahead (every 4 tok.)
0.314
0.891
0.442
+ Lookahead (every 2 tok.)
0.323
0.892
0.451
+ Lookahead (every tok.)
0.322
0.892
0.451
+ CAD
0.281
0.883
0.412
+ CoBa
0.313
0.889
0.436
+ CoBa-d
0.306
0.885
0.428
+ CoBa + CAD
0.281
0.883
0.412
+ CoBa-d + CAD
0.267
0.878
0.399
Nucleus
0.267
0.883
0.406
+ CAD
0.270
0.882
0.404
+ CoBa
0.306
0.888
0.432
+ CoBa-d
0.299
0.882
0.423
+ CoBa + CAD
0.282
0.883
0.412
+ CoBa-d + CAD
0.269
0.879
0.401
XSUM
Greedy
0.422
0.920
0.540
+ Lookahead (every 8 tok.)
0.426
0.921
0.543
+ Lookahead (every 4 tok.)
0.431
0.922
0.546
+ Lookahead (every 2 tok.)
0.467
0.927
0.569
+ Lookahead (every tok.)
0.483
0.929
0.578
+ CAD
0.399
0.916
0.522
+ CoBa
0.431
0.920
0.541
+ CoBa-d
0.462
0.919
0.548
+ CoBa + CAD
0.402
0.916
0.523
+ CoBa-d + CAD
0.440
0.919
0.535
Nucleus
0.297
0.902
0.459
+ CAD
0.335
0.907
0.485
+ CoBa
0.411
0.918
0.527
+ CoBa-d
0.442
0.918
0.534
+ CoBa + CAD
0.388
0.915
0.515
+ CoBa-d + CAD
0.426
0.917
0.526
CNN/DM
Greedy
0.260
0.874
0.388
+ Lookahead (every 8 tok.)
0.261
0.875
0.389
+ Lookahead (every 4 tok.)
0.262
0.875
0.389
+ Lookahead (every 2 tok.)
0.265
0.876
0.395
+ Lookahead (every tok.)
0.265
0.876
0.396
+ CAD
0.248
0.871
0.392
+ CoBa
0.256
0.873
0.382
+ CoBa-d
0.256
0.872
0.383
+ CoBa + CAD
0.248
0.871
0.391
+ CoBa-d + CAD
0.246
0.870
0.389
Nucleus
0.235
0.870
0.370
+ CAD
0.241
0.869
0.386
+ CoBa
0.253
0.872
0.379
+ CoBa-d
0.254
0.871
0.379
+ CoBa + CAD
0.246
0.871
0.389
+ CoBa-d + CAD
0.245
0.870
0.388
13
Table 5: Summarization metrics between the ground truth summaries from the dataset and the generated
summaries using LLaMA. Higher is better.
Method
ROUGE-L↑
BERTScore F1↑
Bleurt↑
Newsroom
Greedy
0.210
0.861
0.438
+ CAD
0.207
0.872
0.429
+ CoBa
0.212
0.870
0.439
+ CoBa-d
0.214
0.868
0.436
XSUM
Greedy
0.376
0.915
0.564
+ CAD
0.362
0.908
0.533
+ CoBa
0.379
0.915
0.564
+ CoBa-d
0.383
0.915
0.564
CNN/DM
Greedy
0.239
0.859
0.406
+ CAD
0.236
0.872
0.401
+ CoBa
0.240
0.861
0.407
+ CoBa-d
0.240
0.860
0.405
14