Question:
Which of the following statements about the experimental results in the article are correct?
A. On the TACRED dataset, the micro-F1 score of the LLaMA-7B + RE4 framework significantly surpasses that of the GPT-3 model(175b), due to the schema constrain of proposed RE4 method.
B. This work demonstrates that open-source LLMs can outperform proprietary large models in complex tasks (e.g., RE) through framework optimization.
C. The proposed method with LLaMA-7b achieves results comparable to, or even surpassing, those of ultra-large models such as GPT-3 across multiple datasets.
D. The RE4 framework enhances parameter efficiency in mid-sized open-source models far beyond traditional supervised models through ontology knowledge distillation and reasoning optimization.

Answer: BCD

Method (#Param.)
SemEval TACRED Google RE SciERC
MTB (336M)
89.5
71.5
92.7
87.4
LUKE (355M)
90.1
72.7
94.0
87.7
IRE (355M)
89.8
74.6
93.1
88.9
KLG (355M)
90.5
75.6
-
-
KnowPrompt (355M)
90.2
72.4
-
-
NLI-DeBERTa (1.5B)
-
73.9
-
-
REBEL (400M)
-
73.7
93.5
86.3
TANL (220M)
-
74.8
-
-
RELA (400M)
90.4
71.2
93.9
90.3
DeepStruct (10B)
-
76.8
-
-
GPT-3 (175B)
70.1
32.5
-
-
GPT-RE (175B)
91.9
72.1
-
-
BART-Base w/ RE4
89.8
71.5
92.4
86.0
BART-Large w/ RE4
90.6
73.3
93.1
87.2
T5-Base w/ RE4
89.9
72.7
92.6
86.3
T5-Large w/ RE4
90.9
75.6
93.4
87.8
LLaMA w/ RE4
92.1
77.2
94.5
91.7
LLaMA w/o RE4
90.6
75.0
92.9
89.5
Table 2: Micro-F1 score of test sets on four RE datasets. Results of
baselines are retrieved from original papers. For ICL-based results,
we use best 30-shot on SemEval and 15-shot on TACRED. Previous
state-of-the-art results are marked with underline, and best results
are bold. Results of RE4 are averaged over three random seeds.

3.2
Main Results
The main results of baselines and RE4 are summarized in
Table 2, where RE4 with LLaMA outperforms all previous
state-of-the-art methods on four RE datasets. Compared to
the classification-based methods, RE4 shows favorable results
without any external dataset usage or additional pre-training
stages, while MTB and LUKE all involve entity and relation
related pre-training tasks. RE4 with most LLMs also sup-
presses two prompt-based methods. Compared to previous
generative-based methods, RE4 consistently achieves com-
petitive or superior results on four datasets. For example,
with same backbone BART-Large, RE4 and REBEL share
similar overall results, while REBEL is pre-trained with a
large external relational triple extraction dataset.
Notably,
RE4 with LLaMA significantly outperforms REBEL on TA-
CRED and SciERC, and vanilla fine-tuned LLaMA still sur-
passes REBEL, due to the fact that LLaMA (7B) has a much
larger number of model parameters than REBEL (400M). Al-
though LLaMA is smaller than DeepStruct which is based
on a pre-trained 10B parameter language model GLM [Du
et al., 2022] and is pre-trained on a collection of large-scale
corpus, RE4 delivers better results on TACRED than Deep-
Struct. This is notable because RE4 adopt smaller founda-
tion models for fine-tuning and utilize no external datasets for
pre-training compared to DeepStruct. In addition, RE4 with
LLaMA achieves comparable results on SemEval and much
better results on TACRED compared to ICL-based method
GPT-RE. Without fine-tuning, if GPT-3 lacks relevant domain
knowledge about specific tasks, then the performance of GPT-
RE is limited and greatly affected by the retrieved demon-
strations. And this is why GPT-RE delivers exceptional re-
sults on SemEval but unattractive results that even worse than
LUKE on TACRED. In other words, solely relying on ICL-
based methods is unstable. The main results demonstrate the
simplicity and effectiveness of RE4 compared to baselines.

Recall, Retrieve and Reason: Towards Better In-Context Relation Extraction
Guozheng Li1 , Peng Wang1,2∗, Wenjun Ke1,2 , Yikai Guo3 ,
Ke Ji1 , Ziyu Shang1 , Jiajun Liu1 and Zijie Xu1
1School of Computer Science and Engineering, Southeast University
2Key Laboratory of New Generation Artificial Intelligence Technology and Its
Interdisciplinary Applications (Southeast University), Ministry of Education
3Beijing Institute of Computer Technology and Application
{gzli, pwang, kewenjun, jiajliu, keji, ziyus1999, zijiexu}@seu.edu.cn
Abstract
Relation extraction (RE) aims to identify relations
between entities mentioned in texts. Although large
language models (LLMs) have demonstrated im-
pressive in-context learning (ICL) abilities in var-
ious tasks, they still suffer from poor performances
compared to most supervised fine-tuned RE meth-
ods. Utilizing ICL for RE with LLMs encounters
two challenges: (1) retrieving good demonstrations
from training examples, and (2) enabling LLMs ex-
hibit strong ICL abilities in RE. On the one hand,
retrieving good demonstrations is a non-trivial pro-
cess in RE, which easily results in low relevance
regarding entities and relations. On the other hand,
ICL with an LLM achieves poor performance in
RE while RE is different from language modeling
in nature or the LLM is not large enough. In this
work, we propose a novel recall-retrieve-reason RE
framework that synergizes LLMs with retrieval cor-
pora (training examples) to enable relevant retriev-
ing and reliable in-context reasoning. Specifically,
we distill the consistently ontological knowledge
from training datasets to let LLMs generate relevant
entity pairs grounded by retrieval corpora as valid
queries. These entity pairs are then used to retrieve
relevant training examples from the retrieval cor-
pora as demonstrations for LLMs to conduct better
ICL via instruction tuning. Extensive experiments
on different LLMs and RE datasets demonstrate
that our method generates relevant and valid entity
pairs and boosts ICL abilities of LLMs, achieving
competitive or new state-of-the-art performance on
sentence-level RE compared to previous supervised
fine-tuning methods and ICL-based methods.
1
Introduction
The emergence of large language models (LLMs) such as
GPT-3 [Brown et al., 2020] represent a significant advance-
ment in natural language processing (NLP). Instead of fol-
lowing a pre-training then fine-tuning pipeline [Radford et
al., 2019; Devlin et al., 2019; Liu et al., 2019; Raffel et
∗Corresponding author
al., 2020], which fine-tunes a pre-trained model on a task-
specific dataset in a fully-supervised manner, LLMs employ a
new paradigm known as in-context learning (ICL) [Brown et
al., 2020] which formulates an NLP task under the paradigm
of language generation and makes predictions by learning
from a few demonstrations. LLMs with ICL demonstrate im-
pressive performance comparable to traditional methods even
with limited examples [Brown et al., 2020; Zhao et al., 2021].
Despite the generally promising ICL performances [Wei
et al., 2022b; Arora et al., 2023; Shang et al., 2024], cur-
rent ICL in relation extraction (RE) [Li et al., 2024] task suf-
fers from relatively poor performance. RE is a pivotal task in
NLP [Li et al., 2022; Wang et al., 2023a; Wang et al., 2023b;
Ji et al., 2023; Liu et al., 2024], necessitating a profound com-
prehension of natural language, which involves identifying a
pre-defined relation between a given entity pair mentioned in
the input sentence or marking it as NA if no relation is iden-
tified. Given a test example input, ICL for RE prompts the
input of LLMs with a few demonstrations retrieved from the
training data and the test input itself, then LLMs generate the
corresponding relation. Recent studies [Jimenez Gutierrez et
al., 2022; Ma et al., 2023] have revealed a significant perfor-
mance gap in LLMs when apply ICL to the RE task. The
main obstacles that utilizing ICL for RE with LLMs are two-
fold: (i) the low relevance regarding entity and relation in the
retrieved demonstrations for ICL [Wan et al., 2023], and (ii)
the failure of utilizing LLMs with moderate size (less than
10B) compared to 175B GPT-3 for ICL [Li et al., 2023c].
Typically, ICL demonstrations are selected randomly or via
similarity-based sentence embedding obtained by sentence
encoder such as Sentence-BERT [Reimers and Gurevych,
2019]. However, similarity-based retrieval is more concerned
with the relevance of the overall sentence semantics and not
as much with the specific entities and relations it contains,
where consequently the test input tends to retrieve a seman-
tically similar sentence but is not desired in terms of enti-
ties and relations. Similar with finding good demonstrations
in ICL for general NLP tasks [Liu et al., 2022], the demon-
strations for RE should contain as much as similar or exactly
same entities and relations regarding the test example, so as
to significantly help the test example infer the relation be-
tween entities. However, current retrieval techniques [Gao et
al., 2021; Wan et al., 2023] fail to locate the delicate demon-
strations considering the consistency of ontology (i.e., gen-
arXiv:2404.17809v1  [cs.CL]  27 Apr 2024
Stanford University is located in Palo Alto,
California, and is one of the world's
renowned private research universities.
Oxford University is situated in the city of
Oxford, England, and is one of the oldest and
most prestigious universities in the United
Kingdom.
Carnegie Mellon University is a neighbor of
the Allegheny City, commonly known as the
North Shore.
The University of Tokyo is a national
university located in Bunkyo, Tokyo, Japan,
and is considered one of the top universities
in Asia.
The Massachusetts Institute of Technology
was founded by William Barton Rogers.
Oxford University is situated in the city of
Oxford, England, and is one of the oldest and
most prestigious universities in the United
Kingdom.
Carnegie Mellon University is a neighbor of
the Allegheny City, commonly known as the
North Shore.
Pittsburgh is a city located in the western part
of the state of Pennsylvania, United States,
which is approximately 300 miles (480
kilometers) west of the state capital,
Harrisburg.
Carnegie Mellon University (CMU) is
located in Pittsburgh, Pennsylvania,
United States.
Carnegie Mellon University (CMU) is
located in Pittsburgh, Pennsylvania,
United States.
Demonstrations
Test Example
Figure 1: Comparison between naive demonstration selection (left)
and our demonstration selection (right) methods. Different colors
represent different relations between entity pairs, while green repre-
sents the golden relation expressed by entity pairs in test example.
eralized entity types and the relations between them). Since
entity types grouped under the similar ontology are often cor-
related and share similar relations [Roche, 2003], we assume
that examples sharing the similar ontology are relevant and
could serve as good demonstrations for ICL. To this end,
we propose to retrieve the relevant demonstrations in RE in-
corporating consistent ontological knowledge. Figure 1 il-
lustrates the basic principles of our demonstration selection
method, where we expect the demonstrations to include as
many entity pairs as possible that express the same relation
(i.e., similar ontology) with the test example. To obtain such
ontological knowledge guided demonstrations, we propose to
let the LLMs learn to generate relevant entity pairs that share
the same relations with each test example, by implicitly ac-
quiring the ontological knowledge of specific RE task during
training phase. Then ICL demonstrations are retrieved from
training examples by these recalled entity pairs. Compared to
similarity-based retrieval, recalling entities explicitly via im-
plicitly ontological knowledge guiding bring more benefits.
Besides the demonstration selection, the parameter scale of
LLMs also plays an important role in ICL. Existing promis-
ing RE results obtaining via ICL all use very large-scale pro-
prietary models [Agrawal et al., 2022; Li et al., 2023c]. Al-
though smaller open-source LLMs such as LLaMA [Touvron
et al., 2023] can also perform ICL similarly in simple clas-
sification tasks like sentiment classification, they are unable
to perform more challenge tasks like extracting relations be-
tween entities through in-context demonstrations.
Due to
the limitations of proprietary models (higher computation
cost and time consumption, concerns about privacy protec-
tion and local deployment, etc.), open-source smaller LLMs
(less than 10B) achieve better ICL performance is of more
applicability. Therefore, recent studies [Chen et al., 2022b;
Min et al., 2022b] attempt to boost the moderate size LLMs
ICL ability via meta in-context learning where an LLM is
tuned to do in-context learning on a large set of training tasks.
Inspired by this, we optimize LLMs to do in-context RE on
training set, making the model becomes more effectively to
reason about the relation between entities in-context by con-
ditioning on a few training examples at inference time. Com-
pared to directly optimize the input-output formats of exam-
ples (sentence and entity pair as input and relation label as
output), this method can conduct in-context reasoning based
on retrieved examples and generate more accurate results.
To alleviate the above issues of irrelevant demonstration
retrieval and inferior ICL capability, we present a recall-
retrieve-reason framework, a novel RE method called RE4
(Relation Extraction with REcall, REtrieve and REason) that
synergizes open-source LLMs with retrieval corpora (train-
ing examples) to retrieve relevant demonstrations and con-
duct in-context reasoning. Specifically, RE4 first generates
entity pairs guided by ontological knowledge and grounded
by retrieval corpora as valid queries via the recalling module.
These entity pairs are then used to retrieve valid training ex-
amples from retrieval corpora in retrieval module to conduct
in-context reasoning by reasoning module. In this way, we
not only retrieve the relevant demonstrations from retrieval
corpora but also consider the guidance of entity pairs and rela-
tions for reasoning. Based on this framework, RE4 is joint op-
timized by two tasks: 1) recalling optimization, where we dis-
till ontological knowledge from retrieval corpora into LLMs
to generate relevant and valid entity pairs as queries; and 2)
reasoning optimization, where we enable LLMs to conduct
in-context reasoning based on retrieved demonstrations and
generate predicted relations. We conduct extensive experi-
ments on RE benchmarks to validate the effectiveness of RE4.
In summary, our contributions are three-fold:
• We propose a novel recall-retrieve-reason framework for
RE that synergizes open-source LLMs with training ex-
amples to retrieve relevant demonstrations and conduct
in-context reasoning. Moreover, the recalling module of
RE4 can be plug-and-play with different LLMs during
inference to improve their performance.
• We distill the consistently ontological knowledge from
training examples to guide LLMs for valid and relevant
entity generation, and propose to boost the open-source
LLMs in-context reasoning for RE via ICL tuning.
• We conduct extensive experiments on RE benchmarks
and the results demonstrate that RE4 achieves state-of-
the-art performance in sentence-level RE.
2
Methodology
2.1
Task Formulations
For a given natural language sentence s contains N words,
relation extraction aims at extracting the pre-defined relation-
ship r between h and t, where h and t are two given target
entities in s. If there is no pre-defined relation between h and
t, predict NA. Following previous works [Cabot and Navigli,
2021; Paolini et al., 2021; Li et al., 2023b], we treat RE as a
generation task following the standard decoding manner. The
optimization procedure can be written as follows:
p(r|s, h, t) =
L
Y
l=1
p(yl|s, h, t, y<l)
(1)
where we use an auto-regressive transformer [Vaswani et al.,
2017] decoder to generate the relation name r with label
The Massachusetts Institute of Technology was founded by William Barton Rogers.
(Massachusetts Institute of Technology, Founded by, William Barton Rogers)
Stanford University is located in Palo Alto, California, and is one of the world's renowned private
research universities.
(Stanford University, Located in, Palo Alto, California)
Carnegie Mellon University is a neighbor of the Allegheny City, commonly known as the North Shore.
(Carnegie Mellon University, Neighbor of, Allegheny City)
Retrieval Corpora (Training Examples)
Test Example:
Carnegie Mellon University (CMU) is located in Pittsburgh, Pennsylvania, United States.
(Carnegie Mellon University (CMU), ?, Pittsburgh, Pennsylvania, United States)
Predicted Relation:
(Carnegie Mellon University (CMU), Located in, Pittsburgh, Pennsylvania, United States)
Large Language Model (LLM)
Relevant Entity Pairs:
Stanford University - Palo Alto, California
Oxford University - Oxford, England
Carnegie Mellon University - Allegheny City
University of Tokyo - Bunkyo, Tokyo, Japan
Relevant Demonstrations:
Stanford University is located in...
Oxford University is situated in the...
Carnegie Mellon University is a neighbor...
The University of Tokyo is a national...
①Recalling
③Reasoning
②Retrieving
Recalling Prompt
Please generate k valid entity pairs with similar
ontologies between them that can be helpful for
identifying the relation from following entity pair
in the sentence.
Sentence: Carnegie Mellon University (CMU) is
located in Pittsburgh, Pennsylvania, United States.
Head Entity: Carnegie Mellon University (CMU)
Tail Entity: Pittsburgh, Pennsylvania, United
States
| Head Entity | Tail Entity |
| Stanford University | Palo Alto, California |
| Oxford University | Oxford, England |
| Carnegie Mellon University | Allegheny City |
| University of Tokyo | Bunkyo, Tokyo, Japan |
|
|
|
Reasoning Prompt
Based on these examples, please identify the
relation from the given entity pair in the sentence.
Sentence: Stanford University is located in Palo
Alto, California, and is one of the world's
renowned private research universities.
Head Entity: Stanford University
Tail Entity: Palo Alto, California
Relation: Located in
Sentence: Carnegie Mellon University (CMU) is
located in Pittsburgh, Pennsylvania, United States.
Head Entity: Carnegie Mellon University (CMU)
Tail Entity: Pittsburgh, Pennsylvania, United
States
Relation: Located in
Inference
Optimization
Prompt Details
Figure 2: Illustration of the RE4 framework. Given a test example, we first prompt LLMs to generate several relevant entity pairs that are
grounded by retrieval corpora as queries. Then we retrieve demonstrations from training examples using the queries. Finally, we conduct
in-context reasoning based on the retrieved entities and relations. The instructions in prompts are marked with underline, and the outputs of
LLMs in prompts are highlighted in blue.
length L, and p(yl|s, h, t, y<l) denotes the probability of each
token y in r generated by LLMs.
2.2
Framework Overview
Recently, many techniques have been explored to improve
the ICL ability of LLMs in RE, which first retrieves relevant
demonstrations from training examples and then conduct in-
context RE based on them [Wan et al., 2023]. However, the
challenges of low-quality demonstration retrieval and infe-
rior ICL ability of open-source models hinder the progress of
ICL for RE in the era of LLMs. To address these issues, we
propose a novel recall-retrieve-reason framework, which re-
calls relevant entity pairs, retrieves good demonstrations and
then perform in-context reasoning for better relation predic-
tions. The overall framework of RE4 is illustrated in Figure 2.
Given a test example “Carnegie Mellon University (CMU) is
located in Pittsburgh, Pennsylvania, United States”, aiming
to identify the relation between “Carnegie Mellon University
(CMU)” and “Pittsburgh, Pennsylvania, United States”, we
generate an entity pair “Stanford University” and “Palo Alto,
California” as the query. This entity pair expresses the similar
relation with the test entity pair as the consistency of ontology
between them (i.e., the head entity type is Organization and
the tail entity type is Location). Then we retrieve the corre-
sponding demonstration from training examples. Finally, we
predict the relation (i.e. Located in) based on demonstrations.
2.3
Framework Optimization
We formulate our RE4 as an optimization problem that aims
to maximize the probability of reasoning the relation r from
a retrieval corpora C w.r.t the test example e by generating
consistently ontological entity pairs z as the queries:
pθ(r|e, C) =
X
z∈Z
pθ(r|e, z, C)pθ(z|e)
(2)
where θ denotes the parameters of LLMs, z denotes the entity
pairs (queries) generated by LLMs, and Z denotes the set of
entity pairs that share similar ontology with the test example
e. The latter term pθ(z|e) is the probability of generating
a valid entity pair z grounded by retrieval corpora given e,
which is realized by the recalling module. The former term
pθ(r|e, z, C) is the probability of reasoning the relation r in
context given the test example e, entity pair z, and retrieval
corpora C, computing by the reasoning module.
Despite the advantage of generating entity pairs as queries,
the LLMs have zero ontological knowledge of the entities and
relations contained in training examples. Therefore, LLMs
cannot directly generate entity pairs grounded by retrieval
corpora as valid queries. Moreover, LLMs might not have
strong ICL ability to conduct effective in-context reasoning
based on them. To address these issues, we design two in-
struction tuning tasks: 1) recalling optimization, which dis-
tills the consistent ontological knowledge from training ex-
amples into LLMs to generate valid entity pairs as queries,
and 2) reasoning optimization, which enables LLMs to per-
form in-context reasoning based on retrieved demonstrations.
The objective function in equation (2) is optimized by maxi-
mizing the evidence lower bound, which is formulated as:
log pθ(r|e, C) ≥Ez∼q(z)[log pθ(r|e, z, C)]
−DKL(q(z) || pθ(z|e))
(3)
where q(z) denotes the posterior distribution of valid and rel-
evant entity pairs grounded by retrieval corpora. The latter
term minimizes the KL divergence between the posterior and
the prior, which encourages LLMs to generate consistently
ontological entity pairs (i.e. recalling optimization). The for-
mer term maximizes the expectation that reasoning module
generates correct relations based on the retrieved demonstra-
tions (i.e. reasoning optimization).
2.4
Recalling Entity Pairs
To make LLMs generate valid entity pairs as queries for
retrieving consistently ontological demonstrations from re-
trieval corpora, we minimize the KL divergence with the pos-
terior distribution of valid entity pairs q(z), which can be ap-
proximated by the valid entity pairs in retrieval corpora C.
Given a test example e and its golden relation r, we could
find the entity pair instances z = (h, t) expressing the same
relation r in training examples. And z can be considered valid
and serve as a query for retrieving the relevant demonstration
of e. And q(z) can be formally approximated as:
q(z) ∼= q(z|r, e, C) =
1
|Z|, ∃z ∈C
(4)
where we assume a uniform distribution over all consistently
ontological entity pairs Z regarding e, and ∃z ∈C denotes
the existence of an entity pair instance connecting e and r in
C. Therefore, the KL divergence can be calculated as:
Lrecall = DKL(q(z) || pθ(z|e)) ∼= −1
|Z∗|
X
z∈Z∗
log pθ(z|e)
(5)
where we use the partial entity pairs Z∗⊂Z relevant to e in
retrieval corpora C as supervision signals [Luo et al., 2023],
where we distill the consistently ontological knowledge from
training examples to LLMs.
To utilize the instruction-
following ability of LLMs [Wei et al., 2022a], we design an
instruction template that prompts LLMs to generate k = |Z∗|
entity pairs. Therefore, the optimization of Lrecall becomes:
−1
|Z∗|
X
z∈Z∗
log pθ(z|e) = −1
|Z∗|
X
z∈Z∗
log
|z|
Y
i=1
pθ(yi|y<i, e)
(6)
where pθ(z|e) denotes the prior distribution of generating
valid entity pair z, and pθ(yi|y<i, e) denotes the probability
of each token in z generated by LLMs.
2.5
Retrieving From Corpora
Given a test example e and an entity pair as query z, the re-
trieving module aims to retrieve the relevant demonstration d
from retrieval corpora. The retrieval process can be simply
conducted by exact match between the generated entity pair
z and the entity pair of each candidate demonstration d:
D = {d | hz = hd, tz = td, z = (hz, tz), d = (s, hd, td)}
(7)
where D denotes the set of retrieved demonstrations for ICL.
Despite we can utilize the retrieved entity pairs to directly get
the predicted relations via majority vote, the retrieved entity
pairs of relations could be noisy and imperfect to the test ex-
ample e, leading to incorrect predictions. Therefore, we use
a reasoning module to boost the ICL ability of LLMs to iden-
tity the important entity pairs of relations and predict relations
based on them via in-context reasoning [Chen et al., 2023].
2.6
Reasoning Through Demonstrations
In reasoning module, we aim to enable LLMs to conduct in-
context reasoning based on the relevant entity pairs. And rea-
soning on multiple relevant entity pairs is formulated as:
pθ(r|e, Z, C) =
Y
z∈Z
pθ(r|e, z, C)
(8)
By approximating the expectation with k sampled entity pairs
Z∗, the objective function of reasoning optimization where
maximizes the probability of LLMs generating golden rela-
tions based on the relevant entity pairs is formalized as :
Lreason = −Ez∼q(z)[log pθ(r|e, z, C)]
= −
X
z∈Z∗
log pθ(r|e, z, C)
= −log pθ(r|e, Z∗, C)
(9)
The reasoning module takes the test example e and a set
of retrieved demonstrations D to generate relation r. Simi-
lar with recalling module, we design a reasoning instruction
prompt to guide LLMs to conduct in-context reasoning based
on the retrieved demonstrations D. The D are formulated as a
series of structural sentences in standard ICL paradigm. The
optimization of Lreason is formulated as:
log pθ(r|e, Z∗, C) = log
X
z∈Z∗
X
d∈D
|r|
Y
i=1
pθ(yi|y<i, e, d) (10)
where pθ(r|e, Z∗, C) denotes probability of reasoning the
golden relation r based on k retrieved demonstrations Z∗,
and yi denotes the i-th token of relation r. To reduce the
impact of error propagation during recalling process and im-
prove the robustness of the model for in-context reasoning
during inference time, we add some noise into Z∗by replac-
ing k∗, (1 ≤k∗≤k) of k demonstrations in Z∗with training
examples share different relations from retrieval corpora with
a uniform distribution. In this way, the LLMs can learn to
in-context reasoning [Chen et al., 2022b; Min et al., 2022b;
Coda-Forno et al., 2023] based on important entities and rela-
tions, avoiding simply deduce the relations via majority vote.
2.7
Joint Optimization and Inference
The final objective function of RE4 is the combination of the
recalling optimization and reasoning optimization, which can
be formulated as:
L = −1
|Z∗|
X
z∈Z∗
log
|z|
Y
i=1
pθ(yi|y<i, e)
−log
X
z∈Z∗
X
d∈D
|r|
Y
i=1
pθ(yi|y<i, e, d)
(11)
We adopt the same LLM for both recalling and reasoning,
which are jointly trained on two instruction tuning tasks.
To enhance the efficiency of the fine-tuning process and re-
duce memory requirements, we utilize Low-Rank Adapta-
tion (LoRA) [Hu et al., 2022], which freezes the pre-trained
model weights and injects trainable rank decomposition ma-
trices into each layer of the Transformer [Vaswani et al.,
2017] architecture, greatly reducing the number of trainable
parameters for downstream tasks. During inference, based
on the recalled entity pairs and retrieved demonstrations, the
LLMs conduct ICL to generate predicted relations.
3
Experiments
3.1
Settings
Datasets and Metrics.
We evaluate RE4 on SemEval
2010 [Hendrickx et al., 2019], TACRED [Zhang et al., 2017],
Google RE 1, SciERC [Luan et al., 2018], four commonly
used RE datasets. Following previous works [Yamada et al.,
2020; Cabot and Navigli, 2021; Li et al., 2023a], we use the
Micro-F1 score excluding NA as the metric for evaluation.
The statistics of datasets are shown in Table 1.
Dataset
#Relation
#Train
#Dev
#Test
SemEval
9
6,507
1,493
2,717
TACRED
41
68,124
22,631
15,509
Google RE
5
38,112
9,648
9,616
SciERC
7
3,219
455
974
Table 1: Statistics of datasets.
Baselines.
We compare RE4 with state-of-the-art RE mod-
els that represent a diverse array of approaches.
Super-
vised fine-tuning (SFT) RE methods can be divided into
three categories. Classification-based methods fine-tune lan-
guage models on RE datasets with classification losses, such
as MTB [Baldini Soares et al., 2019], LUKE [Yamada et
al., 2020], IRE [Zhou and Chen, 2022] and KLG [Li et
al., 2023a]. Prompt-based methods use prompt and treats
RE as a cloze-style task, such as KnowPrompt [Chen et al.,
2022a] and NLI-DeBERTa [Sainz et al., 2021]. Generative-
based methods use text generation models for RE, such as
REBEL [Cabot and Navigli, 2021], TANL [Paolini et al.,
2021], RELA [Li et al., 2023b] and DeepStruct [Wang et al.,
2022]. For ICL-based RE, existing methods typically rely on
the strong ICL ability of large-scale proprietary models with-
out any SFT. We utilize GPT-RE [Wan et al., 2023] which
use PURE [Zhong and Chen, 2021] as the demonstration re-
triever and GPT-3 [Brown et al., 2020] as the base LLM for
ICL. RE4 can be regarded as the combination of SFT and ICL
paradigms. On the one hand, it makes LLMs better adapt to
specific tasks via instruction tuning. On the other hand, it im-
proves the in-context reasoning ability of LLMs and enables
relevant retrieved demonstrations during the ICL process.
Experiment Details.
We experiment RE4 with open-source
LLMs including T5 [Raffel et al., 2020], BART [Lewis et
al., 2020] and LLaMA [Touvron et al., 2023]. For model
scales, we select T5-Base (220M), T5-Large (770M), BART-
Base (140M), BART-Large (400M) and LLaMA-7B for ex-
periments. We utilize LoRA [Hu et al., 2022] to tune LLMs
for simplicity and efficiency. We set the rank r of the LoRA
parameters to 8 and the merging ratio α to 32. We train RE4
for 5 epochs with batch size 4 and learning rate 1e-4. For the
number of generated entity pairs, we set k to 5. The check-
point of LoRA adapter that achieves the best result on the val-
idation set is used for testing. We also directly treat relation
names as generation objectives and fine-tune the LLaMA.
1https://github.com/google-research-datasets/relation-
extraction-corpus
Method (#Param.)
SemEval TACRED Google RE SciERC
MTB (336M)
89.5
71.5
92.7
87.4
LUKE (355M)
90.1
72.7
94.0
87.7
IRE (355M)
89.8
74.6
93.1
88.9
KLG (355M)
90.5
75.6
-
-
KnowPrompt (355M)
90.2
72.4
-
-
NLI-DeBERTa (1.5B)
-
73.9
-
-
REBEL (400M)
-
73.7
93.5
86.3
TANL (220M)
-
74.8
-
-
RELA (400M)
90.4
71.2
93.9
90.3
DeepStruct (10B)
-
76.8
-
-
GPT-3 (175B)
70.1
32.5
-
-
GPT-RE (175B)
91.9
72.1
-
-
BART-Base w/ RE4
89.8
71.5
92.4
86.0
BART-Large w/ RE4
90.6
73.3
93.1
87.2
T5-Base w/ RE4
89.9
72.7
92.6
86.3
T5-Large w/ RE4
90.9
75.6
93.4
87.8
LLaMA w/ RE4
92.1
77.2
94.5
91.7
LLaMA w/o RE4
90.6
75.0
92.9
89.5
Table 2: Micro-F1 score of test sets on four RE datasets. Results of
baselines are retrieved from original papers. For ICL-based results,
we use best 30-shot on SemEval and 15-shot on TACRED. Previous
state-of-the-art results are marked with underline, and best results
are bold. Results of RE4 are averaged over three random seeds.
3.2
Main Results
The main results of baselines and RE4 are summarized in
Table 2, where RE4 with LLaMA outperforms all previous
state-of-the-art methods on four RE datasets. Compared to
the classification-based methods, RE4 shows favorable results
without any external dataset usage or additional pre-training
stages, while MTB and LUKE all involve entity and relation
related pre-training tasks. RE4 with most LLMs also sup-
presses two prompt-based methods. Compared to previous
generative-based methods, RE4 consistently achieves com-
petitive or superior results on four datasets. For example,
with same backbone BART-Large, RE4 and REBEL share
similar overall results, while REBEL is pre-trained with a
large external relational triple extraction dataset.
Notably,
RE4 with LLaMA significantly outperforms REBEL on TA-
CRED and SciERC, and vanilla fine-tuned LLaMA still sur-
passes REBEL, due to the fact that LLaMA (7B) has a much
larger number of model parameters than REBEL (400M). Al-
though LLaMA is smaller than DeepStruct which is based
on a pre-trained 10B parameter language model GLM [Du
et al., 2022] and is pre-trained on a collection of large-scale
corpus, RE4 delivers better results on TACRED than Deep-
Struct. This is notable because RE4 adopt smaller founda-
tion models for fine-tuning and utilize no external datasets for
pre-training compared to DeepStruct. In addition, RE4 with
LLaMA achieves comparable results on SemEval and much
better results on TACRED compared to ICL-based method
GPT-RE. Without fine-tuning, if GPT-3 lacks relevant domain
knowledge about specific tasks, then the performance of GPT-
RE is limited and greatly affected by the retrieved demon-
strations. And this is why GPT-RE delivers exceptional re-
3
5
7
9
Number of generated entity pairs
0.86
0.88
0.90
0.92
0.94
F1 scores
SemEval
ICL
Majority Vote
3
5
7
9
Number of generated entity pairs
0.66
0.68
0.70
0.72
0.74
0.76
0.78
0.80
F1 scores
TACRED
ICL
Majority Vote
Figure 3: The sensitivity of k. ICL denotes we perform in-context
reasoning in reasoning module with retrieved demonstrations, while
Majority Vote denotes we consider the relation with the maximum
number of generated entity pairs as the predicted relation.
Dataset
#Test
#Entity Pairs
#Valid
#Ratio
SemEval
2,717
13,585
13,023
95.86%
TACRED
15,509
77,545
73,482
94.76%
Table 3: Validness of generated entity pairs (k=5). #Valid denotes
the number of valid generated entity pairs, and #Ratio denotes the
ratio of valid entity pairs in all generated entity pairs.
sults on SemEval but unattractive results that even worse than
LUKE on TACRED. In other words, solely relying on ICL-
based methods is unstable. The main results demonstrate the
simplicity and effectiveness of RE4 compared to baselines.
3.3
Discussions
We conduct extensive experiments to verify the effectiveness
of recalling and reasoning module. We also provide detailed
analysis about retrieved demonstrations and in-context rea-
soning abilities. In this section, we only experiment RE4 with
LLaMA on SemEval and TACRED for simplicity.
Number of Generated Entity Pairs.
We vary the number
of generated entity pairs k from 3, 5, 7 to 9. We also consider
the majority vote results, the overall results are shown in Fig-
ure 3. On the one hand, we find that without in-context rea-
soning, the performance are more sensitive to the k-selection.
With the help of ICL, RE4 could achieve more stable and
much better performances, which highlights the importance
of reasoning module. On the other hand, obviously, setting
k to 5 delivers the best results across two datasets. While in-
creasing k generally boosts the performance of majority vote,
the overall performance of RE4 tends to stabilize, even with
a slight decrease. This is because in-context reasoning is sen-
sitive to the distractors (i.e., additional retrieved demonstra-
tions that are not relevant to a test example) [Shi et al., 2023].
Therefore, setting larger k not only increases difficulty in gen-
erating accurately consistently ontological entities, but brings
more distractors that are negative for reasoning module.
Quality of Generated Entity Pairs.
We consider two as-
pects of generated entity pairs for quality checking. We first
examine the validness of generated entity pairs (i.e., whether
it is grounded by retrieval corpora), then we evaluate the rel-
evance of retrieved demonstrations with respect to test exam-
ples. We match the generated entity pairs with the entities in
training examples, the results are shown in Table 3. It can be
Dataset
5
4
3
2
1
0
SemEval
71.03%
7.45%
6.49% 5.45%
5.21%
4.37%
TACRED 51.88% 12.75% 5.20% 9.74% 12.26% 8.17%
Table 4: The number of retrieved examples which share same rela-
tion with a test example (k = 5). We consider all the examples in test
sets and calculate the proportion.
3
5
7
9
Number of demonstrations
0.50
0.55
0.60
0.65
0.70
0.75
0.80
0.85
0.90
F1 scores
SemEval
Sentence-BERT
SimCSE
PURE
Recall
3
5
7
9
Number of demonstrations
0.30
0.35
0.40
0.45
0.50
0.55
0.60
0.65
0.70
F1 scores
TACRED
Sentence-BERT
SimCSE
PURE
Recall
Figure 4: Comparison on different retrieval models.
seen that under strict exact matching, the proportion of valid
generated entity pairs still reaches around 95%, which indi-
cates that the entity pairs generated by recalling module could
serve as the faithful queries for retrieval corpora. However,
we are still interested in those invalid entity pairs. Take the
SemEval dataset as the example, we analyze the remaining
4.14% invalid entity pairs, discovering that 70 of 562 entity
pairs at least one head or tail entity is grounded by retrieval
corpora. The other 492 generated entity pairs all have valid
entities, but cannot express a valid relation. Overall, during
entity pairs generation, the recalling module reduces the hal-
lucination issue of the LLMs and generates faithful queries.
To evaluate the relevance of retrieved demonstrations with
respect to test examples, we consider the number of exam-
ples which share same relation with a test example (i.e. the
consistency of ontology). The results are summarized in Ta-
ble 4. For most cases, the entity pairs generated by the re-
calling module and the demonstrations retrieved by retriev-
ing module are all relevant (i.e., all 5 demonstrations share
same relation with a test example). Note that 84.97% and
69.83% of retrieved results ensure the majority of golden re-
lation examples participation. And these demonstrations can
provide relevant contextual knowledge for ICL. However, one
drawback of recalling is that there may be no golden relation
in the generated results. Existing similarity-based retrievers
such as Sentence-BERT [Reimers and Gurevych, 2019], Sim-
CSE [Gao et al., 2021] and PURE [Zhong and Chen, 2021]
essentially cannot guarantee that the retrieved demonstrations
would always contain the golden relation. The relevance of
retrieved demonstrations actually rely on the performance of
the retrieval model. Therefore, we evaluate the quality of re-
trieved demonstrations from recalling module and similarity
calculation via ICL on GPT-3 2, as shown in Figure 4. We
find that fine-tuned retrievers (PURE and recalling module)
2For GPT-3, we use “text-davinci-003”. For Sentence-
BERT, we use “all-mpnet-base-v2”. For SimCSE, we use
“sup-simcse-bert-base-uncased”.
For PURE, we use
“bert-base-uncased” as the backbone.
Training
Inference
SemEval
TACRED
I
II
Avg.
I
II
Avg.
w/o tuning
Direct / ICL
0.0
0.0
0.0
0.0
0.0
0.0
w/ fine-tuning
Direct
90.7 90.1 90.4 74.6 75.2 74.9
w/ ICL tuning
ICL
91.4 89.8 90.6 76.7 73.7 75.2
Table 5: Comparison between vanilla fine-tuning and ICL tuning. I
denotes the model performance on test examples that their demon-
strations contain at least one golden relation example, while II de-
notes the performance when the random selected demonstrations are
all distractors. Avg. represents the average score of I and II results.
substantially achieve much better ICL results than similarity-
based models without considering entity and relation seman-
tics. For different number of demonstrations, recalling mod-
ule delivers comparable results compared to PURE, showing
the effectiveness of recalling module and allowing seamless
integration with any arbitrary LLMs during inference.
Effectiveness of ICL Tuning.
We remove the recalling in-
struction task and keep the reasoning instruction task. Not
relying on the retrieved demonstrations Z∗, we use 5 training
examples as demonstrations, sampled uniformly at random,
during both training and testing time.
Note that we relax
the assumption of perfect balance between labels on train-
ing examples. We categorize two types of test examples and
then compare the performance of RE4 with vanilla fine-tuned
LLaMA, the results are shown in Table 5. First, without any
tuning process, LLaMA cannot perform ICL in RE. We em-
pirically discover that LLaMA is unable to understand the
structural sentences in standard ICL paradigm and recover
the relation labels of test examples based on demonstrations.
Second, although the distractors can lead LLMs to make inac-
curate predictions, the improvement brought by ICL tuning is
still obvious. Experimental results suggest that the impact of
ICL training is positive when the golden relation is included
in retrieved demonstrations compared to vanilla fine-tuning,
which also highlights the importance and performance of re-
trieval models. During instruction tuning, the model learns to
learn in-context for deducing the golden relation.
Sensitivity of In-Context Reasoning.
We have validated
that relevant contextual knowledge for test examples is ben-
eficial for overall performance. But we might be interested
in whether all retrieved demonstrations share similar ontol-
ogy are equal during ICL. Specifically, for each test example,
we replace the corresponding k generated entity pairs with
other k random entity pairs but share same relations. The
in-context reasoning results are shown in Figure 5. Similar
with other LLMs such as GPT-3, the change of demonstra-
tions also have an impact on the final results. Compared to
demonstrations obtained via generated entity pairs, the re-
placement operation consistently brings slightly performance
drop across different k and datasets, which indicates that the
recalling module actually help to discover similar demonstra-
tions with test examples. However, the performance degrada-
tion caused by this randomness is relatively small compared
to performance in GPT-3 [Zhao et al., 2021]. The ICL tuning
process forces the LLMs to deduce the golden relation based
on important entities and relations, which indicates that RE4
3
5
7
9
Number of demonstrations
0.86
0.88
0.90
0.92
0.94
F1 scores
SemEval
Original
Replaced
3
5
7
9
Number of demonstrations
0.66
0.68
0.70
0.72
0.74
0.76
0.78
0.80
F1 scores
TACRED
Original
Replaced
Figure 5: Sensitivity of in-context reasoning.
is able to perform modestly robust reasoning utilizing rele-
vant but imperfect contextual knowledge.
4
Related Work
The large language models (LLMs), such as GPT-3 [Brown
et al., 2020] and ChatGPT [OpenAI, 2022], perform well
in various downstream tasks without any training or fine-
tuning but only with a few examples as instructions, which
is called in-context learning (ICL). However, ICL with LLMs
achieves poor performance in relation extraction (RE) where
the main obstacles are two-fold: (1) the low relevance re-
garding entity and relation in the retrieved demonstrations
for ICL, and (2) the failure of utilizing LLMs with moder-
ate size for ICL. For the first challenge, existing attempts rely
on sentence embedding in retrieval, including the sentence
encoders such as Sentence-BERT [Reimers and Gurevych,
2019] and SimCSE [Gao et al., 2021]. Considering entity
and relation semantics, GPT-RE [Wan et al., 2023] fine-
tunes PURE [Zhong and Chen, 2021] to provide more RE-
specific and robust representations for retrieval.
For the
second challenge, although ICL has been further improved
by later work [Zhao et al., 2021; Holtzman et al., 2021;
Min et al., 2022a] and shows promising results on a variety of
tasks, these researches mainly focus on GPT-3. To improve
the ICL ability of other LLMs, current methods propose to
make LLMs perform better ICL via meta learning [Chen et
al., 2022b] and multi-task learning [Min et al., 2022b]. In-
spired by these, our method can be viewed as the integration
framework of retrieving and reasoning by instruction tuning.
Guided by ontological knowledge, RE4 generates some pos-
sible similar entities and relations, and then performs reliable
in-context reasoning based on the retrieved demonstrations.
5
Conclusion
In this work, we propose RE4, a novel recall-retrieve-reason
RE framework for open-source LLMs, which recalls relevant
entity pairs, retrieves good demonstrations and then perform
better in-context reasoning for RE. We consider distilling
consistently ontological knowledge to guide the demonstra-
tion retrieval, and tuning LLMs with ICL objective to perform
reliable in-context reasoning. Specially, RE4 allows seamless
integration with any arbitrary LLMs during inference. Em-
pirical results show that RE4 achieves new state-of-the-art
sentence-level RE performance in four RE benchmarks. We
also demonstrate its effectiveness with extensive experiments,
and discuss its advantages and limitations, encouraging more
effective generative-based RE methods in the future research.
Acknowledgments
We thank the reviewers for their insightful comments. This
work was supported by National Science Foundation of China
(Grant Nos.62376057) and the Start-up Research Fund of
Southeast University (RF1028623234). All opinions are of
the authors and do not reflect the view of sponsors.
References
[Agrawal et al., 2022] Monica
Agrawal,
Stefan
Hegsel-
mann, Hunter Lang, et al. Large language models are few-
shot clinical information extractors. In EMNLP, 2022.
[Arora et al., 2023] Simran
Arora,
Avanika
Narayan,
Mayee F Chen, et al. Ask me anything: A simple strategy
for prompting language models. In ICLR, 2023.
[Baldini Soares et al., 2019] Livio Baldini Soares, Nicholas
FitzGerald, Jeffrey Ling, et al. Matching the blanks: Dis-
tributional similarity for relation learning. In ACL, 2019.
[Brown et al., 2020] Tom Brown, Benjamin Mann, Nick Ry-
der, et al.
Language models are few-shot learners.
In
NeurIPS, 2020.
[Cabot and Navigli, 2021] Pere-Llu´ıs
Huguet
Cabot
and
Roberto Navigli. Rebel: Relation extraction by end-to-end
language generation. In Findings of EMNLP, 2021.
[Chen et al., 2022a] Xiang Chen, Ningyu Zhang, Xin Xie,
et al. Knowprompt: Knowledge-aware prompt-tuning with
synergistic optimization for relation extraction. In WWW,
2022.
[Chen et al., 2022b] Yanda Chen, Ruiqi Zhong, Sheng Zha,
et al. Meta-learning via language model in-context tuning.
In ACL, 2022.
[Chen et al., 2023] Zeming Chen, Gail Weiss, Eric Mitchell,
et al. Reckoning: Reasoning through dynamic knowledge
encoding. In NeurIPS, 2023.
[Coda-Forno et al., 2023] Julian Coda-Forno, Marcel Binz,
Zeynep Akata, et al. Meta-in-context learning in large lan-
guage models. In NeurIPS, 2023.
[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Ken-
ton Lee, et al.
Bert: Pre-training of deep bidirectional
transformers for language understanding. In NAACL-HLT,
2019.
[Du et al., 2022] Zhengxiao Du, Yujie Qian, Xiao Liu, et al.
GLM: General language model pretraining with autore-
gressive blank infilling. In ACL, 2022.
[Gao et al., 2021] Tianyu Gao, Xingcheng Yao, and Danqi
Chen.
Simcse: Simple contrastive learning of sentence
embeddings. In EMNLP, 2021.
[Hendrickx et al., 2019] Iris Hendrickx, Su Nam Kim, Zor-
nitsa Kozareva, et al. Semeval-2010 task 8: Multi-way
classification of semantic relations between pairs of nomi-
nals. arXiv preprint arXiv:1911.10422, 2019.
[Holtzman et al., 2021] Ari Holtzman, Peter West, Vered
Shwartz, et al. Surface form competition: Why the highest
probability answer isn’t always right. In EMNLP, 2021.
[Hu et al., 2022] Edward J Hu, Yelong Shen, Phillip Wallis,
et al. Lora: Low-rank adaptation of large language models.
In ICLR, 2022.
[Ji et al., 2023] Ke Ji, Yixin Lian, Jingsheng Gao, and
Baoyuan Wang. Hierarchical verbalizer for few-shot hi-
erarchical text classification. In ACL, 2023.
[Jimenez Gutierrez et al., 2022] Bernal Jimenez Gutierrez,
Nikolas McNeal, Clayton Washington, et al.
Thinking
about GPT-3 in-context learning for biomedical IE? think
again. In Findings of EMNLP, 2022.
[Lewis et al., 2020] Mike Lewis, Yinhan Liu, Naman Goyal,
et al. BART: Denoising sequence-to-sequence pre-training
for natural language generation, translation, and compre-
hension. In ACL, 2020.
[Li et al., 2022] Guozheng Li, Xu Chen, Peng Wang, et al.
Fastre: Towards fast relation extraction with convolutional
encoder and improved cascade binary tagging framework.
In IJCAI, 2022.
[Li et al., 2023a] Bo Li, Wei Ye, Jinglei Zhang, et al. Re-
viewing labels: Label graph network with top-k prediction
set for relation extraction. In AAAI, 2023.
[Li et al., 2023b] Bo Li, Dingyao Yu, Wei Ye, et al.
Se-
quence generation with label augmentation for relation ex-
traction. In AAAI, 2023.
[Li et al., 2023c] Guozheng Li, Peng Wang, and Wenjun Ke.
Revisiting large language models as zero-shot relation ex-
tractors. In Findings of EMNLP, 2023.
[Li et al., 2024] Guozheng Li, Wenjun Ke, Peng Wang, Zi-
jie Xu, Ke Ji, Jiajun Liu, Ziyu Shang, and Qiqing Luo.
Unlocking instructive in-context learning with tabular
prompting for relational triple extraction. arXiv preprint
arXiv:2402.13741, 2024.
[Liu et al., 2019] Yinhan Liu, Myle Ott, Naman Goyal, et al.
Roberta: A robustly optimized bert pretraining approach.
arXiv preprint arXiv:1907.11692, 2019.
[Liu et al., 2022] Jiachang Liu, Dinghan Shen, Yizhe Zhang,
et al. What makes good in-context examples for GPT-3?
In DeeLIO, 2022.
[Liu et al., 2024] Jiajun Liu, Wenjun Ke, Peng Wang, Ziyu
Shang, Jinhua Gao, Guozheng Li, Ke Ji, and Yanhe Liu.
Towards continual knowledge graph embedding via incre-
mental distillation. In AAAI, 2024.
[Luan et al., 2018] Yi Luan, Luheng He, Mari Ostendorf,
et al. Multi-task identification of entities, relations, and
coreference for scientific knowledge graph construction.
arXiv preprint arXiv:1808.09602, 2018.
[Luo et al., 2023] Linhao Luo, Yuan-Fang Li, Gholamreza
Haffari, et al. Reasoning on graphs: Faithful and inter-
pretable large language model reasoning. arXiv preprint
arXiv:2310.01061, 2023.
[Ma et al., 2023] Yubo Ma, Yixin Cao, YongChing Hong,
et al. Large language model is not a good few-shot in-
formation extractor, but a good reranker for hard samples!
In Findings of EMNLP, 2023.
[Min et al., 2022a] Sewon Min, Mike Lewis, Hannaneh Ha-
jishirzi, et al. Noisy channel language model prompting
for few-shot text classification. In ACL, 2022.
[Min et al., 2022b] Sewon Min, Mike Lewis, Luke Zettle-
moyer, et al. Metaicl: Learning to learn in context. In
NAACL-HLT, 2022.
[OpenAI, 2022] OpenAI. Introducing chatgpt, 2022.
[Paolini et al., 2021] Giovanni Paolini, Ben Athiwaratkun,
Jason Krone, et al. Structured prediction as translation be-
tween augmented natural languages. In ICLR, 2021.
[Radford et al., 2019] Alec Radford, Jeffrey Wu, Rewon
Child, et al. Language models are unsupervised multitask
learners. OpenAI blog, 1(8):9, 2019.
[Raffel et al., 2020] Colin Raffel, Noam Shazeer, Adam
Roberts, et al. Exploring the limits of transfer learning
with a unified text-to-text transformer. JMLR, 21(1):5485–
5551, 2020.
[Reimers and Gurevych, 2019] Nils
Reimers
and
Iryna
Gurevych.
Sentence-bert: Sentence embeddings using
siamese bert-networks. In EMNLP, 2019.
[Roche, 2003] Christophe Roche. Ontology: a survey. IFAC
Proceedings Volumes, 36(22):187–192, 2003.
[Sainz et al., 2021] Oscar Sainz, Oier Lopez de Lacalle,
Gorka Labaka, et al. Label verbalization and entailment
for effective zero-and few-shot relation extraction.
In
EMNLP, 2021.
[Shang et al., 2024] Ziyu Shang, Wenjun Ke, Nana Xiu,
Peng Wang, Jiajun Liu, Yanhui Li, Zhizhao Luo, and Ke Ji.
Ontofact: Unveiling fantastic fact-skeleton of llms via
ontology-driven reinforcement learning. In AAAI, 2024.
[Shi et al., 2023] Freda Shi, Xinyun Chen, Kanishka Misra,
et al. Large language models can be easily distracted by
irrelevant context. In ICML, 2023.
[Touvron et al., 2023] Hugo Touvron, Thibaut Lavril, Gau-
tier Izacard, et al. Llama: Open and efficient foundation
language models. arXiv preprint arXiv:2302.13971, 2023.
[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki
Parmar, et al. Attention is all you need. In NIPS, 2017.
[Wan et al., 2023] Zhen Wan, Fei Cheng, Zhuoyuan Mao,
et al. GPT-RE: In-context learning for relation extraction
using large language models. In EMNLP, 2023.
[Wang et al., 2022] Chenguang Wang, Xiao Liu, Zui Chen,
et al. Deepstruct: Pretraining of language models for struc-
ture prediction. In Findings of ACL, 2022.
[Wang et al., 2023a] Peng
Wang,
Tong
Shao,
Ke
Ji,
Guozheng Li, and Wenjun Ke. fmlre: a low-resource rela-
tion extraction model based on feature mapping similarity
calculation. In AAAI, 2023.
[Wang et al., 2023b] Peng Wang, Jiafeng Xie, Xiye Chen,
et al. Pascore: a chinese overlapping relation extraction
model based on global pointer annotation strategy. In IJ-
CAI, 2023.
[Wei et al., 2022a] Jason Wei, Maarten Bosma, Vincent Y
Zhao, et al.
Finetuned language models are zero-shot
learners. In ICLR, 2022.
[Wei et al., 2022b] Jason Wei, Xuezhi Wang, Dale Schuur-
mans, et al. Chain of thought prompting elicits reasoning
in large language models. In NeurIPS, 2022.
[Yamada et al., 2020] Ikuya Yamada, Akari Asai, Hiroyuki
Shindo, et al. Luke: deep contextualized entity represen-
tations with entity-aware self-attention. In EMNLP, 2020.
[Zhang et al., 2017] Yuhao Zhang, Victor Zhong, Danqi
Chen, et al. Position-aware attention and supervised data
improve slot filling. In EMNLP, 2017.
[Zhao et al., 2021] Zihao Zhao, Eric Wallace, Shi Feng, et al.
Calibrate before use: Improving few-shot performance of
language models. In ICML, 2021.
[Zhong and Chen, 2021] Zexuan Zhong and Danqi Chen. A
frustratingly easy approach for entity and relation extrac-
tion. In NAACL-HLT, 2021.
[Zhou and Chen, 2022] Wenxuan Zhou and Muhao Chen.
An improved baseline for sentence-level relation extrac-
tion. In AACL, 2022.


Question:
According to the analysis of error categories and their effects, which of the following models demonstrate improved robustness to specific noise types in the Financial domain?
A. MAC-SQL shows higher accuracy on questions with spelling/syntactical errors due to its schema filtering and context enrichment mechanisms.
B. DIN-SQL outperforms zero-shot GPT-4 on ambiguous questions requiring nested queries and external knowledge integration.
C. Zero-shot GPT-4 achieves better performance on corrected SQL queries but struggles with string capitalization errors in dirty database values.
D. Zero-shot GPT-3.5 exhibits reduced accuracy on questions with synonyms due to limited contextual understanding of domain-specific terminology.

Answer: AC

We apply models to the original dataset, a dataset where SQL has been corrected, and a dataset where both SQL queries and noisy questions have been corrected. Figure 2 presents the results of a single evaluation for all models on all datasets.

MAC-SQL slightly outperforms DIN-SQL and the zero-shot baselines on the original dataset, where noise exists in both questions and queries. However, correcting SQL queries decreases MAC-SQL’s performance, tying it with DIN-SQL as the poorest performers. Surprisingly, even the zero-shot GPT-3.5 baseline outperforms the more advanced DIN-SQL and MAC-SQL. The dataset with corrected SQL queries could also be considered optimal since gold labels are correct and noise in questions is represented. Given the drastic re-ranking of models, it is relevant to question if BIRD-Bench is a reliable assessor of models and a useful tool to assist researchers in developing new methods for Text-to-SQL.

When evaluating models on the dataset with both questions and SQL queries corrected, the accuracy of all models increases significantly. While zero-shot GPT-4 performs the best, the remaining models perform similarly with DIN-SQL slightly ahead. Compared to the ideal scenario where only SQL queries are corrected, the presence of noise noticeably impacts all models’ accuracy. However, models are not equally affected by noise as some models have a more pronounced increase in accuracy. This might suggests that the types of noise also affects models differently, which needs to be studied further.

Example 4: Synonyms
Figure 8 demonstrates how specific wordings can complicate interpretation for an LLM. The term sum, being both a SQL keyword and a descriptor, led to the LLM’s literal interpretation and the incorrect summation of a transaction. The actual intent was to inquire about the transaction’s balance or amount. A rephrased question resulted in the LLM generating the correct SQL query, fetching the intended information, as seen in the figure.
...
Figure 8: Question from data point with ID 177 from the development set of BIRD-Bench containing a difficult synonym, a corrected version of the question with the synonym replaced and corresponding predicted SQL queries by the DIN-SQL (GPT-3.5) model described in Section 3.3. Showcases the difficulty of synonyms on model predictions.

Example 5: String Capitalization
As a consequence of SQL being a case-sensitive language when comparing string values in a query, the way a question is formulated regarding the use of uppercase or lowercase letters when asking for a specific value affects the result. This is because the LLM will most likely use the specific entry as given when generating the query, unless it has knowledge of the case used for different entries in the database. Therefore, in Figure 9, an example is provided where the terms "East" and "North" are mentioned with initial capital letters, as is commonly the case. However, the entries for these column values are in lowercase in the database, which means the question needs to account for this for the LLM to be able to generate a correct query. The corrected question and the SQL query generated from it can also be seen in Figure 9.

Understanding the Effects of Noise in Text-to-SQL:
An Examination of the BIRD-Bench Benchmark
Niklas Wretblad1,*
Fredrik Gordh Riseby1,*
Rahul Biswas2
Amin Ahmadi2
Oskar Holmström1
1Linköping University, 2Silo AI
niklas.wretblad@liu.se
Abstract
Text-to-SQL, which involves translating natu-
ral language into Structured Query Language
(SQL), is crucial for enabling broad access to
structured databases without expert knowledge.
However, designing models for such tasks is
challenging due to numerous factors, includ-
ing the presence of ’noise,’ such as ambiguous
questions and syntactical errors. This study pro-
vides an in-depth analysis of the distribution
and types of noise in the widely used BIRD-
Bench benchmark and the impact of noise on
models. While BIRD-Bench was created to
model dirty and noisy database values, it was
not created to contain noise and errors in the
questions and gold queries.
We found that
noise in questions and gold queries are preva-
lent in the dataset, with varying amounts across
domains, and with an uneven distribution be-
tween noise types. The presence of incorrect
gold SQL queries, which then generate incor-
rect gold answers, has a significant impact on
the benchmark’s reliability. Surprisingly, when
evaluating models on corrected SQL queries,
zero-shot baselines surpassed the performance
of state-of-the-art prompting methods. We con-
clude that informative noise labels and reliable
benchmarks are crucial to developing new Text-
to-SQL methods that can handle varying types
of noise. All datasets, annotations, and code
are available at this URL.
1
Introduction
Text-to-SQL through large language models facili-
tates broader access to structured databases without
requiring expert knowledge. To develop such mod-
els, high-quality open datasets and benchmarks
are essential resources, and over the years, several
benchmarks and datasets have been created. Early
benchmarks, such as WikiSQL (Zhong et al., 2017),
modeled simple scenarios, often with single-table
queries, and following datasets attempts to closer
*Equal Contribution
- What is the average loan amount by male borrowers?
SELECT AVG(T3.amount) FROM client AS T1
INNER JOIN account AS T2 ON T1.district_id = T2.district_id
INNER JOIN loan AS T3 ON T2.account_id = T3.account_id
WHERE T1.gender = 'M'
SELECT AVG(T1.amount) FROM loan AS T1
INNER JOIN account AS T2 ON T1.account_id = T2.account_id
INNER JOIN disp AS T3 ON T2.account_id = T3.account_id
INNER JOIN client AS T4 ON T3.client_id = T4.client_id
WHERE T4.gender = 'M'
 Incorrect Gold Query
 Corrected Query
 Question
Figure 1: Example of an incorrect SQL query that gener-
ates the wrong gold reference answer for the given ques-
tion. The JOIN operation incorrectly matches clients
and accounts by district_id. Due to the possibility of
multiple clients and accounts in the same district, ac-
counts are incorrectly associated with the wrong users.
approximate real-world scenarios: complex queries
with join-statements over several tables (Yu et al.,
2018), unseen domain-specific datasets (Gan et al.,
2021b) (Lee et al., 2021), and noisy questions (Gan
et al., 2021a). BIRD-Bench, a recent and chal-
lenging benchmark, aims to further close the gap
between text-to-SQL research and real-world appli-
cations by for example containing large and dirty
database values and requiring external knowledge
(Li et al., 2023).
While BIRD-Bench does not explicitly introduce
noise to the questions in the data, it could be that
it is added inadvertently due to human error. For
the same reason, noise is an essential aspect of
real-world use cases, as human inputs often are
ambiguous and contain syntactical errors. However,
for the benchmark to be a helpful tool for judging
model properties, such as noise handling, the data
must be valid and inform us in what areas a model
can be improved.
This paper continues the tradition of examining
the suitability and limitations of open datasets and
benchmarks. We specifically focus on how noise
is represented in questions and queries in BIRD-
arXiv:2402.12243v4  [cs.CL]  25 Mar 2024
Bench. We perform a qualitative analysis of what
types of noise exist in the data and the noise dis-
tribution in specific domains. We then study the
effects of noise on different models and prompting
techniques, using both strong baselines and state-
of-the-art methods.
We find that noise in questions and queries is
prevalent, noise is unevenly distributed across do-
mains, and categories of noise types are represented
unequally in the data. Errors in gold SQL queries
are also common and decrease the reliability of
BIRD-Bench. When evaluating models on a dataset
with corrected gold queries, the performance gap
between zero-shot baselines and state-of-the-art
prompting techniques is closed, questioning how
we should interpret model performance on BIRD-
Bench.
2
Related Work
Datasets
WikiSQL is a large Text-to-SQL dataset
containing only simple SELECT and WHERE op-
erations without nested queries or JOIN opera-
tions (Zhong et al., 2017). SPIDER (Yu et al.,
2018) was later developed to approximate real-life
scenarios more closely, requiring models to con-
struct complex queries and understand the database
schema. While complexity is a critical aspect of
real use cases, variations of SPIDER have been cre-
ated to contain noisy questions (Gan et al., 2021a)
and domain-specific questions (Gan et al., 2021b).
BIRD-Bench was created to close the gap between
academic research and real-world applications by
introducing large and dirty database values, ques-
tions requiring external knowledge and optimizing
SQL execution efficiency (Li et al., 2023).
Text-to-SQL Methods
The notable gap in ac-
curacy between automated systems (65.45%) and
human experts (92.96%)1, highlights the need for
ongoing developments in Text-to-SQL models.
Different approaches have been taken to cre-
ate models capable of Text-to-SQL generation. A
more traditional approach is to finetune LLMs on
Text-to-SQL examples. While these models offer
promising results, there is a performance gap to
instruction-tuned LLMs, in particular GPT-4, that
is adapted to the Text-to-SQL task through prompt
engineering (Li et al., 2023). Prompts are often
chained, where each prompt is applied to the task
1BIRD-Bench benchmark as of 2024-02-16 (https://bird-
bench.github.io)
sub-problems, such as schema linking, decompo-
sition of queries, and refinement of model gen-
erations (Pourreza and Rafiei, 2023; Wang et al.,
2023).
Noise in Datasets
The contemporaneous work
of Wang et al. (2023) shows that ambiguous ques-
tions and incorrect SQL queries exist in BIRD-
Bench. However, unlike our work, they do not
study how noise varies across domains or how
the identified noise and errors affect model per-
formance. Katsogiannis-Meimarakis and Koutrika
(2023) points out that database schemas often mis-
align with data entities, which may cause lexical or
syntactic ambiguities affecting text-to-SQL mod-
els.
3
Method
3.1
Data
The BIRD-Bench dataset (Li et al., 2023) is used in
this paper as it is a recent and widely used dataset
that is the most similar to real world scenarios.
Since BIRD contains 12,751 samples across many
domains and because of time-consuming human
annotation, the main focus of the analysis is on the
financial domain2, which includes queries related
to banking operations.
The development set of the financial domain
contains 106 question and SQL query pairs, which
represent approximately 7.5% of the data points in
the development set and are structured around eight
distinct tables, presented in full in Appendix A.1.
Each question is annotated with a difficulty level
(simple, moderate, and challenging). The specific
distribution is found in Figure 4, in Appendix A.2.
We selected four additional domains to validate
our noise analysis of the financial domain and per-
formed the same analysis on 20 randomly sampled
questions from each domain. The selection was
based on question difficulties and model accuracy
of DIN-SQL3 on each of the domains, as presented
in Figure 4. We selected California Schools with
low accuracy and simple questions, Superhero with
high accuracy and simple questions, Toxicology
with similar accuracy to the financial domain but
more complex questions, and Thrombosis Predic-
tion with low accuracy and moderately difficult
questions.
2This was also motivated by the fact this paper was a
collaborative endeavor with the Swedish bank SEB.
3Results of DIN-SQL across domains were provided by
the creators of DIN-SQL.
Statistic
Financial
California Schools
Superhero
Toxicology
Thrombosis Prediction
Data points with noise
52/106 (49%)
9/20 (45%)
3/20 (15%)
7/20 (35%)
8/20 (40%)
Noisy questions
44/106 (41.5%)
5/20 (25%)
2/20 (10%)
6/20 (30%)
3/20 (15%)
Erroneous gold queries
22/106 (20.7%)
8/20 (40%)
1/20 (5%)
2/20 (10%)
6/20 (30%)
Table 1: Statistics of the total amount of data points that contains errors and the amount of errors in questions and
gold queries across five datasets. Note that a data point can have errors in both the question and SQL query.
Noise Type
Financial
California Schools
Superhero
Toxicology
Thrombosis Prediction
Spelling/Syntactical Errors
23
2
1
4
2
Vague/Ambiguous Questions
17
1
1
1
1
Incorrect SQL
22
8
1
2
6
Synonyms
2
0
0
0
0
String Capitalization
7
0
0
0
0
Question does not map to DB
1
4
1
0
0
Total number of errors
72
15
4
7
9
Table 2: Distribution of different types of noise encountered in the domains.
3.2
Annotation of Noise
All questions and SQL queries in the selected do-
mains were annotated to determine whether they
contained errors. The annotations were performed
independently by two authors of this paper, fluent
in English and experts in SQL. The identified errors
were grouped based on similarity and named after
the errors’ common properties, as shown in Table 2.
The annotations were then used to generate two dis-
tinct datasets: one where SQL was corrected, and
one where both SQL queries and noisy questions
were corrected. We have published all annotations
and cleaned datasets.4
3.3
Models and Prompt Techniques
Two models, GPT-3.5 and GPT-4, were used
with three different prompting methods: zero-shot
prompting as a baseline and the more advanced
DIN-SQL (Pourreza and Rafiei, 2023) and MAC-
SQL (Wang et al., 2023). We used GPT-3.5 and
GPT-4 for zero-shot prompting, but for the ad-
vanced prompting techniques, we only used GPT-
3.5 since chaining prompts with GPT-4 was out-
side of the available resources for this project. We
chose the models and prompting methods because
they were the highest-performing publicly avail-
able models on BIRD-Bench at the time of writing.
Information about the database schema is crucial
to generating correct queries for BIRD-Bench ques-
tions. DIN-SQL and MAC-SQL has a predefined
format for adding the database schema. For the
4https://github.com/niklaswretblad/the-effects-of-noise-
in-text-to-SQL
zero-shot model, we provide the database schema
in-context in the form of SQL table creation state-
ments, as this has been shown to improve accuracy
compared to other formats (Nan et al., 2023). The
prompt template for the zero-shot model is found
in Appendix A.3 and the code for running experi-
ments is publicly accessible.4
4
Qualitative Analysis of Noise
Even though BIRD-Bench is not intended to rep-
resent noise in questions and SQL queries, our
analysis reveals that noise exists in all studied do-
mains to different extents. The financial domain
exhibits the highest levels of noise at 49% closely
followed by the California Schools domain at 45%,
as shown in Table 1. In contrast, the Superhero
domain demonstrated the lowest noise levels, with
only 15% of data points containing errors. As pre-
sented in Section 3.1 and Figure 4, the Superhero
domain had the highest accuracy while having a
similar distribution of question difficulties. This
could indicate that model accuracy across tasks
correlates with noise, which implies that noise in
questions and SQL queries need to be carefully
considered during dataset design.
The categories and absolute frequency of noise
per dataset are presented in Table 2, and both ex-
amples and descriptions of the noise types are pre-
sented in Appendix A.4. Our analysis shows that
spelling/syntactic errors and incorrect SQL queries
were most prevalent in the financial domain. The
presence of noise in questions is not necessarily
undesirable, as it more closely mimics real-life
Figure 2: Accuracy of various models on Bird-Bench’s financial domain. Models are evaluated on the original data
(left), corrected SQL queries (middle), and corrected SQL queries and corrected noisy questions.
scenarios. However, noise distribution across the
categories is unequal. While this could be the real-
world distribution, it might unfairly bias the bench-
mark towards models better at handling syntactical
errors. Given the uneven distribution of errors and
the lack of noise labels, the benchmark does not in-
form which noise types are challenging for current
models and how models should improve.
A more severe issue is that all domains contained
incorrect SQL queries, which are used for gener-
ating gold reference answers. An example of an
erroneous SQL query is shown in Figure 1. These
types of errors question the reliability of the bench-
mark to accurately determine model performance,
which is explored in the next section.
5
Impact of Noise on Model Performance
We apply models to the original dataset, a dataset
where SQL has been corrected, and a dataset where
both SQL queries and noisy questions have been
corrected. Figure 2 presents the results of a single
evaluation for all models on all datasets.
MAC-SQL slightly outperforms DIN-SQL and
the zero-shot baselines on the original dataset,
where noise exists in both questions and queries.
However, correcting SQL queries decreases MAC-
SQL’s performance, tying it with DIN-SQL as the
poorest performers. Surprisingly, even the zero-
shot GPT-3.5 baseline outperforms the more ad-
vanced DIN-SQL and MAC-SQL. The dataset with
corrected SQL queries could also be considered op-
timal since gold labels are correct and noise in ques-
tions is represented. Given the drastic re-ranking
of models, it is relevant to question if BIRD-Bench
is a reliable assessor of models and a useful tool to
assist researchers in developing new methods for
Text-to-SQL.
When evaluating models on the dataset with both
questions and SQL queries corrected, the accuracy
of all models increases significantly. While zero-
shot GPT-4 performs the best, the remaining mod-
els perform similarly with DIN-SQL slightly ahead.
Compared to the ideal scenario where only SQL
queries are corrected, the presence of noise no-
ticeably impacts all models’ accuracy. However,
models are not equally affected by noise as some
models have a more pronounced increase in accu-
racy. This might suggests that the types of noise
also affects models differently, which needs to be
studied further.
6
Conclusions and Future Work
This paper analyzed the quality and distribution of
noise in the BIRD-Bench benchmark for Text-to-
SQL. We show that noise in both questions and
SQL queries are prevalent, and noise is unevenly
distributed across noise types and domains. Errors
in gold SQL queries were common, decreasing the
reliability of BIRD-Bench. Surprisingly, when eval-
uating models on corrected gold queries, zero-shot
baselines surpassed more advanced prompting tech-
niques. These findings highlight the necessity for
developing benchmarks that can guide researchers
in designing models that are more resistant to noise.
Therefore, a significant improvement would be to
label noise types across the dataset. In future work,
we plan to study how large language models can be
applied to noise classification, a new task that could
also be critical in systems where Text-to-SQL is
employed.
Overall, this study provides a deeper understand-
ing of how noise is expressed in Text-to-SQL tasks
and how noise and models interact, pinpointing
areas for improvement in the BIRD-Bench dataset.
Limitations
While our study provides valuable insights regard-
ing the influence of dataset noise in Text-to-SQL
translation tasks, it has several limitations. As the
analysis was performed mainly on the BIRD-Bench
dataset’s financial domain, our findings’ generaliz-
ability may be limited. We only examined a small
subset of other domains to validate our findings,
which may represent only some of the noise distri-
bution across domains.
Additionally, annotators may have introduced
subjective bias during noise annotation, even
though we attempt to minimize this by having
two independent annotators. Further, our decision
to categorize noise into six specific classes might
have oversimplified the complexity and diversity
of noise types in these benchmarks.
Our choice of models and prompting techniques
could also be a potential limitation. We only em-
ployed two models, GPT-3.5 and GPT-4, and three
different prompting methods. Evaluating a more
comprehensive array of models and prompting tech-
niques might have given a more comprehensive
understanding of their performance under the influ-
ence of noise.
Lastly, the substantial effort required to correct
SQL queries and noisy questions in the dataset may
have introduced errors despite the review process.
This might influence the model performances we
report when evaluating models on the corrected
datasets.
Acknowledgments
We extend our gratitude to Mohammadreza Pour-
reza for the results from the DIN-SQL model. We
are also grateful to SEBx for their generous sup-
port and the provision of resources. Additionally,
this research was partial funded by the National
Graduate School of Computer Science in Sweden
(CUGS).
References
Yujian Gan, Xinyun Chen, Qiuping Huang, Matthew
Purver, John R. Woodward, Jinxia Xie, and Peng-
sheng Huang. 2021a. Towards robustness of text-
to-SQL models against synonym substitution. In
Proceedings of the 59th Annual Meeting of the Asso-
ciation for Computational Linguistics and the 11th
International Joint Conference on Natural Language
Processing (Volume 1: Long Papers), pages 2505–
2515, Online. Association for Computational Lin-
guistics.
Yujian Gan, Xinyun Chen, and Matthew Purver. 2021b.
Exploring underexplored limitations of cross-domain
text-to-SQL generalization. In Proceedings of the
2021 Conference on Empirical Methods in Natural
Language Processing, pages 8926–8931, Online and
Punta Cana, Dominican Republic. Association for
Computational Linguistics.
George Katsogiannis-Meimarakis and Georgia Koutrika.
2023. A survey on deep learning approaches for text-
to-sql. The VLDB Journal, 32(4):905–936.
Chia-Hsuan Lee, Oleksandr Polozov, and Matthew
Richardson. 2021. KaggleDBQA: Realistic evalu-
ation of text-to-SQL parsers. In Proceedings of the
59th Annual Meeting of the Association for Compu-
tational Linguistics and the 11th International Joint
Conference on Natural Language Processing (Vol-
ume 1: Long Papers), pages 2261–2273, Online. As-
sociation for Computational Linguistics.
Jinyang Li, Binyuan Hui, Ge Qu, Binhua Li, Jiaxi Yang,
Bowen Li, Bailin Wang, Bowen Qin, Rongyu Cao,
Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao
Ma, Guoliang Li, Kevin C. C. Chang, Fei Huang,
Reynold Cheng, and Yongbin Li. 2023. Can llm
already serve as a database interface? a big bench
for large-scale database grounded text-to-sqls. In
Advances in Neural Information Processing Systems.
Spotlight Poster.
Linyong Nan, Yilun Zhao, Weijin Zou, Narutatsu
Ri, Jaesung Tae, Ellen Zhang, Arman Cohan, and
Dragomir Radev. 2023. Enhancing text-to-SQL capa-
bilities of large language models: A study on prompt
design strategies.
In Findings of the Association
for Computational Linguistics: EMNLP 2023, pages
14935–14956, Singapore. Association for Computa-
tional Linguistics.
Mohammadreza Pourreza and Davood Rafiei. 2023.
Din-sql: Decomposed in-context learning of text-
to-sql with self-correction. In Advances in Neural
Information Processing Systems 36. Accepted for
poster presentation, full citation details to be updated.
Bing Wang, Changyu Ren, Jian Yang, Xinnian Liang,
Jiaqi Bai, Qian-Wen Zhang, Zhao Yan, and Zhou-
jun Li. 2023. Mac-sql: A multi-agent collaborative
framework for text-to-sql.
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga,
Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingn-
ing Yao, Shanelle Roman, Zilin Zhang, and Dragomir
Radev. 2018. Spider: A large-scale human-labeled
dataset for complex and cross-domain semantic pars-
ing and text-to-SQL task. In Proceedings of the 2018
Conference on Empirical Methods in Natural Lan-
guage Processing, pages 3911–3921, Brussels, Bel-
gium. Association for Computational Linguistics.
Victor Zhong, Caiming Xiong, and Richard Socher.
2017. Seq2SQL: Generating Structured Queries from
Natural Language using Reinforcement Learning.
arXiv e-prints, page arXiv:1709.00103.
A
Appendix
A.1
Database Schema of the Financial Domain
Figure 3: Database schema of the database in the financial domain of BIRD-Bench.
Figure 3 displays the database schema for the financial domain. This schema contains various tables,
such as those for loans, transactions, accounts, cards and clients, all reflecting the financial orientation
of the database. Descriptions of what information these tables contain are presented in Table 3. The
database consists of 55 columns distributed across eight distinct tables. While the majority of the column
names are intuitively understandable, some present interpretative challenges, as evident in the schema. An
illustrative example is the district table, which incorporates 16 unique columns. This includes a column
titled district_ID along with 15 other columns, ranging from A2 to A16. The latter columns’ names do not
readily convey the nature of the data they hold, making them less intuitive to understand. In practice, a
database schema will often be accompanied by a data dictionary or documentation that explains each table
and column in detail. Such documentation would typically provide the context needed to fully understand
the meaning of each element in the schema, the range of possible values for fields with unspecified
types, and the business logic underlying the relationships. Without this additional documentation, fully
interpreting and effectively using the database can be challenging as illustrated by the column names in
the districts table. The BIRD-Bench dataset includes a unique feature for each question termed hint. This
feature is designed to offer insights or supplementary information corresponding to the specifics detailed
in such database documentation. This feature is provided to all models described in 3.3 for each question
during the experiments.
Table 3: Table descriptions of the tables in the database of the financial domain of BIRD-Bench.
Table Name
Description
loan
Contains details of loans.
order
Holds information about monetary orders.
trans
Represents financial transactions.
account
Contains account information.
disp
Links clients to accounts (dispositions).
card
Contains details about cards issued.
client
Holds client information.
district
Contains details about districts or regions.
Further, the lines in Figure 3 between the tables represent relationships, where the nature of the
relationship is indicated by the shape of the tail end of the lines where they connect to each table. A
one-to-many relationship is indicated by the line beginning with a single line and the one digit above
it, and then ending in a crow’s foot (three lines) at the opposite end. For example, an account can have
multiple orders, transactions, dispositions, and loans associated with it, but each of those entities is only
linked to one account. An account can have many loans, but one loan is exclusively only linked to one
account, which makes sense. Further, clients and accounts are related through the disposition table in a
many-to-many relationship. An account can have many different clients associated with it, for example,
one client listed as the owner of the account and multiple other clients listed as users for the account. This
could for example be practical for sharing an account in a family, where one parent could be the owner of
the account and then multiple other family members listed as users. A single client can also be related to
many different accounts in the other way around.
A.2
Distribution of question difficulty and model accuracy
Figure 4: Distribution of question difficulties and execution accuracy of the DIN-SQL model on the different
domains of the BIRD-Bench development set.
A.3
Prompt Templates
1 """ Database schema in the form of CREATE_TABLE statements:
2
3 {database_schema}
4
5 Using valid SQL , answer the following question based on the
6 tables provided above.
7
8 Hint helps you to write the correct sqlite SQL query.
9 Question: {question}
10 Hint: {evidence}
11 DO NOT return anything else except the SQL query."""
Listing 1: Zero-Shot Prompting Template.
The prompt template underlying the zero-shot models described in Section 3.3 can be found in Listing
1. The prompt integrates a given question, the associated database schema, an instruction directing the
LLM to generate valid SQL, and a hint provided by the BIRD-Bench dataset. The hint is designed to offer
insights or supplementary information needed in order to accurately interpret the database schema and to
correctly convert the question into a SQL query. Note that the other models implemented in this research
is also provided with this feature.
A.4
Examples of Errors and Corrections
This section provides examples of erroneous data points and their corrections from the different error
categories found in Table 1.
Example 1: Spelling/Syntactical Error
In Figure 5, an example question with a syntactical error is provided, representing the question with
ID 125 from the financial domain in the BIRD-Bench development set. The grammatical structure of
the question complicates the interpretation of its meaning for a human reader and makes it difficult to
understand which information it is asking for. Therefore, there is a chance that an LLM might also
misinterpret the question. A corrected version of the question can be seen in the figure.
For loans contracts which are still running where client are in debt, list the
district of the and the state the percentage unemployment rate increment
from year 1995 to 1996.
For loan contracts that are still active and where clients are in debt,
state the percentage increase in unemployment rate from 1995 to 1996.
Corrected Question
Original Question With Noise
Figure 5: Question with ID 125 from the development set of BIRD-Bench which contains syntactical errors and a
corrected version of the question.
Example 2: Ambiguous/Vague Question
Figure 6 displays the data point with ID 159 from the financial domain of the development set of
BIRD-Bench. It contains an error which were grouped into the ambiguous/vague question category.
The challenge lies in the natural language question’s ambiguity, specifically in the phrase “List all the
withdrawals...” This ambiguity revolves around determining which columns to return when executing the
SQL query.
List all the withdrawals in cash transactions that the client with the id 3356
makes.
SELECT T4.trans_id
FROM client AS T1
INNER JOIN disp AS T2 ON T1.client_id = T2.client_id
INNER JOIN account AS T3 ON T2.account_id = T3.account_id
INNER JOIN trans AS T4 ON T3.account_id = T4.account_id
WHERE T1.client_id = 3356 AND T4.operation = 'VYBER'
Gold Query
Question With Ambiguity
List the transaction ID of all withdrawals in cash transactions that the client
with the id 3356 makes.
Corrected Question
Figure 6: Question, gold SQL query and a corrected version of the question corresponding to the data point with ID
159 from the development set of BIRD-Bench, showcasing an error in the ambiguous/vague category.
Example 3: Incorrect Gold SQL
Figure 7 showcases an incorrect golden SQL query found in the data point with ID 132 of the financial
domain of the development set of BIRD-Bench. The JOIN operation incorrectly matches clients and
accounts by district_id. Due to the possibility of multiple clients and accounts in the same district,
accounts are incorrectly associated with the wrong users.
- What is the average loan amount by male borrowers?
SELECT AVG(T3.amount) FROM client AS T1
INNER JOIN account AS T2 ON T1.district_id = T2.district_id
INNER JOIN loan AS T3 ON T2.account_id = T3.account_id
WHERE T1.gender = 'M'
SELECT AVG(T1.amount) FROM loan AS T1
INNER JOIN account AS T2 ON T1.account_id = T2.account_id
INNER JOIN disp AS T3 ON T2.account_id = T3.account_id
INNER JOIN client AS T4 ON T3.client_id = T4.client_id
WHERE T4.gender = 'M'
 Incorrect Gold Query
 Corrected Query
 Question
Figure 7: Example of an incorrect SQL query that generates the wrong gold reference answer for the given question.
The JOIN operation incorrectly matches clients and accounts by district_id. Due to the possibility of multiple clients
and accounts in the same district, accounts are incorrectly associated with the wrong users.
Example 4: Synonyms
Figure 8 demonstrates how specific wordings can complicate interpretation for an LLM. The term sum,
being both a SQL keyword and a descriptor, led to the LLM’s literal interpretation and the incorrect
summation of a transaction. The actual intent was to inquire about the transaction’s balance or amount.
A rephrased question resulted in the LLM generating the correct SQL query, fetching the intended
information, as seen in the figure.
What is the sum that client number 4's account has following transaction
851? Who owns this account, a man or a woman?
SELECT
  SUM(trans.amount)
  client.gender
FROM trans
JOIN account ON trans.account_id = account.account_id
JOIN disp ON account.account_id = disp.account_id
JOIN client ON disp.client_id = client.client_id
WHERE trans.trans_id = 851 AND client.client_id = 4;
Predicted Query (Incorrect)
Original Question With SQL Keyword/Synonym
What is the balance of client number 4's account following transaction
851? Who owns this account, a man or a woman?
Corrected Question
SELECT
  trans.balance
  client.gender
FROM
  trans
  JOIN disp ON trans.account_id = disp.account_id
  JOIN client ON disp.client_id = client.client_id
WHERE trans.trans_id = 851 AND client.client_id = 4;
 Predicted Query (Correct)
Figure 8: Question from data point with ID 177 from the development set of BIRD-Bench containing a difficult
synonym, a corrected version of the question with the synonym replaced and corresponding predicted SQL queries
by the DIN-SQL (GPT-3.5) model described in Section 3.3. Showcases the difficulty of synonyms on model
predictions.
Example 5: String Capitalization
As a consequence of SQL being a case-sensitive language when comparing string values in a query,
the way a question is formulated regarding the use of uppercase or lowercase letters when asking for a
specific value affects the result. This is because the LLM will most likely use the specific entry as given
when generating the query, unless it has knowledge of the case used for different entries in the database.
Therefore, in Figure 9, an example is provided where the terms "East" and "North" are mentioned with
initial capital letters, as is commonly the case. However, the entries for these column values are in
lowercase in the database, which means the question needs to account for this for the LLM to be able to
generate a correct query. The corrected question and the SQL query generated from it can also be seen in
Figure 9.
What was the difference in the number of crimes committed in East and
North Bohemia in 1996?
SELECT
SUM(IIF(A3 = 'East Bohemia', A16, 0)) - SUM(IIF(A3 = 'North
Bohemia', A16, 0))
FROM district
Gold Query
Original Question With Dirty Values
What was the difference in the number of crimes committed in east and
north Bohemia in 1996?
Corrected Question
SELECT
    SUM(IIF(A3 = 'east Bohemia', A16, 0)) - SUM(IIF(A3 = 'north Bohemia',
A16, 0))
FROM district
Corrected Query
Figure 9: Example Ambiguous.
Example 6: Database Schema Non-Alignment
Table 4: Examples of questions that does not map to the database schema and accompanying descriptions of why
they do not.
Incorrect Question
Description
What is the disposition ID of the client who made
$5100 USD transaction on 1998/9/2?
The question asks for a single disposition
ID, which does not reflect that there is a
one-to-many relation between client and
disposition, and most likely it won’t be
possible to return a single ID.
List out the account numbers of clients who are
youngest and have highest average salary?
There is no information about salaries of
specific clients in the database.
Table 4 showcases two questions improperly aligned with the database schema’s structure, meaning the
questions cannot be accurately answered with data from the database. The structure of the database
schema can be viewed in Figure 3. The first question overlooks the possibility of clients having multiple
dispositions, as well as multiple disposition IDs. The second question instead cannot be converted into a
SQL correctly since there is no information in the database about salaries of specific clients.