Question:
Why use ChatGPT for data augmentation?
A. ChatGPT has powerful natural language generation capabilities and can ensure the quality of data.
B. It can improve the quality of preferred knowledge representation and the diversity of the training data.
C. The quality of the original data is too poor to improve the performance of the model after training.
D. In order to make the gap in the knowledge representation of preference pairs large enough to improve the effect of DPO training.

Answer: ABD

Data Augmentation based on ChatGPT. ChatGPT is able to produce higher-quality knowledge representations, compared with open-source LLMs. Therefore, in order to improve the quality of preferred knowledge representation and enhance the diversity of the training data, we employ ChatGPT to paraphrase k+. In addition to the question q, the retrieved subgraph G′, and the preferred knowledge representation k+, we also provide the answer entity e. This allows ChatGPT to organize relevant knowledge around e, ensuring that the rewritten knowledge covers key evidence. We concatenate the question q and the textualized subgraph G′ using a prompt template as the input x, and use the paraphrased knowledge representation k++ and k−as the preferred pair. Finally, we construct the preference dataset PN = (x1, k++1, k−1), (x2, k++2, k−2), ..., (xN, k++N, k−N). The prompt for knowledge augmentation is in Appendix A.6.

CoTKR: Chain-of-Thought Enhanced Knowledge Rewriting for Complex
Knowledge Graph Question Answering
Yike Wu1,2,∗,† , Yi Huang3,∗,†, Nan Hu1,2,∗,†,
Yuncheng Hua4, Guilin Qi1,2, Jiaoyan Chen5, Jeff Z. Pan6†
1Southeast University, Nanjing, Jiangsu, China
2Key Laboratory of New Generation Artificial Intelligence Technology and
Its Interdisciplinary Applications (Southeast University), Ministry of Education
3China Mobile Research Institute, Beijing, China 4Monash University, Melbourne, Australia
5University of Manchester, Manchester, UK 6University of Edinburgh, Edinburgh, UK
{yike.wu,nanhu}@seu.edu.cn, huangyi@chinamobile.com
Abstract
Recent studies have explored the use of Large
Language Models (LLMs) with Retrieval Aug-
mented Generation (RAG) for Knowledge
Graph Question Answering (KGQA). They
typically require rewriting retrieved subgraphs
into natural language formats comprehensi-
ble to LLMs. However, when tackling com-
plex questions, the knowledge rewritten by
existing methods may include irrelevant in-
formation, omit crucial details, or fail to
align with the question’s semantics. To ad-
dress them, we propose a novel rewriting
method CoTKR, Chain-of-Thought Enhanced
Knowledge Rewriting, for generating reason-
ing traces and corresponding knowledge in an
interleaved manner, thereby mitigating the limi-
tations of single-step knowledge rewriting. Ad-
ditionally, to bridge the preference gap between
the knowledge rewriter and the question an-
swering (QA) model, we propose a training
strategy PAQAF, Preference Alignment from
Question Answering Feedback, for leveraging
feedback from the QA model to further op-
timize the knowledge rewriter. We conduct
experiments using various LLMs across sev-
eral KGQA benchmarks.
Experimental re-
sults demonstrate that, compared with previous
knowledge rewriting methods, CoTKR gener-
ates the most beneficial knowledge represen-
tation for QA models, which significantly im-
proves the performance of LLMs in KGQA 1.
1
Introduction
Large Language Models (LLMs) have achieved
remarkable performance across various natural lan-
guage processing tasks, marking a significant mile-
stone (Sanh et al., 2022; Brown et al., 2020; Zhang
et al., 2022; Azaria et al., 2024; Chen, 2024). De-
spite their superior performance in zero-shot sce-
narios (Wei et al., 2022a; Kojima et al., 2022), they
∗Equal contribution. †Corresponding author.
1Our
code
is
available
at
https://github.com/wuyike2000/CoTKR.
still encounter factual errors, known as “hallucina-
tions” (Ji et al., 2023b), especially in knowledge-
intensive tasks like question answering (QA) (Hu
et al., 2023; Tan et al., 2023; Li et al., 2023; He
et al., 2023). This issue arises due to the intrinsic
limitations of LLMs, including factual inaccura-
cies and outdated knowledge (Pan et al., 2023).
To address this challenge, a substantial of work
(Ma et al., 2023a; Trivedi et al., 2023; Wu et al.,
2023b) retrieves task-relevant knowledge from ex-
ternal sources as context, thereby enhancing the
capabilities of LLMs in downstream tasks, known
as Retrieval-Augmented Generation (RAG) (Lewis
et al., 2020; Gao et al., 2023; Huang et al., 2023).
Recent work (Wu et al., 2023b; Baek et al., 2023;
Sen et al., 2023; Wang et al., 2024) under the RAG
paradigm explores the use of Knowledge Graphs
(KGs) (Pan et al., 2017b,a) as an information source
to enhance the capabilities of LLMs in Question
Answering (QA). Unlike typical QA tasks, a key
challenge in KGQA under this paradigm lies in
transforming question-related subgraphs into natu-
ral language that LLMs can understand while pre-
serving the structural information (Ko et al., 2024;
Ding et al., 2024; Wu et al., 2023b). This pro-
cess is referred to as Knowledge Rewriting (KR)
in this study. As illustrated in Figure 1, this pa-
per summarizes the commonly used knowledge
rewriting methods in existing work. Most prior
studies (Baek et al., 2023; Sen et al., 2023; Wang
et al., 2023a) employ simple linear concatenation
method (Triple), which concatenates the subject, re-
lation, and object of each triple to form triple-form
text. Additionally, considering that LLMs are pre-
trained on text corpora and struggle with structured
triple-form text, some efforts (Wu et al., 2023b;
Bian et al., 2021; Chen et al., 2022) focus on con-
verting triples into natural language through KG-to-
Text. Furthermore, given that retrieved subgraphs
often contain redundant information irrelevant to
the question, other studies (Ko et al., 2024; Dern-
arXiv:2409.19753v3  [cs.CL]  19 Mar 2025
Figure 1: The commonly used knowledge rewriting methods in existing work.
bach et al., 2024) aim to extract question-relevant
knowledge from the triples to generate summary
pertinent to the question.
Although these strategies are effective, they ex-
hibit several limitations: (1) Redundancy or omis-
sions. As illustrated in Figure 1, knowledge gen-
erated by Triple and KG-to-Text are verbose, con-
taining excessive irrelevant information.
Sum-
mary provides a question-related summary but at-
tempts to organize all relevant knowledge in one
step. Given the extensive knowledge necessary to
address complex questions, this method may not
encapsulate all critical information, potentially re-
sulting in the omission of key points. (2) Semantic
mismatch. The three existing methods shown in
Figure 1 ignore the semantics of the question and
lack a logical organization that aligns with the ques-
tion’s reasoning path.
To this end, we propose Chain-of-Thought En-
hanced Knowledge Rewriting, CoTKR. Inspired by
ReAct (Yao et al., 2023), the core of our method in-
volves generating reasoning traces and correspond-
ing knowledge in an interleaved manner. As shown
in Figure 1, we alternate the following two op-
erations: (1) Reasoning: decomposing the ques-
tion to identify the knowledge required for infer-
ence; (2) Summarization: summarizing the rele-
vant knowledge from retrieved triples, informed
by the reasoning step’s output.
By integrating
Chain-of-Thought (CoT) (Wei et al., 2022b) with
knowledge rewriting, CoTKR filters out irrelevant
information and extracts question-related knowl-
edge.
Moreover, it generates a well-organized
knowledge representation2 semantically aligned
with the question. Unlike traditional CoT appli-
cations in QA, our framework employs the knowl-
2In this paper, “knowledge representation” refers to the
natural language form of question-related knowledge.
edge rewriter to first summarize knowledge, which
then serves as contextual information to enhance
QA performance. This strategy offers superior ro-
bustness. Although the summary might be inaccu-
rate, it still contributes valuable information, po-
tentially leading to correct answers. However, ap-
plying CoT to QA requires more precise reasoning
chains, which are significantly affected by the error
propagation (Wang et al., 2023a; Yao et al., 2023).
To train knowledge rewriters based on LLMs, we
design a training framework for CoTKR. In the
first stage, inspired by previous work (Ma et al.,
2023a; Wu et al., 2023b; Ko et al., 2024), we use
knowledge representations generated by ChatGPT
to guide the supervised fine-tuning of the knowl-
edge rewriter, enabling it to initially master the
capability of knowledge rewriting. In the second
stage, we introduce Preference Alignment from
Question Answering Feedback (PAQAF) to bridge
the preference gap between the knowledge rewriter
and the QA model. This method evaluates the qual-
ity of different knowledge representations based on
the corresponding responses from the QA model.
Subsequently, it constructs preference pairs, and
fine-tunes LLMs through direct preference opti-
mization (DPO) (Rafailov et al., 2023).
We conduct experiments on GrailQA (Gu et al.,
2021) and GraphQuestions (Su et al., 2016), com-
paring commonly used knowledge rewriting meth-
ods in existing work. Contrary to previous findings
(Dai et al., 2024; Baek et al., 2023), which suggest
that LLMs perform better with knowledge in triple-
form rather than in natural language, our findings
demonstrate that LLMs can significantly benefit
from knowledge represented in carefully crafted
natural language. This indicates that our method
could substantially enhance the performance of
LLMs in KGQA.
The main contributions of this paper are:
• We propose CoTKR, a Chain-of-Thought En-
hanced Knowledge Rewriting method to im-
prove the quality of knowledge representation
through the application of CoT. This method
generates reasoning traces and corresponding
knowledge in an interleaved manner, thereby
producing well-organized knowledge repre-
sentations that are coherent with the ques-
tion’s semantics.
• We propose a training strategy PAQAF,
Preference
Alignment
from
Question
Answering Feedback, to bridge the prefer-
ence gap between the knowledge rewriter
and the QA model. It assesses the quality
of different knowledge representations by
evaluating corresponding responses from the
QA model. Then, it constructs preference
pairs and employs DPO to optimize the
knowledge rewriter.
• We conduct experiments on two KGQA
benchmarks. Compared with other knowledge
rewriting methods, CoTKR can generate the
most beneficial knowledge representation for
the QA model and further enhance the perfor-
mance of LLMs in KGQA. Additionally, con-
sidering privacy and cost issues, we evaluate
the performance of open-source and closed-
source LLMs as the foundational models for
knowledge rewriting and QA.
2
Related Work
2.1
KG-Augmented LLMs for KGQA
To mitigate hallucination in LLMs, existing work
(Wu et al., 2023b; Baek et al., 2023; Sen et al.,
2023; Wang et al., 2024) attempts to enhance LLMs
with KGs in the RAG paradigm. The naïve ap-
proach involves retrieving question-related triples
from KGs as contextual information for QA (Baek
et al., 2023; Sen et al., 2023). Although this method
has proven effective, there remains ample room for
improvement. Some studies (Wang et al., 2024,
2023a) integrate Chains-of-Thought (CoT) (Wei
et al., 2022b) with RAG to tackle complex ques-
tions.
Keqing(Wang et al., 2024) decomposes
complex questions using predefined templates, re-
trieves candidate entities from KG, reasons through
sub-questions, and ultimately generates answers
with clear reasoning paths. KD-CoT (Wang et al.,
2023a) validates and adjusts reasoning traces in
CoT through interactions with external knowledge,
thereby addressing issues of hallucinations and er-
ror propagation. Furthermore, alternative efforts
(Wu et al., 2023b; Ko et al., 2024) address the limi-
tations of LLMs in processing structured knowl-
edge and the noise in retrieved triples by post-
processing these triples into natural language or
summaries pertinent to the questions.
This paper focuses on optimizing the knowledge
representation under the RAG paradigm for KGQA.
Unlike previous work that transforms triples into
the natural language in one step, we adopt CoT to
summarize relevant knowledge step-by-step, ensur-
ing comprehensiveness and semantic coherence in
the generated knowledge.
2.2
Preference Alignment for LLMs on
Question Answering
LLMs have the potential to generate content that
contains gender discrimination, unethical elements,
and racial biases, inconsistent with human values
(Wu et al., 2023a; Ray, 2023). To address this issue,
Preference Alignment (PA) (Ji et al., 2023a; Wang
et al., 2023b) aims to fine-tune LLMs to align with
human preferences. Existing QA work based on
LLMs uses PA to bridge the gap between model
preferences and those of humans or the QA tasks.
KnowPAT (Zhang et al., 2023) trains LLMs on a
knowledge preference set to align their knowledge
biases with human preferences, selecting better fac-
tual knowledge as context. BGM (Ke et al., 2024)
utilizes downstream task metrics as rewards to opti-
mize the bridging model between retrievers and QA
models. Rewrite-Retrieve-Read (Ma et al., 2023b)
employs QA evaluation metrics as reward signals,
fine-tuning the query rewriting module. EFSUM
(Ko et al., 2024) constructs preference pairs sam-
pled from LLMs and fine-tunes the knowledge sum-
marizer using the Direct Preference Optimization
(DPO) (Rafailov et al., 2023) algorithm.
Our study innovatively employs responses from
the QA model to evaluate the quality of knowledge
representations. We then construct preference pairs
from these evaluations and optimize the knowledge
rewriter using DPO.
3
Preliminaries
Knowledge Graph (KG) is a structured collection
of triples in the form of (s, r, o), where s, r, o rep-
resent the subject, the relation, and the object. This
Figure 2: Illustration of our KGQA framework. CoTKR generates reasoning traces and corresponding knowledge
in an interleaved manner.
collection is denoted by G = {(s, r, o) | s, o ∈
E, r ∈R}, where E represents the set of entities
and R represents the set of relations.
Knowledge Graph Question Answering (KGQA)
aims to answer natural language questions by utiliz-
ing a set of facts within KGs. Following previous
work (Saxena et al., 2020; Jiang et al., 2023b), we
assume that the subject entity of the question is
given. Given a question q and a subject entity e,
the objective is to generate a response a using the
relevant facts in the KG G that accurately addresses
the question.
Knowledge Rewriting (KR) for KGQA aims to
transform question-related triples into natural lan-
guage that can be consumed by LLMs. Given a
question q and a subgraph G′ = {(s, r, o) | s, o ∈
E, r ∈R} retrieved from KG G, the task is to gen-
erate a natural language sequence X that provides
contextual information to answer the question.
4
Methods
4.1
Chain-of-Thought Enhanced Knowledge
Rewriting
The architecture of our QA framework is depicted
in Figure 2. Initially, our framework retrieves a
question-related subgraph from the KG, which is
subsequently transformed into contextual knowl-
edge using CoTKR. This contextual knowledge,
along with the question, prompts the QA model to
generate an answer. The core of this framework is
the knowledge rewriter. Briefly, it alternatively con-
ducts the following two operations: Reasoning: de-
composing the question and generating a reasoning
trace based on generated knowledge representation
and pointing out the specific knowledge needed for
the current step; Summarization: summarizing the
relevant knowledge based on the current reasoning
trace from the subgraph.
Assume we have the reasoning traces at step
t −1 as xt−1,r and the summarized knowledge at
step t −1 as xt−1,k. The corresponding knowledge
representation, i.e., Xt−1 is represented as:
Xt−1 = [x1,r, x1,k, ..., xt−1,r, xt−1,k].
(1)
For knowledge rewriting at step t, given the ques-
tion q, the subgraph G′, and the previously gener-
ated content Xt−1, the knowledge rewriter R first
generates the reasoning trace xt,r:
xt,r = R(q, G′, Xt−1)
(2)
Subsequently, based on the question q, the sub-
graph G′, the previously generated content Xt−1,
and the reasoning trace at step t xt,r, CoTKR sum-
marizes relevant knowledge xt,k:
xt,k = R(q, G′, Xt−1, xt,r)
(3)
xt,r and xt,k are attached to Xt−1 for the knowl-
edge representation at step t. Note in step 1, X0 is
initialized to None.
4.2
Training Framework for CoTKR
Figure 3 illustrates the training framework of
CoTKR.
4.2.1
Supervised Fine-tuning with Knowledge
Distilled from ChatGPT
This stage enables open-source LLMs to initially
acquire the knowledge rewriting capability through
supervised fine-tuning. This primarily comprises
two steps: reference knowledge representation gen-
eration and supervised fine-tuning.
Reference Knowledge Representation Genera-
tion. Inspired by previous work (Ma et al., 2023a;
Wu et al., 2023b; Ko et al., 2024), we employ Chat-
GPT as the data generator to construct training
corpora. We verbalize the question-related sub-
graph, G′, through simple linear concatenation and
Figure 3: Our training framework for CoTKR.
combine it with the question, q, to form the in-
put prompt x. Subsequently, ChatGPT generates
the reference knowledge representation k based on
several examples (i.e., demonstrations) and the pro-
vided input x. Finally, we construct the training
dataset DT = {(x1, k1), (x2, k2), ..., (xT , kT )}.
Supervised Fine-tuning. For every pair of input
and output (xi, ki) in the training dataset DT , our
knowledge rewriter Rθ is trained to generate ki
based on xi using the following objective:
LSFT = −1
T
T
X
i=1
log pθ(ki|xi)
(4)
where θ represents the parameters of the knowledge
rewriter Rθ and pθ(ki|xi) signifies the probability
that Rθ generates ki, given the input xi.
4.2.2
Preference Alignment from Question
Answering Feedback (PAQAF)
In this stage, Preference Alignment (PA) is em-
ployed to bridge the preference gap between the
knowledge rewriter and the QA model. This stage
includes four steps: candidate knowledge represen-
tation sampling, preference annotation based on
QA feedback, data augmentation based on Chat-
GPT, and direct preference optimization (DPO).
Candidate Knowledge Representation Sampling.
We input the question q and the corresponding sub-
graph G′, then sample M candidate knowledge rep-
resentations, k1, k2, ..., kM, from the knowledge
rewriter Rθ.
Preference Annotation based on Question An-
swering Feedback. Among the candidate knowl-
edge representations, we select the two, k1 and
k2, with the greatest semantic difference (i.e., the
lowest similarity) to facilitate faster convergence
during training. Utilizing standard evaluation meth-
ods for assessing these representations is subop-
timal, as they fail to align with the preferences
of QA models. Inspired by the findings in previ-
ous work(Wu et al., 2023b; Ko et al., 2024; Zhang
et al., 2023), we posit that better knowledge rep-
resentations generally lead to better performance
on QA. Consequently, we adopt k1 and k2 as con-
textual knowledge, prompting the QA model Q to
answer the question q, generating answers a1 and
a2, respectively. Subsequently, we prompt Chat-
GPT to assess the quality of a1 and a2 from the
perspectives of accuracy and relevance. This eval-
uation aims to identify the preferred knowledge
representation k+ and the dispreferred knowledge
representation k−. Details of the evaluation prompt
are provided in Appendix A.5.
Data Augmentation based on ChatGPT. Chat-
GPT is able to produce higher-quality knowledge
representations, compared with open-source LLMs.
Therefore, in order to improve the quality of
preferred knowledge representation and enhance
the diversity of the training data, we employ
ChatGPT to paraphrase k+. In addition to the
question q, the retrieved subgraph G′, and the
preferred knowledge representation k+, we also
provide the answer entity e. This allows ChatGPT
to organize relevant knowledge around e, ensuring
that the rewritten knowledge covers key evidence.
We concatenate the question q and the textualized
subgraph G′ using a prompt template as the
input x, and use the paraphrased knowledge
representation k++ and k−as the preferred pair.
Finally, we construct the preference dataset PN =
(x1, k++
1
, k−
1 ), (x2, k++
2
, k−
2 ), ..., (xN, k++
N , k−
N).
The prompt for knowledge augmentation is in
Appendix A.6.
Direct Preference Optimization. We employ Di-
rect Preference Optimization (DPO) on our knowl-
edge rewriter, Rθ, to develop a preference-tuned
version, Rθ∗. It minimizes the following objective:
LDPO(θ∗; θ) =
−1
N
N
X
i=1
log σ[r(xi, k++
i
) −r(xi, k−
i )]
(5)
r(xi, ki) = pθ∗(ki|xi)
pθ(ki|xi)
(6)
Considering the varying preferences of different
QA models, CoTKR is specifically trained for each
QA model. Through the two stages of training,
CoTKR tends to generate more favorable knowl-
edge representation k++ for each QA model, while
avoiding unhelpful knowledge representation k−.
5
Experiments
5.1
Datasets
GrailQA (Gu et al., 2021) is a challenging, large-
scale multi-hop KGQA benchmark that consists of
64,331 questions (44,337 train, 6,763 dev, 13,231
test). The training and dev sets provide annotated
SPARQL query and answer entities, while the test
set comprises only the questions. For evaluation
convenience, the dev set is used for testing.
GraphQuestions
(Su
et
al.,
2016)
is
a
characteristic-rich
dataset
for
factoid
ques-
tion answering based on Freebase. It comprises
5,166 questions (2,771 train, 2,395 test).
For
each question, the dataset provides corresponding
SPARQL query and answer entities.
5.2
Large Language Models
In this experiment, Llama-2 (7B) (Touvron et al.,
2023b), Llama-3 (8B) (AI@Meta, 2024), and Chat-
GPT 3 are employed for knowledge rewriting,
while ChatGPT and Mistral (7B) (Jiang et al.,
2023a) are used for QA tasks. The details of these
LLMs are provided in Appendix A.3.
5.3
Baselines
We compare CoTKR (without PAQAF) and
CoTKR+PAQAF (CoTKR+PA for shortness)
with other knowledge rewriting methods in KGQA:
Simple linear concatenation (Triple) (Baek et al.,
2023; Sen et al., 2023) concatenates the subject,
predicate, and object of a triple to generate triple-
form text. This method does not require additional
models for knowledge rewriting.
KG-to-Text (Wu et al., 2023b) transforms facts
into the free-form text for each relation path with
3https://api.openai.com/
a KG-to-Text model, addressing the limitations of
LLMs in understanding structured triple-form text.
Summary (Ko et al., 2024) converts triples into a
question-relevant summary, alleviating the issue of
redundant contextual knowledge.
To ensure a fair comparison, we employ the same
corpus generation method for both the baselines
and our method. All baselines undergo supervised
fine-tuning without preference alignment.
5.4
Retrieval Methods
The retrieval module is not the focus of our re-
search. Therefore, we adopt three commonly used
retrieval methods. For detailed implementation,
please refer to Appendix A.4.
2-Hop. We retain 30 triples from the 2-hop sub-
graph of the head entity, prioritizing those with
higher semantic similarity to the question.
BM25. We follow the processing method in De-
cAF (Yu et al., 2023), simply linearizing the 1-hop
subgraph of the topic entity as the article. We take
the top 30 triples corresponding to the candidate
documents as the retrieval results.
Ground Truth Subgraph (GS). We modify the
SPARQL queries from the datasets to obtain the
ground truth subgraphs. These subgraphs represent
the results of an ideal retriever.
5.5
Evaluation Metrics
Following previous work on generative KGQA
(Wu et al., 2023b; Baek et al., 2023; Ko et al.,
2024), we adopt Accuracy (Acc) as one of our eval-
uation metrics. It measures whether the model’s
response includes at least one answer entity. For a
dataset comprising N questions, Acc is calculated
as follows:
Acc =
PN
i=1 Acci
N
(7)
Acci =











1,
if at least one answer entity
appears in the response
0,
if no answer entity appears in
the response
(8)
In order to comprehensively assess the perfor-
mance of KGQA, we employ Recall to evaluate
the proportion of correct answer entities present in
the model’s response. For a dataset containing N
questions, Recall is calculated as follows:
Recall =
PN
i=1 Recalli
N
(9)
Recalli = Nappear
Ntotal
(10)
where Nappear refers to the number of answer enti-
ties contained in the model’s responses, and Ntotal
refers to the total number of the answer entities.
Additionally, we consider utilizing Exact Match
(EM) as an evaluation metric. Given that the re-
sponses generated by LLMs consist of multiple
paragraphs, while the corresponding answers are
entities, we adjust the traditional EM metric. Our
modified EM metric assesses whether all answer
entities are included in the model’s responses. For a
dataset consisting of N questions, EM is calculated
as follows:
EM =
PN
i=1 EMi
N
(11)
EMi =





1,
if all answer entities
appear in the response
0,
other cases
(12)
5.6
Main Results
To comprehensively evaluate various knowledge
rewriting methods, we employ the widely used 2-
Hop retrieval method. Table 1 presents the overall
results. We observe that: (1) Our method out-
performs the baselines across most evaluation
metrics and LLMs, confirming the effectiveness
of our knowledge rewriting strategy. This also
demonstrates the broad practical applicability of
CoTKR, effective for both open-source LLMs re-
quiring fine-tuning and closed-source LLMs us-
ing ICL. Integrating question-related knowledge
significantly improves QA performance compared
with direct question answering, underscoring the
efficacy of the RAG paradigm in KGQA. KG-to-
Text exhibits the weakest performance, indicating
that mere conversion of triples into text may re-
sult in loss of information inherent in the subgraph.
Summary outperforms KG-to-Text but generally
lags behind CoTKR/CoTKR+PA, suggesting that
filtering out irrelevant knowledge is effective but
not adequate. (2) CoTKR+PA matches or even
surpasses the performance of ChatGPT as the
knowledge rewriter, proving the effectiveness
of our training framework and the preference
alignment. CoTKR+PA outperforms CoTKR, in-
dicating that preference alignment can bridge the
preference gap between the knowledge rewriter
and the QA model, thereby enhancing the quality
of knowledge representation. (3) A well-crafted
knowledge representation is crucial for LLM
used in KGQA. Although Triple does not require
an additional knowledge rewriting module, it pro-
vides a strong baseline and, in some cases, out-
performs KG-to-Text and Summary. Conversely,
CoTKR+PA consistently surpasses Triple. This
indicates that Triple is simple yet effective and ex-
plains its widespread use in existing work. On
the other hand, it demonstrates that a carefully de-
signed knowledge representation can effectively
enhance the performance of KGQA.
5.7
Impact of Retrieval Methods
To investigate the impact of retrieval methods, we
select Llama-3 as the knowledge rewriter and Chat-
GPT as the QA model. According to the results
shown in Figure 4, we have the following observa-
tions: (1) 2-Hop retrieval method may be insuf-
ficient for more challenging questions, but it is
suitable for simpler ones. Both BM25 and 2-Hop
perform similarly on GrailQA, but 2-Hop shows a
significant advantage over BM25 on GraphQues-
tions. This is likely because GrailQA is a more
complex benchmark with a larger question-related
subgraph, making 2-hop subgraphs often inade-
quate. Conversely, for GraphQuestions, a 2-hop
subgraph usually provides precise context for most
questions. (2) The design of a high-quality re-
triever remains an open problem. GS signifi-
cantly outperforms BM25 and 2-Hop, indicating
that retrieval noise substantially affects KGQA per-
formance. (3) CoTKR consistently outperforms
all baselines across various retrieval methods,
demonstrating its robustness and practicality.
Figure 4: KGQA results using different knowledge
rewriters and different retrieval methods.
5.8
Comparison with Triple Method
Several studies (Dai et al., 2024; Baek et al., 2023)
suggest that LLMs can better comprehend triple-
form text compared with natural language. How-
ever, our results show the contrary. Therefore, we
delve deeper into this issue by comparing knowl-
edge rewriting methods that use triple-form text
KR LLMs
Methods
GrailQA
GraphQuestions
Acc
Recall
EM
Acc
Recall
EM
ChatGPT as QA model
None
No Knowledge
28.91
22.81
20.14
35.87
25.76
22.09
Triple
57.76
49.67
44.73
55.03
46.65
41.63
Llama-2
KG-to-Text
54.75
47.35
42.44
49.73
40.00
33.74
Summary
58.14
51.38
46.38
52.94
44.70
38.41
CoTKR
58.64
52.33
47.88
51.36
45.20
39.96
CoTKR+PA
59.25
53.52
49.64
56.78
47.99
42.46
Llama-3
KG-to-Text
55.76
48.41
43.90
52.40
45.06
39.83
Summary
57.55
51.06
46.80
54.95
46.86
40.75
CoTKR
58.33
52.55
48.65
53.19
47.23
43.17
CoTKR+PA
61.51
56.08
52.67
56.37
49.31
45.26
ChatGPT
KG-to-Text
56.32
49.05
44.73
53.53
45.59
41.17
Summary
58.54
51.81
47.29
55.62
48.93
44.97
CoTKR
59.87
53.19
49.02
54.28
48.18
44.68
Mistral as QA model
None
No Knowledge
29.44
23.13
20.30
38.20
26.92
22.13
Triple
54.47
47.78
43.25
51.32
45.97
41.67
Llama-2
KG-to-Text
49.49
42.91
38.41
44.59
37.98
32.82
Summary
54.10
47.79
43.15
49.85
42.33
36.45
CoTKR
56.75
51.10
46.71
50.19
43.73
38.54
CoTKR+PA
58.15
52.98
49.13
55.07
47.02
41.71
Llama-3
KG-to-Text
50.64
44.32
40.13
49.06
43.04
38.25
Summary
53.84
47.71
43.49
52.03
44.30
38.50
CoTKR
56.47
51.33
47.36
52.65
46.48
42.21
CoTKR+PA
59.31
54.13
50.24
54.82
47.76
43.09
ChatGPT
KG-to-Text
51.04
44.87
40.97
49.14
43.04
38.83
Summary
54.44
48.16
43.97
52.28
47.10
43.30
CoTKR
57.28
51.14
47.09
52.82
47.13
43.55
Table 1: The overall results of CoTKR and the baselines on GrailQA and GraphQuestions using 2-Hop as retrieval
method. For each combination of a knowledge rewriter (KR) LLM and a QA model, the best and second-best results
are highlighted in bold and underlined, respectively.
as input (i.e., KG-to-Text, Summary, CoTKR,
CoTKR+PA) with Triple. We use Accuracy as the
criterion to evaluate the correctness of responses.
For each method, we consider three scenarios: (1)
Incorrect→Correct: Triple provides a wrong an-
swer, but the comparative method answers cor-
rectly. (2) Correct->Incorrect: Triple answers
correctly, but the comparative method answers in-
correctly. (3) No change: both Triple and the com-
parative method answer correctly or incorrectly.
We adopt Llama-3 as the knowledge rewriter and
ChatGPT as the QA model, with 2-Hop as the re-
trieval method. Then we calculate the proportions
of three distinct cases within GrailQA. From the
results shown in Figure 5, we draw the following
conclusions: (1) KG-to-Text and Summary have
a predominantly negative impact, partially val-
idating the conclusions of prior studies. Triple
provides a strong baseline, and the adoption of
KG-to-Text and Summary leads to more incor-
rect answers. This indicates that LLMs can under-
stand triple-form text effectively, and using sim-
ple knowledge rewriting methods leads to loss of
information. (2) Well-designed knowledge rep-
resentations substantially benefit the question-
answering model. The knowledge representations
rewritten by CoTKR/CoTKR+PA generally en-
hance the QA model’s performance. This reflects
that the suboptimal knowledge representations in
previous work are key contributors to performance
degradation. Our method generates comprehensive
and semantically coherent knowledge representa-
tions, thereby improving the efficacy of KGQA.
Figure 5: The comparative results on GrailQA. We use
2-Hop as the retrieval method.
5.9
Effectiveness of Data Augmentation
KR LLMs
Methods
ChatGPT
Mistral
Acc
Recall
EM
Acc
Recall
EM
Llama-2
CoTKR
51.36
45.20
39.96
50.19
43.73
38.54
CoTKR+PA*
54.57
47.40
42.88
52.99
45.69
41.09
CoTKR+PA
56.78
47.99
42.46
55.07
47.02
41.71
Llama-3
CoTKR
53.19
47.23
43.17
52.65
46.48
42.21
CoTKR+PA*
54.70
48.79
44.72
53.24
47.64
43.63
CoTKR+PA
56.37
49.31
45.26
54.82
47.76
43.09
Table 2: Ablation study results for data augmentation on
GraphQuestions, employing the 2-Hop retrieval method.
To evaluate the effectiveness of data augmenta-
tion, we compare the performance of three variants:
CoTKR, CoTKR+PA, and CoTKR+PA* (using
supervised fine-tuning and preference alignment
Question: what is the unit of area that the measurement system that have an electric field strength units of volt per metre have?
Answer: Square kilometer, Square meter
KR Methods
KR Results
Summary
The International System of Units includes various measurement systems for different physical quantities such as length,
area, volume, conductance, resistance, current, force, and pressure. The measurement system with an electric field
strength unit of Volt per meter does not directly relate to a specific area unit within the International System of Units.
CoTKR
Reason: I need to know the unit of area associated with the measurement system that has an electric field strength unit of
Volt per meter.
Summarize: The International System of Units includes various units for different physical quantities such as length,
area, volume, etc.
Reason: I need to find the specific unit of area in the International System of Units that corresponds to the electric field
strength unit of Volt per meter.
Summarize: The International System of Units includes units like Square kilometer, Square meter, etc. for measuring
area.
CoTKR+PA
Reason: I need to know which measurement system includes the unit for electric field strength in Volt per meter.
Knowledge: The International System of Units contains the unit for electric field strength in Volt per meter.
Reason: I need to identify the unit of area within the International System of Units.
Knowledge: The International System of Units includes the units Square kilometer and Square meter as area units.
Table 3: An example of knowledge rewriting results for different methods.
without data augmentation). The experimental re-
sults are shown in Table 2, from which we can draw
two conclusions: (1) CoTKR+PA* generally out-
performs CoTKR+PA, indicating that PAQAF
does not solely rely on data augmentation based
on ChatGPT. (2) CoTKR+PA performs best in
most scenarios, proving that data augmentation
enhances the preference alignment.
5.10
Case Study
In this section, we compare Summary with
CoTKR through an example. (Please refer to Ap-
pendix B for the full comparison result.) As il-
lustrated in Table 3, Summary struggles to ex-
tract useful information when faced with an abun-
dance of triples. In contrast, CoTKR, leveraging
CoT reasoning, effectively emphasizes the key ev-
idence (i.e., Square meter) in the second rewrit-
ing step. Furthermore, after preference alignment,
CoTKR+PA is capable of generating more natural
reasoning steps, significantly enhancing its applica-
bility to KGQA.
6
Conclusion
In this paper, we propose Chain-of-Thought En-
hanced Knowledge Rewriting, CoTKR, for higher
quality knowledge representation of triples in KG
augmented QA. To bridge the preference gap be-
tween the knowledge rewriter and the QA model,
we propose Preference Alignment from Question
Answering Feedback, PAQAF. Experimental re-
sults demonstrate that, compared with existing
knowledge rewriting methods, CoTKR can gen-
erate the most beneficial knowledge representation
for QA models. In future work, we will go beyond
KGQA to explore knowledge representations for
other kinds of structured data for RAG.
7
Limitations
We acknowledge the limitations of this work. (1)
This study is limited to KGQA and does not explore
broader application scenarios. Therefore, we did
not design experiments to explore whether CoTKR
is effective for all or most RAG scenarios. In future
work, we aim to expand the range of data sources.
We intend to design a knowledge rewriting method
that can be applied to not only KGs but also tables,
textual data, and other formats. This enhancement
will allow the QA framework to access knowledge
from a wider range of sources, thus improving its
practicality. Furthermore, we plan to investigate
a knowledge representation beneficial for various
downstream tasks, such as fact verification and
dialogue generation. (2) The training framework
for CoTKR depends on the powerful capabilities
of closed-source LLMs. However, these models
have inherent limitations, and the training data they
generate contains noise, which constrains the per-
formance ceiling of CoTKR.
8
Ethical Considerations
We explore optimizing knowledge representations
for KGQA on public benchmarks, avoiding any
potential harm to any individuals or groups. To
promote transparency and facilitate replication of
our research, we provide the technical details nec-
essary for reproducing our results and release both
the source code and the collected data. Our code
and data are available for academic research, com-
mercial use, and other applications.
It is important to acknowledge the potential risks
and ethical considerations associated with LLMs.
In this study, we construct the training data using
ChatGPT and implement our knowledge rewriters
based on LLMs. Due to the inherent limitations
of LLMs, including factual inaccuracies, racial
discrimination, and gender bias, our knowledge
rewriters might generate incorrect content or inad-
vertently reflect prevalent societal biases.
Acknowledgments
This work is partially supported by National
Nature Science Foundation of China under No.
U21A20488, by the project "Key Laboratory of
rich-media Digital Publishing Content Organiza-
tion and Knowledge Service Open Fund-Research
on Knowledge-enhanced Training Techinques of
Large Language Model" No. ZD2024-04/01, by
the EPSRC project OntoEm (EP/Y017706/1), and
by Southeast University-China Mobile Research
Institute Joint Innovation Center. We thank the Big
Data Computing Center of Southeast University
for providing the facility support on the numerical
calculations in this paper. ChatGPT was used to
enhance the readability of some of the text and im-
prove the language of this paper, after the content
was first added manually. All material was then
checked manually before submission.
References
AI@Meta. 2024. Llama 3 model card.
Amos Azaria, Rina Azoulay, and Shulamit Reches.
2024. Chatgpt is a remarkable tool - for experts.
Data Intell., 6(1):240–296.
Jinheon Baek, Alham Fikri Aji, and Amir Saffari. 2023.
Knowledge-augmented language model prompting
for zero-shot knowledge graph question answering.
In Proceedings of the 1st Workshop on Natural
Language Reasoning and Structured Explanations
(NLRSE), pages 78–106, Toronto, Canada. Associa-
tion for Computational Linguistics.
Ning Bian, Xianpei Han, Bo Chen, and Le Sun. 2021.
Benchmarking knowledge-enhanced commonsense
question answering via knowledge-to-text transfor-
mation. In AAAI, pages 12574–12582. AAAI Press.
Kurt D. Bollacker, Robert P. Cook, and Patrick Tufts.
2007. Freebase: A shared database of structured
general human knowledge. In AAAI, pages 1962–
1963. AAAI Press.
Kurt D. Bollacker, Colin Evans, Praveen K. Paritosh,
Tim Sturge, and Jamie Taylor. 2008. Freebase: a
collaboratively created graph database for structuring
human knowledge. In SIGMOD Conference, pages
1247–1250. ACM.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie
Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind
Neelakantan, Pranav Shyam, Girish Sastry, Amanda
Askell, Sandhini Agarwal, Ariel Herbert-Voss,
Gretchen Krueger, Tom Henighan, Rewon Child,
Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu,
Clemens Winter, Christopher Hesse, Mark Chen, Eric
Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess,
Jack Clark, Christopher Berner, Sam McCandlish,
Alec Radford, Ilya Sutskever, and Dario Amodei.
2020. Language models are few-shot learners. In
NeurIPS.
Huajun Chen. 2024. Large knowledge model: Perspec-
tives and challenges. Data Intelligence, 6(3):587–
620.
Zhuo Chen, Yufeng Huang, Jiaoyan Chen, Yuxia Geng,
Yin Fang, Jeff Z. Pan, Ningyu Zhang, and Wen Zhang.
2022. Lako: Knowledge-driven visual question an-
swering via late knowledge-to-text injection.
In
IJCKG, pages 20–29. ACM.
Xinbang Dai, Yuncheng Hua, Tongtong Wu, Yang
Sheng, and Guilin Qi. 2024. Counter-intuitive: Large
language models can better understand knowledge
graphs than we thought. CoRR, abs/2402.11541.
Stefan Dernbach, Khushbu Agarwal, Alejandro Zu-
niga, Michael Henry, and Sutanay Choudhury. 2024.
Glam: Fine-tuning large language models for domain
knowledge graph alignment via neighborhood parti-
tioning and generative subgraph encoding. In AAAI
Spring Symposia, pages 82–89. AAAI Press.
Wentao Ding, Jinmao Li, Liangchuan Luo, and Yuzhong
Qu. 2024. Enhancing complex question answering
over knowledge graphs through evidence pattern re-
trieval. In WWW, pages 2106–2115. ACM.
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia,
Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Qianyu Guo,
Meng Wang, and Haofen Wang. 2023. Retrieval-
augmented generation for large language models: A
survey. CoRR, abs/2312.10997.
Yu Gu, Sue Kase, Michelle Vanni, Brian M. Sadler,
Percy Liang, Xifeng Yan, and Yu Su. 2021. Beyond
I.I.D.: three levels of generalization for question an-
swering on knowledge bases. In WWW, pages 3477–
3488. ACM / IW3C2.
Jie He, Simon Chi Lok U, Víctor Gutiérrez-Basulto, and
Jeff Z. Pan. 2023. BUCA: A Binary Classification
Approach to Unsupervised Commonsense Question
Answering. In Proceedings of the 61st Annual Meet-
ing of the Association for Computational Linguistics
(ACL 2023).
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan
Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and
Weizhu Chen. 2022. Lora: Low-rank adaptation of
large language models. In ICLR. OpenReview.net.
Nan Hu, Yike Wu, Guilin Qi, Dehai Min, Jiaoyan Chen,
Jeff Z. Pan, and Zafar Ali. 2023. An empirical study
of pre-trained language models in simple knowledge
graph question answering. World Wide Web (WWW),
26(5):2855–2886.
Wenyu Huang, Mirella Lapata, Pavlos Vougiouklis,
Nikos Papasarantopoulos, and Jeff Z. Pan. 2023. Re-
trieval augmented generation with rich answer en-
coding. In Proceedings of the 13th International
Joint Conference on Natural Language Processing
and the 3rd Conference of the Asia-Pacific Chapter of
the Association for Computational Linguistics, pages
1012–1025.
Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang,
Hantao Lou, Kaile Wang, Yawen Duan, Zhong-
hao He, Jiayi Zhou, Zhaowei Zhang, Fanzhi Zeng,
Kwan Yee Ng, Juntao Dai, Xuehai Pan, Aidan
O’Gara, Yingshan Lei, Hua Xu, Brian Tse, Jie Fu,
Stephen McAleer, Yaodong Yang, Yizhou Wang,
Song-Chun Zhu, Yike Guo, and Wen Gao. 2023a.
AI alignment: A comprehensive survey.
CoRR,
abs/2310.19852.
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu,
Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea
Madotto, and Pascale Fung. 2023b. Survey of halluci-
nation in natural language generation. ACM Comput.
Surv., 55(12):248:1–248:38.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Men-
sch, Chris Bamford, Devendra Singh Chaplot, Diego
de las Casas, Florian Bressand, Gianna Lengyel, Guil-
laume Lample, Lucile Saulnier, et al. 2023a. Mistral
7b. arXiv preprint arXiv:2310.06825.
Jinhao Jiang, Kun Zhou, Xin Zhao, and Ji-Rong Wen.
2023b. Unikgqa: Unified retrieval and reasoning for
solving multi-hop question answering over knowl-
edge graph. In ICLR. OpenReview.net.
Zixuan Ke, Weize Kong, Cheng Li, Mingyang Zhang,
Qiaozhu Mei, and Michael Bendersky. 2024. Bridg-
ing the preference gap between retrievers and llms.
CoRR, abs/2401.06954.
SungHo Ko, Hyunjin Cho, Hyungjoo Chae, Jinyoung
Yeo, and Dongha Lee. 2024. Evidence-focused fact
summarization for knowledge-augmented zero-shot
question answering. CoRR, abs/2403.02966.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu-
taka Matsuo, and Yusuke Iwasawa. 2022. Large lan-
guage models are zero-shot reasoners. In NeurIPS.
Patrick S. H. Lewis, Ethan Perez, Aleksandra Pik-
tus, Fabio Petroni, Vladimir Karpukhin, Naman
Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih,
Tim Rocktäschel, Sebastian Riedel, and Douwe
Kiela. 2020.
Retrieval-augmented generation for
knowledge-intensive NLP tasks. In NeurIPS.
Linhan Li, Huaping Zhang, Chunjin Li, Haowen You,
and Wenyao Cui. 2023. Evaluation on chatgpt for chi-
nese language understanding. Data Intell., 5(4):885–
903.
Jimmy Lin, Xueguang Ma, Sheng-Chieh Lin, Jheng-
Hong Yang, Ronak Pradeep, and Rodrigo Frassetto
Nogueira. 2021. Pyserini: A python toolkit for repro-
ducible information retrieval research with sparse and
dense representations. In SIGIR, pages 2356–2362.
ACM.
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang,
Ruochen Xu, and Chenguang Zhu. 2023. G-eval:
NLG evaluation using gpt-4 with better human align-
ment. In EMNLP, pages 2511–2522. Association for
Computational Linguistics.
Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao,
and Nan Duan. 2023a. Query rewriting in retrieval-
augmented large language models. In EMNLP, pages
5303–5315. Association for Computational Linguis-
tics.
Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao,
and Nan Duan. 2023b. Query rewriting in retrieval-
augmented large language models. In Proceedings of
the 2023 Conference on Empirical Methods in Natu-
ral Language Processing, pages 5303–5315, Singa-
pore. Association for Computational Linguistics.
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike
Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer,
Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023.
Factscore: Fine-grained atomic evaluation of factual
precision in long form text generation. In EMNLP,
pages 12076–12100. Association for Computational
Linguistics.
Jeff Z. Pan, Simon Razniewski, Jan-Christoph Kalo,
Sneha Singhania, Jiaoyan Chen, Stefan Dietze, Hajira
Jabeen, Janna Omeliyanenko, Wen Zhang, Matteo
Lissandrini, ussa Biswas, Gerard de Melo, Angela
Bonifati, Edlira Vakaj, Mauro Dragoni, and amien
Graux. 2023. Large language models and knowledge
graphs: Opportunities and challenges. Transactions
on Graph Data and Knowledge.
J.Z. Pan, D. Calvanese, T. Eiter, I. Horrocks, M. Kifer,
F. Lin, and Y. Zhao, editors. 2017a. Reasoning Web:
Logical Foundation of Knowledge Graph Construc-
tion and Querying Answering. Springer.
J.Z. Pan, G. Vetere, J.M. Gomez-Perez, and H. Wu, edi-
tors. 2017b. Exploiting Linked Data and Knowledge
Graphs for Large Organisations. Springer.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christo-
pher D. Manning, Stefano Ermon, and Chelsea Finn.
2023. Direct preference optimization: Your language
model is secretly a reward model. In NeurIPS.
Partha Pratim Ray. 2023. Chatgpt: A comprehensive
review on background, applications, key challenges,
bias, ethics, limitations and future scope. internet of
things and cyber-physical systems, 3, 121–154. URL
https://doi. org/10.1016/j. iotcps, 3.
Stephen E. Robertson and Hugo Zaragoza. 2009. The
probabilistic relevance framework: BM25 and be-
yond. Found. Trends Inf. Retr., 3(4):333–389.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H.
Bach, Lintang Sutawika, Zaid Alyafeai, Antoine
Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey,
M Saiful Bari, Canwen Xu, Urmish Thakker,
Shanya Sharma Sharma, Eliza Szczechla, Taewoon
Kim, Gunjan Chhablani, Nihal V. Nayak, Debajyoti
Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han
Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong,
Harshit Pandey, Rachel Bawden, Thomas Wang, Tr-
ishala Neeraj, Jos Rozen, Abheesht Sharma, An-
drea Santilli, Thibault Févry, Jason Alan Fries, Ryan
Teehan, Teven Le Scao, Stella Biderman, Leo Gao,
Thomas Wolf, and Alexander M. Rush. 2022. Multi-
task prompted training enables zero-shot task gener-
alization. In ICLR. OpenReview.net.
Apoorv Saxena, Aditay Tripathi, and Partha P. Taluk-
dar. 2020. Improving multi-hop question answering
over knowledge graphs using knowledge base em-
beddings. In ACL, pages 4498–4507. Association for
Computational Linguistics.
Priyanka Sen, Sandeep Mavadia, and Amir Saffari. 2023.
Knowledge graph-augmented language models for
complex question answering.
In Proceedings of
the 1st Workshop on Natural Language Reasoning
and Structured Explanations (NLRSE), pages 1–8,
Toronto, Canada. Association for Computational Lin-
guistics.
Andrea Sottana, Bin Liang, Kai Zou, and Zheng Yuan.
2023. Evaluation metrics in the era of GPT-4: reli-
ably evaluating large language models on sequence
to sequence tasks. In EMNLP, pages 8776–8788.
Association for Computational Linguistics.
Yu Su, Huan Sun, Brian M. Sadler, Mudhakar Srivatsa,
Izzeddin Gur, Zenghui Yan, and Xifeng Yan. 2016.
On generating characteristic-rich question sets for
QA evaluation. In EMNLP, pages 562–572. The
Association for Computational Linguistics.
Alon Talmor and Jonathan Berant. 2018. The web as a
knowledge-base for answering complex questions. In
NAACL-HLT, pages 641–651. Association for Com-
putational Linguistics.
Yiming Tan, Dehai Min, Yu Li, Wenbo Li, Nan Hu,
Yongrui Chen, and Guilin Qi. 2023. Evaluation of
chatgpt as a question answering system for answering
complex questions. CoRR, abs/2303.07992.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier
Martinet, Marie-Anne Lachaux, Timothée Lacroix,
Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal
Azhar, Aurélien Rodriguez, Armand Joulin, Edouard
Grave, and Guillaume Lample. 2023a. Llama: Open
and efficient foundation language models. CoRR,
abs/2302.13971.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Al-
bert, Amjad Almahairi, Yasmine Babaei, Nikolay
Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti
Bhosale, et al. 2023b.
Llama 2: Open founda-
tion and fine-tuned chat models.
arXiv preprint
arXiv:2307.09288.
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot,
and Ashish Sabharwal. 2023. Interleaving retrieval
with chain-of-thought reasoning for knowledge-
intensive multi-step questions. In ACL (1), pages
10014–10037. Association for Computational Lin-
guistics.
Chaojie Wang, Yishi Xu, Zhong Peng, Chenxi Zhang,
Bo Chen, Xinrun Wang, Lei Feng, and Bo An. 2024.
keqing: knowledge-based question answering is a
nature chain-of-thought mentor of LLM.
CoRR,
abs/2401.00426.
Keheng Wang, Feiyu Duan, Sirui Wang, Peiguang Li,
Yunsen Xian, Chuantao Yin, Wenge Rong, and Zhang
Xiong. 2023a. Knowledge-driven cot: Exploring
faithful reasoning in llms for knowledge-intensive
question answering. CoRR, abs/2308.13259.
Yufei Wang, Wanjun Zhong, Liangyou Li, Fei Mi,
Xingshan Zeng, Wenyong Huang, Lifeng Shang,
Xin Jiang, and Qun Liu. 2023b.
Aligning large
language models with human: A survey.
CoRR,
abs/2307.12966.
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin
Guu, Adams Wei Yu, Brian Lester, Nan Du, An-
drew M. Dai, and Quoc V. Le. 2022a. Finetuned
language models are zero-shot learners. In ICLR.
OpenReview.net.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten
Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le,
and Denny Zhou. 2022b. Chain-of-thought prompt-
ing elicits reasoning in large language models. In
NeurIPS.
Xiaodong Wu, Ran Duan, and Jianbing Ni. 2023a. Un-
veiling security, privacy, and ethical concerns of chat-
gpt. CoRR, abs/2307.14192.
Yike Wu, Nan Hu, Sheng Bi, Guilin Qi, Jie Ren, Anhuan
Xie, and Wei Song. 2023b. Retrieve-rewrite-answer:
A kg-to-text enhanced llms framework for knowledge
graph question answering. CoRR, abs/2309.11206.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak
Shafran, Karthik R. Narasimhan, and Yuan Cao. 2023.
React: Synergizing reasoning and acting in language
models. In ICLR. OpenReview.net.
Donghan Yu, Sheng Zhang, Patrick Ng, Henghui
Zhu, Alexander Hanbo Li, Jun Wang, Yiqun Hu,
William Yang Wang, Zhiguo Wang, and Bing Xiang.
2023. Decaf: Joint decoding of answers and logical
forms for question answering over knowledge bases.
In ICLR. OpenReview.net.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel
Artetxe, Moya Chen, Shuohui Chen, Christopher
Dewan, Mona T. Diab, Xian Li, Xi Victoria Lin,
Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shus-
ter, Daniel Simig, Punit Singh Koura, Anjali Srid-
har, Tianlu Wang, and Luke Zettlemoyer. 2022.
OPT: open pre-trained transformer language mod-
els. CoRR, abs/2205.01068.
Yichi Zhang, Zhuo Chen, Yin Fang, Lei Cheng, Yanxi
Lu, Fangming Li, Wen Zhang, and Huajun Chen.
2023.
Knowledgeable preference alignment for
llms in domain-specific question answering. CoRR,
abs/2311.06503.
A
Experimental Details
A.1
Data Construction Details
To construct the supervised fine-tuning dataset, we
set the temperature to 0 and adopt GPT-3.5 Turbo.
We concatenate question q and its related subgraph
G′ using a prompt template to form the input x.
ChatGPT generates candidate knowledge represen-
tation k based on 3 examples as demonstrations
and input x under ICL paradigm. Given that, the
objective of k is to augment the performance of
the QA model, we evaluate the quality of k by ex-
amining the QA model’s results. We utilize k as
contextual knowledge for the QA model to gen-
erate the answer a. If the answer a encompasses
all the answer entities, knowledge k is considered
helpful for answering the question, and (x, k) is
used as an input-output pair for supervised training.
For the construction of the preference dataset, we
sample knowledge representations from the knowl-
edge rewriter after supervised fine-tuning. We set
the temperature to 1 to foster greater diversity. Dur-
ing preference annotation, given that the knowledge
rewriter aims to generate contextual knowledge
beneficial for QA, we first assess the quality of
knowledge representation based on the number of
answer entities they contain. This naïve strategy is
robust and cost-effective, avoiding additional API
calls for evaluations using ChatGPT, thus saving
time and reducing costs. We label the candidate
with the highest number of answer entities as the
preferred knowledge representation k+ and the one
with the fewest as the dispreferred k−. If the num-
ber of answer entities is the same, we select the two,
k1 and k2, with the greatest semantic difference by
using all-MiniLM-L6-v24 as the encoder. This se-
lection process ensures a significant semantic gap
between the two chosen representations, facilitat-
ing more rapid model convergence during training.
Subsequently, k1 and k2 serve as contextual knowl-
edge to prompt the QA model, yielding answers
a1 and a2. We annotate the preferred knowledge
representation k+ and the dispreferred knowledge
representation k−by evaluating the quality of a1
and a2 using ChatGPT. Finally, ChatGPT is used
4https://huggingface.co/sentence-transformers/all-
MiniLM-L6-v2
to paraphrase the preferred knowledge representa-
tion k+ into an enhanced version k++, forming a
preference pair k++ and k−for direct preference
optimization (DPO).
A.2
Datasets
GrailQA5 (Gu et al., 2021) is a challenging, large-
scale multi-hop KGQA benchmark. It is an English
dataset that utilizes Freebase (Bollacker et al., 2007,
2008) as KG. It spans 86 domains, such as Sports,
Location, and Computer Video Games, and com-
prises 64,331 questions (44,337 train, 6,763 dev,
13,231 test). This dataset features a large number of
entities and relations, complex logical forms, and
noise in entity mentions within the questions. The
training and dev sets provide annotated SPARQL
queries and answer entities, while the test set com-
prises only the questions. For evaluation conve-
nience, the dev set is used for testing.
GraphQuestions6
(Su
et
al.,
2016)
is
a
characteristic-rich
dataset
for
factoid
ques-
tion answering based on Freebase across 70
domains, like People, Astronomy, and Medicine.
This English dataset focuses on the following
question characteristics:
structure complexity,
function, commonness, paraphrasing, and answer
cardinality. It comprises 5,166 questions (2,771
train, 2,395 test), with nearly half requiring
multi-hop reasoning.
For each question, the
dataset provides corresponding SPARQL query
and answer entities.
A.3
Large Language Models
Llama-27 (Touvron et al., 2023b), an updated ver-
sion of Llama-1 (Touvron et al., 2023a), is devel-
oped using a training corpus comprising 2 trillion
tokens and features a context length twice that
of Llama-1. To better accomplish the knowledge
rewriting task, we select Llama-2-7B-Chat8.
Llama-39 (AI@Meta, 2024) is the latest model in
the Llama series. It is renowned for its mastery
of language nuances, contextual comprehension,
5This dataset is distributed under the CC BY-SA 4.0 license
and our utilization complies with the terms specified in the
license.
6This dataset is licensed under the Creative Commons
Attribution 4.0 and our usage aligns with the intended purposes
outlined in this license.
7The
license
of
Llama-2
is
available
at
https://ai.meta.com/resources/models-and-libraries/llama-
downloads/.
8https://huggingface.co/meta-llama/Llama-2-7b-chat-hf
9The
license
of
Llama-3
is
available
at
https://llama.meta.com/llama3/license.
and proficiency in executing complex tasks such as
translation and generating dialogues. We choose
Llama-3-8B-Instruct10 for knowledge rewriting.
Mistral11 (Jiang et al., 2023a) is an open-source
LLM developed by Mistral AI. We select the lat-
est instruction-tuned version, Mistral-7B-Instruct-
v0.312, as our QA model.
ChatGPT13, developed by OpenAI, is a milestone
in the era of LLMs. Its robust capabilities in natural
language understanding and generation facilitate
superior performance across various tasks. We
leverage GPT-3.5-turbo via the API14 for knowl-
edge rewriting and question answering.
All the LLMs above are general-domain models.
Regarding language support, Llama-2, Llama-3,
and Mistral only support English, while ChatGPT
is multilingual. In this study, the use of these LLMs
complies with their respective licenses or terms.
KG-to-Text Prompt
[Instruction]
Your task is to transform a knowledge graph to a
sentence or multiple sentences.
The knowledge
graph is: {triples}. The sentence is:
Summary Prompt
[Instruction]
Your task is to summarize the relevant knowledge
that is helpful to answer the question from the
following triples.
Triples: {triples}
Question: {question}
Knowledge:
CoTKR Prompt
[Instruction]
Your task is to summarize the relevant information
that is helpful to answer the question from the
following triples.
Please think step by step and
iteratively generate the reasoning chain and the
corresponding knowledge.
Triples: {triples}
Question: {question}
Table 4: Prompts for Knowledge Rewriting Methods.
10https://huggingface.co/meta-llama/Meta-Llama-3-8B-
Instruct
11It is under the Apache 2 License.
12https://huggingface.co/mistralai/Mistral-7B-Instruct-
v0.3
13The terms of use for ChatGPT are available at
https://openai.com/policies/terms-of-use/.
14https://api.openai.com/
Prompt for Question Answering with Triple/KG-
to-Text/Summary Knowledge
[Instruction]
Your task is to answer the question based on the
knowledge that might be relevant. Try to use the
original words from the given knowledge to answer
the question. But if it is not useful, just ignore it and
generate your own guess.
Knowledge: {knowledge}
Question: {question}
Answer:
Prompt
for
Question
Answering
with
CoTKR/CoTKR+PA Knowledge
[Instruction]
Your task is to answer the question based on the
reasoning chain that might be relevant. Try to use the
original words from the given knowledge to answer
the question. But if it is not useful, just ignore it and
generate your own guess.
Knowledge: {knowledge}
Question: {question}
Answer:
Prompt for Question Answering without Context
Question: {question}
Answer:
Table 5: Prompts for Question Answering.
A.4
Retrieval Methods Details
2-Hop subgraph is a naïve question-related context.
Most KBQA studies under RAG paradigms con-
sider triples within the N-hop subgraph of the head
entity as contextual knowledge (Baek et al., 2023;
Sen et al., 2023; Wang et al., 2024; Ko et al., 2024).
To retrieve the 2-hop subgraph around the head
entity, we execute SPARQL queries on Freebase.
Given the large size of the 2-hop subgraph, we
use all-MiniLM-L6-v215 to encode all 1-hop and
2-hop relations of the head entity and the question,
excluding meaningless relations, such as “com-
mon.topic.webpage”. Then, we select semantically
similar relation paths based on cosine similarity.
Finally, we sample the corresponding triples from
KG based on these relation paths. In this experi-
ment, we select the top 30 triples as our retrieval
results. However, the small size of the 2-hop sub-
graphs for some entities may result in fewer than
30 triples being retrieved.
BM25 (Robertson and Zaragoza, 2009) is a re-
trieval method based on TF-IDF scores of sparse
15https://huggingface.co/sentence-transformers/all-
MiniLM-L6-v2
Preference Annotation Prompt
[Instruction]
Your task is to evaluate the quality of two responses
to the question based on predefined criteria. Avoid
any position biases and ensure that the order in which
the responses were presented does not influence your
decision. Do not allow the length of the responses
to influence your evaluation.
Be as objective as
possible.
[Criteria]
For this evaluation, you should primarily consider
the following criteria:
Accuracy: The response should contain as many
answer entities as possible, and use the original
words of the answer entities.
Relevance: The response should be to the point of
the question.
Question: {question}
Answer: {answer}
Response A: {response A}
Response B: {response B}
[Evaluation Rule]
Begin your evaluation by comparing the two re-
sponses and provide a short explanation. Then output
only the single character: "A" if Response A is better,
"B" if Response B is better, and "C" for a tie. At the
end, repeat just the letter again by itself on a new line.
Table 6: Preference Annotation Prompt.
word matching between input questions and pas-
sages. This method is commonly used for inte-
grating multimodal data sources or text-based QA
(Wang et al., 2023a; Yu et al., 2023). We follow
the processing method in DecAF (Yu et al., 2023),
simply linearizing the 1-hop subgraph of the topic
entity as the article. We use the BM25 implemented
by Pyserini (Lin et al., 2021) and collect the triples
corresponding to the candidate articles as the re-
trieval result. Specifically, we initially retrieve 10
candidate articles, each containing up to 10 triples.
Subsequently, we remove any redundant triples or
those containing meaningless relations. Given the
limited context length of LLMs, we select the top
30 triples as the context information for question
answering. After filtering, the number of triples
in the candidate articles may be less than 30, thus
resulting in the retrieval subgraphs for some ques-
tions containing fewer than 30 triples.
Ground Truth Subgraph (GS) refers to a sub-
graph consisting of the triples necessary for an-
swering a question. In this experiment, we modify
the SPARQL queries provided in the datasets and
execute them on Freebase to obtain the ground truth
subgraphs. We use this subgraph to represent the
results of an ideal retriever, aiming to explore the
performance upper bound of different knowledge
rewriting strategies for the QA model.
Data Augmentation Prompt
[Instruction]
You are a knowledge graph summarizer for Question
Answering. I will give you "Question", "Triple",
"Answer" and "Knowledge". Your task is to para-
phrase the original "Knowledge" into a more helpful
representation format for Question Answering.
The "Paraphrased Knowledge" should contain the
original words of all the answer entities.
Question: {question}
Triple: {triples}
Knowledge: {knowledge}
Paraphrased Knowledge:
Table 7: Data Augmentation Prompt.
A.5
Implementation Details
We utilize LoRA (Hu et al., 2022) to achieve
parameter-efficient fine-tuning. For supervised fine-
tuning and DPO, the batch size, learning rate, lora
rank, lora alpha, and lora dropout are set to 128,
1e-4, 64, 128, and 0.05, respectively. In supervised
fine-tuning, we train for 10 epochs and save the
best model based on validation set results. In DPO,
we observe that more training steps may lead to
decreased model performance. Consequently, we
train for 1 to 2 epochs on GraphQuestions using
approximately 2,000 training samples, and for 5
to 20 steps on GrailQA using 640 to 2560 training
samples. During inference, the temperature is set
to 0 for ChatGPT and 0.01 for open-source LLMs,
ensuring relatively stable output. All the parameter
settings mentioned above are the optimal results
after multiple trials.
Our implementation utilizes PyTorch16, Trans-
formers17, DeepSpeed18, Datasets19, PEFT20, and
TRL21. We use Datasets for data preprocessing.
Both training and inference are based on PyTorch
and Transformers. Supervised fine-tuning and DPO
are implemented using PEFT, TRL, and Deep-
Speed. Experiments are conducted on 4 NVIDIA
A100-SXM4-40GB GPUs, with each training or
inference session completed within one day. Due to
16https://pytorch.org/
17https://huggingface.co/docs/transformers/en/index
18https://www.deepspeed.ai/
19https://huggingface.co/docs/datasets/en/index
20https://huggingface.co/docs/peft/en/index
21https://huggingface.co/docs/trl/en/index
Question: what is the unit of area that the measurement system that have an electric field strength units of volt per metre have?
Answer: Square kilometer, Square meter
KR Methods
KR Result
QA Response
Triple
(International
System
of
Units,
elec-
tric_field_strength_units, Volt per metre) (International
System of Units, area_units, Square meter) (Interna-
tional System of Units, area_units, Square kilometer)
(Volt per metre, strength_in_volts_per_meter, 1.0) ...
Square meter.
KG-to-Text
The International System of Units includes various units
for different physical quantities such as length, area,
potential, substance, conductance, volume, resistance,
current, force, and pressure. For example, the unit for
electric field strength in volts per meter is 1.0. Addi-
tionally, it covers a wide range of units including square
kilometers, square meters ...
The unit of area in the International System of Units that
has an electric field strength unit of volt per meter is
square meters (m²).
Summary
The International System of Units includes various mea-
surement systems for different physical quantities such
as length, area, volume, conductance, resistance, current,
force, and pressure. The measurement system with an
electric field strength unit of Volt per meter does not
directly relate to a specific area unit within the Interna-
tional System of Units.
There is no direct relationship between the measurement
system with an electric field strength unit of Volt per
meter and any specific area unit within the International
System of Units.
CoTKR
Reason: I need to know the unit of area associated
with the measurement system that has an electric field
strength unit of Volt per meter.
Summarize: The International System of Units includes
various units for different physical quantities such as
length, area, volume, etc.
Reason: I need to find the specific unit of area in the
International System of Units that corresponds to the
electric field strength unit of Volt per meter.
Summarize: The International System of Units includes
units like Square kilometer, Square meter, etc. for mea-
suring area.
The unit of area in the measurement system that has an
electric field strength unit of Volt per meter is Square
meter (m²).
CoTKR+PA
Reason: I need to know which measurement system
includes the unit for electric field strength in Volt per
meter.
Knowledge: The International System of Units contains
the unit for electric field strength in Volt per meter.
Reason: I need to identify the unit of area within the
International System of Units.
Knowledge: The International System of Units includes
the units Square kilometer and Square meter as area
units.
The unit of area within the International System of Units
that the measurement system having an electric field
strength unit of volt per metre has is either Square kilo-
meter or Square meter.
Table 8: An example of knowledge rewriting results for different methods. We use Llama-3 and Mistral as the
Knowledge Rewriting (KR) backbone and the QA model, respectively.
the high computational costs of LLMs, we conduct
each experiment once and then report the results.
All the prompts involved in this experiment are
as follows. Table 4 shows the prompts for different
knowledge rewriting methods. Table 5 shows the
prompts for question answering. Table 6 shows the
preference annotation prompt.
A.6
Prompt for Data Augmentation
Table 7 shows the data augmentation prompt.
B
Case Study
In this section, we compare different knowledge
rewriting strategies through an example. As illus-
trated in Table 8, the knowledge generated by both
Triple and KG-to-Text contains excessive redun-
dant information. This redundancy complicates the
process for the QA model, making it challenging
to extract relevant knowledge. Summary strug-
gles to extract useful information when faced with
an abundance of triples. In contrast, CoTKR and
CoTKR+PA summarize the most pertinent knowl-
edge in the rewriting step, thereby enabling the
QA model to provide a concise and accurate an-
swer. Furthermore, after preference alignment, our
knowledge rewriter is capable of generating more
natural reasoning steps, significantly enhancing its
applicability to KGQA.
C
Additional Experimental Results
C.1
Experiments on ComplexWebQuestions
To evaluate the robustness of CoTKR, we
conduct our experiments on ComplexWebQues-
tions(Talmor and Berant, 2018). We utilize Chat-
GPT as the knowledge rewriter, Mistral as the
question-answering model, and 2-Hop as the re-
trieval method.
The experimental results, pre-
sented in Table 9, demonstrate the effectiveness
of CoTKR.
Methods
Acc
Recall
EM
No Knowledge
36.82
31.30
27.62
Triple
39.29
33.94
30.86
KG-to-Text
35.96
31.53
28.96
Summary
38.53
33.87
30.89
CoTKR
40.70
35.74
32.72
Table 9: Experiments on ComplexWebQuestions use
ChatGPT as the knowledge rewriter, Mistral as the QA
model, and 2-Hop for retrieval.
C.2
Knowledge Rewriter with GPT-4
To assess the applicability of CoTKR to GPT-4,
we further conduct the experiments with GPT-4
as the knowledge rewriter, Mistral as the question-
answering model, and 2 hop as the retrieval method
on 1,000 test questions from GrailQA. The detailed
results are presented in Table 10. The findings
show that CoTKR outperforms other approaches,
with CoTKR utilizing GPT-4 achieving the highest
performance. This suggests that employing a more
advanced LLM backbone, such as GPT-4, leads to
superior outcomes.
Methods
Acc
Recall
EM
No Knowledge
29.10
21.87
18.80
Triple
54.30
47.15
42.40
KG-to-Text
53.20
45.42
40.60
Summary
56.00
48.63
43.60
CoTKR
57.50
52.16
48.20
Table 10: Experiments on 1,000 GrailQA test questions
use GPT-4 for rewriting, Mistral for QA, and 2-Hop as
the retrieval method.
C.3
Time Analysis
We conduct experiments to analyze the average
time cost of the knowledge rewriting methods
discussed in this paper.
We adopt Llama-3 as
the knowledge rewriter, Mistral as the question-
answering model, and 2-Hop as the retrieval
method.
The experiments are conducted on
GraphQuestions, utilizing one A100-SXM4-40GB
GPU. The average runtime for each question by
different methods is shown in Table 11 (unit: sec-
onds). The average runtime of each question for
all methods is within an acceptable range (i.e., less
than 1.5 seconds). Although our method is the
most time-consuming, it exhibits a clear advantage
in performance.
C.4
GPT-4-score as Evaluation Metrics
Given the powerful natural language understanding
and generation capabilities of closed-source large
Process
No Knowledge
Triple
KG-to-Text
Summary
CoTKR
Rewrite
0
0
1.0828
0.4615
0.9835
Answer
0.3787
0.5998
0.2701
0.2502
0.4414
Rewrite+Answer
0.3787
0.5998
1.3529
0.7117
1.4249
Table 11: Time analysis on GraphQuestions (seconds)
using Llama-3 as the knowledge rewriter, Mistral for
question answering, and 2-Hop for retrieval.
models, many existing works employ ChatGPT
as an evaluator to provide high-quality evaluation
results (Sottana et al., 2023; Liu et al., 2023; Min
et al., 2023). In our approach, we use GPT-4 as the
evaluator to assess whether all answer entities are
present in the responses. We refer to this evaluation
metric as GPT-4-score. Compared to EM, this
metric is more flexible, as LLMs are capable of
recognizing synonyms of answer entities. We use
it to evaluate the first 300 questions from GrailQA
using Llama-3 as knowledge rewriter, ChatGPT as
question-answering model, and 2 hop as retrieval
method. We provide the prompt for GPT-4-score
in Table 12.
GPT-4-score Prompt
[Instruction]
Your task is to evaluate the quality of the response to
the question. You should consider whether all the
answer entities appear in the response.
Question: {question}
Answer: {answer}
Response: {response}
Begin your evaluation by comparing the response
and the answer and provide a short explanation.
Then output only the single number: "1" if all the
answer entities appear in the response, and "0" if not.
At the end, repeat just the number again by itself on
a new line.
Table 12: GPT-4-score Prompt.
The experimental results are shown in Table 13.
The results indicate that our implemented GPT-4-
score is effective and consistent with the outcomes
reflected by other evaluation metrics. Furthermore,
this also demonstrates that CoTKR possesses sig-
nificant advantages compared to other knowledge
rewriting methods.
C.5
Qualitative Analysis on Data
Augmentation
To clearly demonstrate the importance of data aug-
mentation, we perform a qualitative analysis. The
results reveal that data augmentation enhances
performance in three key areas: reducing redun-
dant reasoning steps, supplementing critical infor-
Methods
GPT-4-score
No Knowledge
0.2767
Triple
0.4700
KG-to-Text
0.4749
Summary
0.4800
CoTKR
0.5167
CoTKR+PA
0.5567
Table 13: GPT-4-score on GraphQuestions. We use
Llama-3 as knowledge rewriter, ChatGPT as question-
answering model, and 2 hop as retrieval method.
mation, and generating more concise summaries.
These improvements are exemplified by three rep-
resentative cases, as shown in Table 14.
C.6
Detailed Experimental Results
This section presents all the experimental results
of this study. As shown in Table 15, Table 16,
and Table 17, CoTKR/CoTKR+PA achieves the
best performance in most scenarios. This indi-
cates that CoTKR is effective for both open-source
LLMs after training and closed-source LLMs using
ICL. Besides, the results also reveal the robustness
of CoTKR, demonstrating its applicability across
KGQA systems with various retrieval methods and
QA models.
D
AI Assistants in Research or Writing
In this research, we primarily utilize ChatGPT for
the construction of training data and as the foun-
dational model for knowledge rewriting and QA.
For academic writing, ChatGPT is used to correct
grammatical errors.
Effect
Question
k+
k++
Avoiding Redundant
Reasoning Step
Katy börner is the curator for
what exhibition?
Reason: I need to know which exhibi-
tion Katy Börner curated.
Summarize: Katy Börner curated the ex-
hibition "Places & Spaces".
Reason: I need to know the subjects of
the exhibition "Places & Spaces" curated
by Katy Börner.
Summarize: The subjects of the exhibi-
tion "Places & Spaces" curated by Katy
Börner include Cartography, Informa-
tion visualization, Information design,
and Scientometrics.
Reason: I need to know which exhibi-
tion Katy Börner curated.
Summarize: Katy Börner curated the ex-
hibition "Places & Spaces".
Key Points
Supplementation
Person of interest deals with
how many tv subjects?
Reason: I need to know how many TV
subjects are associated with "Person of
Interest".
Summarize: "Person of Interest" is asso-
ciated with multiple TV subjects such as
Espionage, Mass surveillance, Hacker,
and Vigilante.
Reason: I need to know how many TV
subjects are associated with "Person of
Interest".
Summarize: "Person of Interest" deals
with a total of 5 TV subjects including
Espionage, Mass surveillance, Hacker,
Surveillance, and Vigilante.
More Concise
Summarization
Who are the owners of gree?
Reason:
I need to know who owns
GREE, Inc.
Summarize: GREE, Inc. is owned by
GREE, Inc. itself, according to the triple
(GREE, internet.website.owner, GREE,
Inc.).
Reason:
I need to know who owns
GREE, Inc.
Summarize: GREE, Inc. is owned by
GREE, Inc. itself.
Table 14: Representative cases for improvements through data augmentation.
KR LLMs
Methods
GrailQA
GraphQuestions
Acc
Recall
EM
Acc
Recall
EM
ChatGPT as QA model
None
No Knowledge
28.91
22.81
20.14
35.87
25.76
22.09
Triple
57.76
49.67
44.73
55.03
46.65
41.63
Llama-2
KG-to-Text
54.75
47.35
42.44
49.73
40.00
33.74
Summary
58.14
51.38
46.38
52.94
44.70
38.41
CoTKR
58.64
52.33
47.88
51.36
45.20
39.96
CoTKR+PA
59.25
53.52
49.64
56.78
47.99
42.46
Llama-3
KG-to-Text
55.76
48.41
43.90
52.40
45.06
39.83
Summary
57.55
51.06
46.80
54.95
46.86
40.75
CoTKR
58.33
52.55
48.65
53.19
47.23
43.17
CoTKR+PA
61.51
56.08
52.67
56.37
49.31
45.26
ChatGPT
KG-to-Text
56.32
49.05
44.73
53.53
45.59
41.17
Summary
58.54
51.81
47.29
55.62
48.93
44.97
CoTKR
59.87
53.19
49.02
54.28
48.18
44.68
Mistral as QA model
None
No Knowledge
29.44
23.13
20.30
38.20
26.92
22.13
Triple
54.47
47.78
43.25
51.32
45.97
41.67
Llama-2
KG-to-Text
49.49
42.91
38.41
44.59
37.98
32.82
Summary
54.10
47.79
43.15
49.85
42.33
36.45
CoTKR
56.75
51.10
46.71
50.19
43.73
38.54
CoTKR+PA
58.15
52.98
49.13
55.07
47.02
41.71
Llama-3
KG-to-Text
50.64
44.32
40.13
49.06
43.04
38.25
Summary
53.84
47.71
43.49
52.03
44.30
38.50
CoTKR
56.47
51.33
47.36
52.65
46.48
42.21
CoTKR+PA
59.31
54.13
50.24
54.82
47.76
43.09
ChatGPT
KG-to-Text
51.04
44.87
40.97
49.14
43.04
38.83
Summary
54.44
48.16
43.97
52.28
47.10
43.30
CoTKR
57.28
51.14
47.09
52.82
47.13
43.55
Table 15: The overall results of CoTKR and the baselines on GrailQA and GraphQuestions using 2-Hop as retrieval
method. For each combination of the Knowledge Rewriter (KR) LLM and the QA model, the best and second-best
results are highlighted in bold and underlined, respectively.
KR LLMs
Methods
GrailQA
GraphQuestions
Acc
Recall
EM
Acc
Recall
EM
ChatGPT as QA model
None
No Knowledge
28.91
22.81
20.14
35.87
25.76
22.09
Triple
58.42
49.06
43.87
48.43
40.44
36.83
Llama-2
KG-to-Text
52.80
43.93
39.01
41.67
32.09
27.68
Summary
57.62
49.13
43.80
46.47
36.84
31.44
CoTKR
58.32
50.32
45.57
45.34
37.21
32.73
CoTKR+PA
58.32
50.77
46.41
49.52
40.06
35.16
Llama-3
KG-to-Text
55.14
46.64
41.96
44.68
34.51
30.19
Summary
59.19
50.83
46.19
44.97
36.97
32.44
CoTKR
58.89
51.13
46.93
46.18
38.59
35.03
CoTKR+PA
59.69
52.34
48.14
50.44
42.03
37.58
ChatGPT
KG-to-Text
55.61
47.00
42.38
46.14
36.64
31.98
Summary
58.76
50.47
45.66
46.14
39.20
35.66
CoTKR
59.66
51.06
46.40
45.18
38.03
34.36
Mistral as QA model
None
No Knowledge
29.44
23.13
20.30
38.20
26.92
22.13
Triple
55.58
47.10
42.07
43.34
36.98
33.86
Llama-2
KG-to-Text
47.33
39.90
35.64
35.24
28.35
25.05
Summary
52.94
44.94
40.12
43.47
34.56
29.52
CoTKR
55.69
48.37
43.77
45.39
37.07
32.73
CoTKR+PA
57.13
50.17
45.94
49.02
39.46
34.53
Llama-3
KG-to-Text
49.83
42.40
38.07
39.21
31.86
28.48
Summary
54.33
46.73
42.35
41.63
34.41
30.15
CoTKR
56.76
49.75
45.57
44.38
37.14
33.57
CoTKR+PA
58.27
51.00
46.72
48.14
39.34
34.91
ChatGPT
KG-to-Text
50.35
42.69
38.40
39.46
33.36
29.52
Summary
54.98
47.15
42.63
41.42
35.60
31.86
CoTKR
56.84
48.87
44.17
43.42
36.23
32.57
Table 16: The overall results of CoTKR and the baselines on GrailQA and GraphQuestions using BM25 as retrieval
method. For each combination of the Knowledge Rewriter (KR) LLM and the QA model, the best and second-best
results are highlighted in bold and underlined, respectively.
KR LLMs
Methods
GrailQA
GraphQuestions
Acc
Recall
EM
Acc
Recall
EM
ChatGPT as QA model
None
No Knowledge
28.91
22.81
20.14
35.87
25.76
22.09
Triple
77.41
67.14
61.47
83.17
70.89
62.76
Llama-2
KG-to-Text
80.05
70.20
65.06
76.24
64.47
57.58
Summary
85.14
75.90
70.03
80.88
69.31
60.00
CoTKR
87.18
78.46
73.56
80.67
70.73
63.72
CoTKR+PA
87.79
79.86
75.50
83.67
75.02
68.85
Llama-3
KG-to-Text
82.17
73.10
68.28
78.66
68.32
59.92
Summary
85.44
77.01
72.07
83.55
72.47
64.47
CoTKR
88.27
80.35
75.93
85.51
75.89
69.14
CoTKR+PA
91.48
84.02
79.93
87.85
79.66
73.95
ChatGPT
KG-to-Text
80.39
71.19
66.46
83.05
73.60
67.22
Summary
86.53
77.88
72.87
88.39
79.26
72.99
CoTKR
87.77
78.97
74.38
89.81
80.09
73.49
Mistral as QA model
None
No Knowledge
29.44
23.13
20.30
38.20
26.92
22.13
Triple
74.12
65.83
60.79
83.26
72.17
64.22
Llama-2
KG-to-Text
72.47
64.18
59.53
71.19
59.62
51.98
Summary
78.87
70.43
64.71
78.71
67.45
58.29
CoTKR
84.27
76.57
71.60
78.66
68.75
61.80
CoTKR+PA
85.81
79.26
74.52
81.38
72.88
67.06
Llama-3
KG-to-Text
74.54
66.58
62.38
76.37
66.56
58.75
Summary
79.64
71.65
66.89
80.84
70.07
62.21
CoTKR
84.73
77.86
73.31
84.18
75.04
68.64
CoTKR+PA
87.80
81.24
76.65
84.63
77.38
73.24
ChatGPT
KG-to-Text
72.53
65.06
60.92
81.71
72.35
65.43
Summary
81.34
72.53
67.53
86.47
77.15
70.56
CoTKR
84.77
76.72
72.11
88.60
78.95
72.36
Table 17: The overall results of CoTKR and the baselines on GrailQA and GraphQuestions using GS as retrieval
method. For each combination of the Knowledge Rewriter (KR) LLM and the QA model, the best and second-best
results are highlighted in bold and underlined, respectively.