Question:
According to this paper, which statements below explain the capability of the RoG framework?
A. RoG retrieves relevant reasoning paths from the KG， which is created by the author containing question entities in these two datasets.
B. Reasoning on a faithful KGs could migate the hallucination problem of LLMs which enhance the quality of the response.
C. RoG extracts not only the knowledge contained within knowledge graphs, but also captures the structural information of the graph.
D. RoG is a specialized framework for GPTs, it cannot be utilized with DeepSeek-R1 when implementing long chain of thought.

Answer: BC

Large language models (LLMs) have demonstrated impressive reasoning abilities in complex tasks. However, they lack up-to-date knowledge and experience hallucinations during reasoning, which can lead to incorrect reasoning processes and diminish their performance and trustworthiness. Knowledge graphs (KGs), which capture vast amounts of facts in a structured format, offer a reliable source of knowledge for reasoning. Nevertheless, existing KG-based LLM reasoning methods only treat KGs as factual knowledge bases and overlook the importance of their structural information for reasoning. In this paper, we propose a novel method called reasoning on graphs (RoG) that synergizes LLMs with KGs to enable faithful and interpretable reasoning. Specifically, we present a planning-retrieval-reasoning framework, where RoG first generates relation paths grounded by KGs as faithful plans. These plans are then used to retrieve valid reasoning paths from the KGs for LLMs to conduct faithful reasoning. Furthermore, RoG not only distills knowledge from KGs to improve the reasoning ability of LLMs through training but also allows seamless integration with any arbitrary LLMs during inference. Extensive experiments on two benchmark KGQA datasets demonstrate that RoG achieves state-of-the-art performance on KG reasoning tasks and generates faithful and interpretable reasoning results.

Published as a conference paper at ICLR 2024
REASONING
ON GRAPHS:
FAITHFUL
AND INTER-
PRETABLE LARGE LANGUAGE MODEL REASONING
Linhao Luo, Yuan-Fang Li, Gholamreza Haffari
Monash University
Australia
{linhao.luo,yuanfang.li,Gholamreza.Haffari}@monash.edu
Shirui Pan∗
Griffith University
Australia
s.pan@griffith.edu.au
ABSTRACT
Large language models (LLMs) have demonstrated impressive reasoning abili-
ties in complex tasks. However, they lack up-to-date knowledge and experience
hallucinations during reasoning, which can lead to incorrect reasoning processes
and diminish their performance and trustworthiness. Knowledge graphs (KGs),
which capture vast amounts of facts in a structured format, offer a reliable source
of knowledge for reasoning. Nevertheless, existing KG-based LLM reasoning
methods only treat KGs as factual knowledge bases and overlook the importance
of their structural information for reasoning. In this paper, we propose a novel
method called reasoning on graphs (RoG) that synergizes LLMs with KGs to
enable faithful and interpretable reasoning. Specifically, we present a planning-
retrieval-reasoning framework, where RoG first generates relation paths grounded
by KGs as faithful plans. These plans are then used to retrieve valid reasoning
paths from the KGs for LLMs to conduct faithful reasoning. Furthermore, RoG
not only distills knowledge from KGs to improve the reasoning ability of LLMs
through training but also allows seamless integration with any arbitrary LLMs dur-
ing inference. Extensive experiments on two benchmark KGQA datasets demon-
strate that RoG achieves state-of-the-art performance on KG reasoning tasks and
generates faithful and interpretable reasoning results1.
1
INTRODUCTION
Large language models (LLMs) have shown great performance in many NLP tasks (Brown et al.,
2020; Bang et al., 2023). What’s especially striking is their ability to handle complex tasks through
reasoning (Wei et al., 2022; Huang & Chang, 2023). To further unleash LLMs’ reasoning ability, the
plan-and-solve paradigm (Wang et al., 2023c) has been proposed, in which LLMs are prompted to
generate a plan and execute each reasoning step. In this way, LLMs decompose complex reasoning
tasks into a series of sub-tasks and solve them step by step (Khot et al., 2022).
Despite their success, LLMs are still limited by the lack of knowledge and prone to hallucinations
during reasoning, which can lead to errors in reasoning processes (Hong et al., 2023; Wang et al.,
2023b). For example, as shown in Figure 1, LLMs do not have the latest knowledge and hallucinate
an incorrect reasoning step: “has a daughter”. These issues largely diminish the performance and
trustworthiness of LLMs in high-stakes scenarios, such as legal judgment and medical diagnosis.
To tackle the issues, knowledge graphs (KGs) have been incorporated to improve the reasoning
ability of LLMs (Pan et al., 2024; Luo et al., 2023a). KGs capture abundant factual knowledge in a
structured format, which provides a faithful knowledge source for reasoning. As a typical reasoning
∗Corresponding author.
1Code and data are available at: https://github.com/RManLuo/reasoning-on-graphs
1
Published as a conference paper at ICLR 2024
Question
Sorry, I do not have knowledge after Sept. 2021.
Could you provide some additional information?
Question
Justin Bieber is the child of Jeremy Bieber, who
has a daughter named Allie Bieber. Thus, the
brother of Justin Bieber is Allie Bieber.
Lack of Knowledge
Hallucination
Who is the brother of Justin Bieber
What product did Apple release in 2023?
Triple: (Iphone 15,
, 2023)
Relation path:
Output
Output
Factual Knowledge
Reasoning Guidance
Figure 1: The issues of lack of knowledge and hallucination in LLMs reasoning and how they can
be addressed by triples and relation paths from KGs.
task, knowledge graph question answering (KGQA) aims to obtain answers based on knowledge
from KGs (Sun et al., 2019). Previous works that jointly use KGs and LLMs for KGQA reasoning
can be broadly divided into two categories: 1) semantic parsing methods (Lan & Jiang, 2020; Ye
et al., 2022), which use LLMs to convert questions into logical queries that are executed on KGs
to obtain answers; and 2) retrieval-augmented methods (Li et al., 2023; Jiang et al., 2023), which
retrieve triples from KGs as knowledge context and uses LLMs to obtain the final answers.
Although semantic parsing methods can generate more accurate and interpretable results by lever-
aging reasoning on KGs, the generated logical queries can often be non-executable and yield no
answers, due to syntax and semantic limitations (Yu et al., 2022a). Retrieval-augmented methods
are more flexible and exploit the ability of LLMs for reasoning. However, they only treat KGs as
factual knowledge bases and overlook the importance of their structural information for reasoning
(Jiang et al., 2022). For instance, as shown in Figure 1, a relation path, which is a sequence of rela-
tions, “child of→has son” can be used to obtain answers to the question “Who is the brother
of Justin Bieber?”. Therefore, it is essential to enable LLMs to directly reason on KGs to achieve
faithful and interpretable reasoning.
In this paper, we propose a novel method called reasoning on graphs (RoG) that synergizes LLMs
with KGs to conduct faithful and interpretable reasoning. To alleviate the issues of hallucinations
and lack of knowledge, we present a planning-retrieval-reasoning framework, where RoG first gen-
erates relation paths grounded by KGs as faithful plans via the planning module. These plans are
then used to retrieve valid reasoning paths from KGs to conduct faithful reasoning by the retrieval-
reasoning module. In this way, we not only retrieve the latest knowledge from KGs but also consider
the guidance of KG structure for reasoning and explanations. Moreover, the planning module of RoG
can be plug-and-play with different LLMs during inference to improve their performance. Based on
this framework, RoG is optimized by two tasks: 1) planning optimization, where we distill knowl-
edge from KGs into LLMs to generate faithful relation paths as plans; and 2) retrieval-reasoning
optimization, where we enable LLMs to conduct faithful reasoning based on retrieved paths and gen-
erate interpretable results. We conduct extensive experiments on two benchmark KGQA datasets,
and the results demonstrate that RoG achieves state-of-the-art performance on KG reasoning tasks
and generates faithful and interpretable reasoning results.
2
RELATED WORK
LLM Reasoning Prompt. Many studies have been proposed to harness the reasoning ability of
LLMs to handle complex tasks through prompting (Wei et al., 2022; Wang et al., 2022; Yao et al.,
2023; Besta et al., 2023). Plan-and-solve (Wang et al., 2023c) prompts LLMs to generate a plan and
conduct reasoning based on it. DecomP (He et al., 2021) prompts LLMs to decompose the reasoning
task into a series of sub-tasks and solve them step by step. However, the problem of hallucinations
and lack of knowledge affect the faithfulness of LLMs’ reasoning. ReACT (Yao et al., 2022) treats
LLMs as agents, which interact with the environment to get the latest knowledge for reasoning.
To explore faithful reasoning, FAME (Hong et al., 2023) introduces the Monte-Carlo planning to
generate faithful reasoning steps. RR (He et al., 2022) and KD-CoT Wang et al. (2023b) further
retrieve relevant knowledge from KGs to produce faithful reasoning plans for LLMs.
Knowledge Graph Question Answering (KGQA). Conventional embedding-based methods rep-
resent the entities and relations in embedding space and design special model architectures (e.g.,
Key-Value memory networks, sequential models, and graph neural networks) to reason answers
2
Published as a conference paper at ICLR 2024
① Planning
Prompt: Please generate
helpful relation paths for
answering the question.
Relation Paths
Knowledge Graphs (KGs)
Joe Biden
President
profession
USA
Scranton
born_in
city_of
Washington
D.C.
work_in
city_of
University of
Delaware
graduate_from
located_in
Jill Biden
marry_to
Democratic
Party
member_of
New York
city_of
Miami
city_of
Reasoning Paths
Large Language Models (LLMs)
② Retrieval
③ Reasoning
Prompt: Please answer the
question based on the reasoning
paths and explain why.
Answer: The answer is
USA, because .....
Question: What is the
nationality of Joe Biden?
Planning Optimization
Reasoning Optimization
Inference
Instruction
Finetuning
Figure 2: The overall framework of reasoning on graphs (RoG). 1) given a question, we first prompt
LLMs to generate several relation paths that are grounded by KGs as plans. 2) Then, we retrieve
reasoning paths from KGs using the plans. 3) Finally, we conduct faithful reasoning based on the
retrieved reasoning paths and generate answers with interpretable explanations. The orange and red
rectangles denote the entities mentioned in the question and answer, respectively.
(Miller et al., 2016; He et al., 2021; Yasunaga et al., 2021). To integrate LLMs for KGQA, retrieval-
augmented methods aim to retrieve the relative facts from the KGs to improve the reasoning per-
formance (Li et al., 2023; Karpukhin et al., 2020). Recently, UniKGQA (Jiang et al., 2022) which
unifies the graph retrieval and reasoning process into a single model with LLMs, achieves STOA per-
formance. Semantic parsing methods convert the question into a structural query (e.g., SPARQL)
by LLMs, which can be executed by a query engine to reason the answers on KGs (Sun et al., 2020;
Lan & Jiang, 2020). However, these methods heavily rely on the quality of generated queries. If the
query is not executable, no answers will be generated. DECAF (Yu et al., 2022a) combines semantic
parsing and LLMs reasoning to jointly generate answers, which also reach salient performance on
KGQA tasks.
3
PRELIMINARY
Knowledge Graphs (KGs) contain abundant factual knowledge in the form of a set of triples: G =
{(e, r, e′)|e, e′ ∈E, r ∈R}, where E and R denote the set of entities and relations, respectively.
Relation Paths are a sequence of relations: z = {r1, r2, . . . , rl}, where ri ∈R denotes the i-th
relation in the path and l denotes the length of the path.
Reasoning Paths are the instances of a relation path z in KGs: wz = e0
r1
−→e1
r2
−→. . .
rl
−→el,
where ei ∈E denotes the i-th entity and ri denotes the i-th relation in the relation path z.
Example 1. Given a relation path: z = marry to →father of, a reasoning path instance
could be: wz = Alice
marry to
−−−−−−→Bob
father of
−−−−−−→Charlie, which denotes “Alice” is married to
“Bob” and “Bob” is the father of “Charlie”.
Knowledge Graph Question Answering (KGQA) is a typical reasoning task based on KGs. Given
a natural language question q and a KG G, the task aims to design a function f to predict answers
a ∈Aq based on knowledge from G, i.e., a = f(q, G). Following previous works (Sun et al., 2019;
Jiang et al., 2022), we assume the entities eq ∈Tq mentioned in q and answers a ∈Aq are labeled
and linked to the corresponding entities in G, i.e., Tq, Aq ⊆E.
4
APPROACH
In this section, we introduce our method: reasoning on graphs (RoG), which contains two compo-
nents: 1) a planning module that generates relation paths grounded by KGs as faithful plans; 2)
3
Published as a conference paper at ICLR 2024
a retrieval-reasoning module that first retrieves valid reasoning paths from KGs according to the
plans, then conducts faithful reasoning based on retrieved reasoning paths and generates answers
with interpretable explanations. The overall framework of RoG is illustrated in Figure 2.
4.1
REASONING ON GRAPHS: PLANNING-RETRIEVAL-REASONING
Recently, many techniques have been explored to improve the reasoning ability of LLMs by plan-
ning, which first prompts LLMs to generate a reasoning plan and then conduct reasoning based on it
(Wang et al., 2023c). However, LLMs are known for having hallucination issues, which are prone to
generating incorrect plans and leading to wrong answers (Ji et al., 2023). To address this issue, we
present a novel planning-retrieval-reasoning framework, which makes the reasoning plans grounded
by KGs and then retrieves faithful reasoning paths for LLM reasoning.
Relation paths, which capture semantic relations between entities, have been utilized in many rea-
soning tasks on KGs (Wang et al., 2021; Xu et al., 2022). Moreover, compared to the dynamically
updated entities, the relations in KGs are more stable (Wang et al., 2023a). By using relation paths,
we can always retrieve the latest knowledge from KGs for reasoning. Therefore, relation paths can
serve as faithful plans for reasoning the answer to KGQA task.
Example 2. Given a question “Who is the child of Alice”, we can generate a relation path as the
plan: z = marry to →father of. This relation path expresses the plan: 1) find the person that
“Alice” is married to; 2) find the child of that person. We can execute the plan (relation path) by
retrieving a reasoning path from KGs as: wz = Alice
marry to
−−−−−−→Bob
father of
−−−−−−→Charlie. Finally,
we can answer the question based on the reasoning path, which is “Charlie”.
By treating relation paths as plans, we can make sure the plans are grounded by KGs, which enables
LLMs to conduct faithful and interpretable reasoning on graphs. In a nutshell, we formulate our
RoG as an optimization problem that aims to maximize the probability of reasoning the answer from
a knowledge graph G w.r.t the question q by generating relation paths z as the plan:
Pθ(a|q, G) =
X
z∈Z
Pθ(a|q, z, G)Pθ(z|q),
(1)
where θ denotes the parameters of LLMs, z denotes the relation paths (plans) generated by LLMs,
and Z denotes the set of possible relation paths. The latter term Pθ(z|q) is the probability of gen-
erating a faithful relation path z grounded by KG, given the question q, which is realized by the
planning module. The former term Pθ(a|q, z, G) is the probability of reasoning an answer a given
the question q, relation path z, and KG G, which is computed by the retrieval-reasoning module.
4.2
OPTIMIZATION FRAMEWORK
Despite the advantage of generating relation paths as plans, the LLMs have zero knowledge of the
relations contained in KGs. Therefore, LLMs cannot directly generate relation paths grounded by
KGs as faithful plans. Moreover, LLMs might not understand the reasoning paths correctly and
conduct reasoning based on them. To address these issues, we design two instruction tuning tasks:
1) planning optimization, which distills the knowledge from KGs into LLMs to generate faithful
relation paths as plans; 2) retrieval-reasoning optimization, which enables LLMs to reason based on
the retrieved reasoning paths.
The objective function in equation 1 can be optimized by maximizing the evidence lower bound
(ELBO) (Jordan et al., 1999), which is formulated as
log P(a|q, G) ≥Ez∼Q(z)[log Pθ(a|q, z, G)] −DKL(Q(z)∥Pθ(z|q)),
(2)
where Q(z) denotes the posterior distribution of faithful relation paths grounded by KGs. The
latter term minimizes the KL divergence between the posterior and the prior, which encourages
LLMs to generate faithful relation paths (planning optimization). The former term maximizes the
expectation that retrieval-reasoning module generates correct answers based on the relation paths
and KGs (retrieval-reasoning optimization).
Planning optimization. In planning optimization, we aim to distill the knowledge from KGs into
LLMs to generate faithful relation paths as plans. This can be achieved by minimizing the KL
divergence with the posterior distribution of faithful relation paths Q(z), which can be approximated
by the valid relation paths in KGs.
4
Published as a conference paper at ICLR 2024
Given a question q and answer a, we could find the path instances wz(eq, ea) = eq
r1
−→e1
r2
−→
. . .
rl
−→ea connecting eq and ea in KGs. The corresponding relation path z = {r1, r2, . . . , rl}
can be considered valid and serve as a faithful plan for answering the question q. The posterior
distribution Q(z) can be formally approximated as
Q(z) ≃Q(z|a, q, G) =



1
|Z|, ∃wz(eq, ea) ∈G,
0, else,
(3)
where we assume a uniform distribution over all valid relation paths Z, and ∃wz(eq, ea) ∈G denote
the existence of a path instance connecting the question eq and answer ea entities in G. Therefore,
the KL divergence can be calculated as
Lplan = DKL(Q(z)∥Pθ(z|q)) = DKL(Q(z|a, q, G)∥Pθ(z|q)) ≃−1
|Z∗|
X
z∈Z∗
log Pθ(z|q),
(4)
where we use the shortest paths Z∗⊆Z between eq and ea in KGs as supervision signals (Zhang
et al., 2022). The detailed derivation of can be found in Appendix A.1. By optimizing the equa-
tion 4, we maximize the probability of LLMs generating faithful relation paths through distilling the
knowledge from KGs.
Retrieval-reasoning optimization. In retrieval-reasoning optimization, we aim to enable LLMs to
conduct reasoning based on the retrieved reasoning paths. For the retrieval-reasoning module, we
follow the FiD framework (Izacard & Grave, 2021; Singh et al., 2021), which allows reasoning on
multiple retrieved reasoning paths2, formulated as
Pθ(a|q, Z, G) =
Y
z∈Z
Pθ(a|q, z, G).
(5)
By approximating the expectation with K sampled plans Z∗
K ⊆Z∗, the objective function of rea-
soning optimization can be written as
Lreason = Ez∼Q(z|a,q,G)[log Pθ(a|q, z, G)]=
X
z∈Z∗
K
log Pθ(a|q, z, G) = log Pθ(a|q, Z∗
K, G).
(6)
This maximizes the probability of LLMs generating correct answers based on the retrieved reasoning
paths.
The final objective function of RoG is the combination of the planning optimization and retrieval-
reasoning optimization, which can be formulated as
L = log Pθ(a|q, Z∗
K, G)
|
{z
}
Retrieval-reasoning
+
1
|Z∗|
X
z∈Z∗
log Pθ(z|q)
|
{z
}
Planning
.
(7)
From equation 7, we can see that we adopt the same LLM for both planning and reasoning, which
are jointly trained on two instruction-tuning tasks, i.e., (planning and retrieval-reasoning). We will
discuss the implementation details of these two tasks in the following subsections.
4.3
PLANNING MODULE
The planning module aims to generate faithful relation paths as plans for answering the question. To
utilize the instruction-following ability of LLMs (Wei et al., 2021), we design a simple instruction
template that prompts LLMs to generate relation paths:
Please generate a valid relation path that can be helpful for answering the following question:
<Question>
where <Question> indicates the question q. The question together with the instruction template
is fed into LLMs to generate the relation paths, which are structurally formatted as a sentence:
2The FiD framework assumes each z ∈Z contributes independently, where the probability Pθ(a|q, Z, G)
can be approximated as the product of each Pθ(a|q, z, G).
5
Published as a conference paper at ICLR 2024
z = <PATH> r1 <SEP> r2 <SEP> . . . <SEP> rl </PATH>
where <PATH>, <SEP>, </PATH> are special tokens indicating the start, separator, and end of the
relation path, respectively3.
Therefore, the optimization of Lplan can be achieved as
arg max
θ
1
|Z∗|
X
z∈Z∗
log Pθ(z|q) =
1
|Z∗|
X
z∈Z∗
log
|z|
Y
i=1
Pθ(ri|r<i, q),
(8)
where Pθ(z|q) denotes the prior distribution of generating faithful relation path z, and Pθ(ri|r<i, q)
denotes the probability of each token in z generated by LLMs.
4.4
RETRIEVAL-REASONING MODULE
Retrieval. Given a question q and a relation path as plan z, the retrieval module aims to retrieve the
reasoning paths wz from KG G. The retrieval process can be conducted by finding paths in G that
start from the question entities eq and follow the relation paths z, formulated as
Wz = {wz(eq, e∗)|wz(eq, e∗) = eq
r1
−→e1
r2
−→. . .
rl
−→ea∗, wz(eq, e∗) ∈G}.
(9)
We adopt a constrained breadth-first search to retrieve the reasoning paths wz from KGs. In exper-
iments, all retrieved paths are used for reasoning. The detailed retrieval algorithm can be found in
Appendix A.3.
Despite we can utilize the retrieved reasoning paths and directly get the answers with majority
voting. The retrieved reasoning paths could be noisy and irrelevant to the questions, which leads to
incorrect answers (He et al., 2021; Zhang et al., 2022). Therefore, we propose a reasoning module
to explore the ability of LLMs to identify the important reasoning paths and answer the questions
based on them.
Reasoning. The reasoning module takes the question q and a set of reasoning paths Wz to generate
answers a. Similarly, we design a reasoning instruction prompt to guide LLMs to conduct reasoning
based on the retrieved reasoning paths Wz. The Wz are also formatted as a series of structural
sentences. The detailed prompt can be found in Appendix A.10.
The optimization of Lreason can be written as
arg max
θ
log Pθ(a|q, Z∗
K, G) = log
X
z∈Z∗
K
X
wz∈Wz
|a|
Y
i=1
Pθ(ti|t<i, q, wz),
(10)
where Pθ(a|q, ZK, G) denotes probability of reasoning the correct answer a based on K relation
paths ZK, and t∗denote the tokens of answers a.
5
EXPERIMENT
In our experiments, we aim to answer the following research questions: RQ1: Can RoG achieve
state-of-the-art performance on the KGQA tasks? RQ2: Can the planning module of RoG be inte-
grated with other LLMs to improve their performance? RQ3: Can RoG conduct faithful reasoning
and generate interpretable reasoning results?
5.1
EXPERIMENT SETTINGS
Datasets. We evaluate the reasoning ability of RoG on two benchmark KGQA datasets: WebQues-
tionSP (WebQSP) (Yih et al., 2016) and Complex WebQuestions (CWQ) (Talmor & Berant, 2018),
which contain up to 4-hop questions. Freebase (Bollacker et al., 2008) is the background knowledge
graph for both datasets, which contains around 88 million entities, 20 thousand relations, and 126
million triples. The details of the datasets are described in Appendix A.4.
3The relation name r∗could be split into multiple tokens. For example, “born in” could be split into
“born” and “ in” by tokenizer. In this way, we could fully utilize the semantic information in relation names
and generalize to different KGs.
6
Published as a conference paper at ICLR 2024
Table 1: Performance comparison with different baselines on the two KGQA datasets.
Type
Methods
WebQSP
CWQ
Hits@1
F1
Hits@1
F1
Embedding
KV-Mem (Miller et al., 2016)
46.7
34.5
18.4
15.7
EmbedKGQA (Saxena et al., 2020)
66.6
-
45.9
-
NSM (He et al., 2021)
68.7
62.8
47.6
42.4
TransferNet (Shi et al., 2021)
71.4
-
48.6
-
KGT5 Saxena et al. (2022)
56.1
-
36.5
-
Retrieval
GraftNet (Sun et al., 2018)
66.4
60.4
36.8
32.7
PullNet (Sun et al., 2019)
68.1
-
45.9
-
SR+NSM (Zhang et al., 2022)
68.9
64.1
50.2
47.1
SR+NSM+E2E (Zhang et al., 2022)
69.5
64.1
49.3
46.3
Semantic Parsing
SPARQL (Sun et al., 2020)
-
-
31.6
-
QGG (Lan & Jiang, 2020)
73.0
73.8
36.9
37.4
ArcaneQA (Gu & Su, 2022)
-
75.3
-
-
RnG-KBQA (Ye et al., 2022)
-
76.2
-
-
LLMs
Flan-T5-xl (Chung et al., 2022)
31.0
-
14.7
-
Alpaca-7B (Taori et al., 2023)
51.8
-
27.4
-
LLaMA2-Chat-7B (Touvron et al., 2023)
64.4
-
34.6
-
ChatGPT
66.8
-
39.9
-
ChatGPT+CoT
75.6
-
48.9
-
LLMs+KGs
KD-CoT (Wang et al., 2023b)
68.6
52.5
55.7
-
UniKGQA (Jiang et al., 2022)
77.2
72.2
51.2
49.1
DECAF (DPR+FiD-3B) (Yu et al., 2022a)
82.1
78.8
-
-
RoG
85.7
70.8
62.6
56.2
Baselines.
We compare RoG with 21 baselines grouping into 5 categories:
1) Embedding-
based methods, 2) Retrieval-augmented methods, 3) Semantic parsing methods, 4) LLMs, and 5)
LLMs+KGs methods. The details of each baseline are described in Appendix A.5.
Evaluation Metrics. Following previous works, we use Hits@1 and F1 as the evaluation metrics.
Hits@1 measures the proportion of questions whose top-1 predicted answer is correct. Since a
question may correspond to multiple answers, F1 considers the coverage of all answers, which
balances the precision and recall of the predicted answers.
Implementations. For RoG, we use LLaMA2-Chat-7B (Touvron et al., 2023) as the LLM backbone,
which is instruction finetuned on the training split of WebQSP and CWQ as well as Freebase for 3
epochs. We generate the top-3 relation paths using beam-search for each question. Since UniKGQA
(Jiang et al., 2022) and DECAF (Yu et al., 2022a) are state-of-the-art methods, we directly refer their
results and those of the other baselines reported in their papers for comparisons. For LLMs, we use
zero-shot prompting to conduct KGQA. The detailed settings are described in Appendix A.6.
5.2
RQ1: KGQA PERFORMANCE COMPARISON
Main Results. In this section, we compare RoG with other baselines on KGQA tasks. The results
are shown in Table 1. Our method achieves the best performance on both datasets across most
metrics. Specifically, compared to the SOTA method DECAF (Yu et al., 2022a) on WebQSP, our
method improves Hits@1 by 4.4%. On the CWQ dataset, which is more challenging due to multi-
hop questions, our method improves both Hits@1 and F1 by 22.3% and 14.4% against the SOTA
model UniKGQA (Jiang et al., 2022). These results demonstrate the superior reasoning ability of
our method in KGQA.
Among other methods, retrieval-augmented approaches outperform conventional embedding-based
methods by retrieving relevant subgraphs from KGs, which reduces reasoning complexity. Fur-
thermore, SR+NSM and SR+NSM+E2E adopt relation paths-based retrieval which achieves better
performance, highlighting the importance of relation paths. Semantic parsing methods perform bet-
ter than retrieval methods on WebQSP but worse on CWQ due to the complexity of generating
logical queries for complex questions in CWQ. Although LLMs-based methods achieve comparable
performance, they are limited by hallucinations and lack of knowledge as shown in Section 5.4. The
LLMs+KGs methods achieve the second-best performance, which demonstrates the effectiveness of
unifying KGs and LLMs for reasoning.
Ablation Study. We conduct an ablation study to analyze the effectiveness of the planning module
and reasoning module in our method (RoG). We compare four variants: 1) w/o planning, where
we remove the planning module and perform reasoning without retrieved reasoning paths; 2) w/o
reasoning, where we remove the reasoning module and use all answers from retrieved reasoning
7
Published as a conference paper at ICLR 2024
1
2
3
4
5
Top-K relation paths
0
10
20
30
40
# Retrieved Reasoning Paths
1
2
3
4
5
Top-K relation paths
0
25
50
75
100
125
# Retrieved Reasoning Paths
30
40
50
60
70
80
Answer Coverage (%)
WebQSP
20
40
60
Answer Coverage (%)
CWQ
# Retrieved Reasoning Paths
Retrieval-Precision
Retrieval-Recall
Reasoning-F1
Figure 3: Faithfulness of top-K generated relation paths. Green bars denote the average number of
retrieved reasoning paths, solid lines denote the answer coverage of the retrieved paths, and dashed
line denote the answer coverage of the reasoning module based on the retrieved reasoning paths.
paths as results; 3) w/ random plans, where we randomly retrieve reasoning paths from KGs and
feed them into the reasoning module; 4) w/ vote reasoning, where we adopt the majority voting to
select top-5 answers from retrieved reasoning paths. The results are shown in Table 2.
Table 2: Ablation studies of RoG.
Method
WebQSP
CWQ
Precision
Recall
F1
Precision
Recall
F1
RoG
74.77
75.84
70.81
57.69
58.19
56.17
RoG w/o planning
57.26
50.16
49.69
35.35
34.77
33.76
RoG w/o reasoning
46.90
79.85
49.56
18.88
67.89
22.26
RoG w/ random plans
38.66
38.31
35.24
38.99
39.29
37.64
RoG w/ vote reasoning
54.80
60.44
47.96
22.92
47.98
26.52
From the results, it is evident that
without a planning module,
our
method degenerates to conventional
LLMs that solely rely on questions
as input, suffering from the lack of
knowledge issue. Although removing
the reasoning module leads to high
recall due to an increased number of
answers, precision drops significantly because of noise in retrieved paths. This demonstrates the ef-
fectiveness of the reasoning module in identifying important reasoning paths and filtering out noise.
Furthermore, using random plans achieves worse performance than removing the planning module,
highlighting the importance of a planning module who generates faithful reasoning plans. Using
a simple majority vote reasoning can improve the results which also demonstrate the necessity of
reasoning module.
5.3
RQ2: PLUG-AND-PLAY ROG PLANNING MODULE
Table 3: Effects of integrating the planning mod-
ule of RoG with different LLMs for reasoning.
Methods
WebQSP
CWQ
Hits@1
Recall
Hits@1
Recall
ChatGPT
66.77
49.27
39.90
35.07
ChatGPT + RoG Planning
81.51
71.60
52.68
48.51
Alpaca-7B
51.78
33.65
27.44
23.62
Alpaca-7B + RoG Planning
56.16
74.20
44.04
38.46
LLaMA2-Chat-7B
64.37
44.61
34.60
29.91
LLaMA2-Chat-7B + RoG Planning
74.20
56.16
56.41
51.99
Flan-T5-xl
30.95
17.08
14.69
12.25
Flan-T5-xl + RoG Planning
67.87
44.93
37.81
32.57
In this section, we evaluate the effectiveness of
integrating the planning module of RoG with
different LLMs during inference to improve
their performance. Specifically, we first adopt
the planning module of RoG to generate re-
lation paths and feed the retrieved reasoning
paths as context into different LLMs for rea-
soning. The results are presented in Table 3. To
account for the fact that it is difficult to extract
the number of answers from LLM’s output. We
only report the Hits@1 and Recall metrics.
From the results, we can notice that the performance of all LLMs is substantially improved by
integrating the planning module of RoG. Specifically, the Hits@1 of ChatGPT, Alpaca, LLaMA2,
and Flan-T5 are improved by 8.5%, 15.3%, and 119.3%, respectively. This demonstrates that the
planning module of RoG can be seamlessly integrated with other LLMs to improve their performance
without retraining.
5.4
RQ3: FAITHFUL REASONING AND INTERPRETABLE RESULTS
Quantitative Results. To evaluate the faithfulness of the relation paths, we illustrate the quantita-
tive results in Figure 3. In experiments, we sweep the number of top-K relation paths generated
by RoG. From results, we can see that the number of retrieved reasoning paths increases with K,
which also leads to more answers to be covered (recall). This demonstrates the faithfulness of the
8
Published as a conference paper at ICLR 2024
Table 4: Example of lack of knowledge for ChatGPT-CoT and interpretable predictions of RoG.
Question
What type of government is used in the country with Northern District?
Answer
Parliamentary system
ChatGPT + CoT
To determine what type of government is used in a country with a Northern District, we
need more specific information. The term ”Northern District” does not specify the name
of any particular country or provide enough context to identify its government type. Could
you please provide additional details or clarify the specific country you are referring to?
Reasoning Paths
Northern District →location.administrative division.first level division of →Israel →
government.form of government.countries →Parliamentary system
RoG
The type of government used in the country with the Northern District is a Parliamentary
system.
Explanations:
1. Northern District →location.administrative division.first level division of →Israel:
The Northern District is a part of Israel.
2. Israel →government.form of government.countries →Parliamentary system: Israel has
a Parliamentary system of government.
So, by following the reasoning paths provided, we can conclude that the country with the
Northern District, which is Israel, uses a Parliamentary system of government.
Table 5: Example of the hallucination of ChatGPT-CoT and faithful reasoning of RoG.
Question
Where is the home stadium of the team who won the 1946 World Series championship?
Answer
Busch Stadium
ChatGPT + CoT
Step 1: Identify the team that won the 1946 World Series championship. The team that
won the 1946 World Series championship was the St. Louis Cardinals.
Step 2: Determine the home stadium of the St. Louis Cardinals during that time. The St.
Louis Cardinals’ home stadium during the 1946 season was Sportsman’s Park.
Therefore, the possible answer to the question ”Where is the home stadium of the team
who won the 1946 World Series championship?” is: - Sportsman’s Park.
Reasoning Paths
1946 World Series →sports.sports team.championships →St.
Louis Cardinals →
sports.sports team.arena stadium →Busch Stadium
1946 World Series →sports.sports team.championships →St.
Louis Cardinals →
sports.sports team.arena stadium →Roger Dean Stadium
RoG
Based on the reasoning paths provided, the home stadium of the team who won the 1946
World Series championship is Busch Stadium. This conclusion is based on the following
path:
1. 1946 World Series →sports.sports team.championships →St. Louis Cardinals →
sports.sports team.arena stadium →Busch Stadium
The reasoning path leads us to the St. Louis Cardinals as the team that won the 1946
World Series, and Busch Stadium is the stadium associated with the St. Louis Cardinals.
Therefore, Busch Stadium is the home stadium of the team that won the 1946 World Series
championship.
relation paths by retrieving the answers. However, more retrieved reasoning paths also lead to more
noise and retrieval time (shown in Appendix A.7.4), which decreases the precision and makes little
contribution to the final results (reasoning-f1). Therefore, we set K = 3 in experiments.
Case studies. We also illustrate two case studies in Table 4 and Table 5. In Table 4, we can
find that ChatGPT+CoT suffers from the lack of knowledge issue and cannot answer the question.
On the contrary, RoG can generate faithful relation paths and retrieve valid reasoning paths from
KGs for reasoning. Besides, RoG can provide interpretable explanations based on the reasoning
paths. In Table 5, we can see that ChatGPT+CoT suffers from hallucinations and generates incorrect
answers. In contrast, although the retrieved reasoning paths contain noises, the reasoning module
can identify the correct reasoning paths and conduct faithful reasoning. These results demonstrate
the effectiveness of RoG in conducting faithful reasoning and generating interpretable results. More
cases can be found in Appendices A.8 and A.9.
6
CONCLUSION
In this paper, we propose a novel method called reasoning on graphs (RoG) that synergizes LLMs
with KGs to conduct faithful and interpretable reasoning. To alleviate the issues of hallucinations
and lack of knowledge, we present a planning-retrieval-reasoning framework, which allows LLMs
to access the latest knowledge while reasoning based on faithful plans on graphs. RoG not only en-
hances the reasoning capability of LLMs by distilling knowledge from KGs through training but also
enables seamless integration with any LLMs during inference. Extensive experiments on two bench-
mark KGQA datasets demonstrate the superiority of RoG in reasoning ability and interpretability.
9
Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This material is based on research partially sponsored by the DARPA Assured Neuro Sym-
bolic Learning and Reasoning (ANSR) program under award number FA8750-23-2-1016 and the
DARPA Knowledge Management at Scale and Speed (KMASS) program under award number
HR00112220047. This research is partly supported by the ARC Future Fellowship FT190100039
to G.H. S. Pan was supported in part by the Australian Research Council (ARC) under grants
FT210100097 and DP240101547, and the CSIRO – National Science Foundation (US) AI Research
Collaboration Program.
ETHICS STATEMENT
Our work is solely dedicated to tackling scientific problems and does not involve any human sub-
jects, animals, or environmentally sensitive materials. Consequently, we do not anticipate any po-
tential ethical risks or conflicts of interest in our study. Our research strives to adhere to the highest
standards of scientific integrity and ethical conduct throughout the entire process, guaranteeing the
validity and reliability of our findings.
REPRODUCIBILITY STATEMENT
Our model has been meticulously formalized within the main body of the text to ensure clarity
and facilitate comprehensive understanding. Additionally, we provide detailed discussions of im-
plementation in Appendices A.4 to A.6, encompassing dataset information, baselines, experimental
settings, and model configurations. The code and pre-trained model weights have been publicized.
REFERENCES
Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia,
Ziwei Ji, Tiezheng Yu, Willy Chung, et al. A multitask, multilingual, multimodal evaluation of
chatgpt on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302.04023, 2023.
Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Lukas Gianinazzi, Joanna
Gajda, Tomasz Lehmann, Michal Podstawski, Hubert Niewiadomski, Piotr Nyczyk, et al.
Graph of thoughts: Solving elaborate problems with large language models.
arXiv preprint
arXiv:2308.09687, 2023.
Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. Freebase: a collab-
oratively created graph database for structuring human knowledge. In Proceedings of the 2008
ACM SIGMOD international conference on Management of data, pp. 1247–1250, 2008.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal,
Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are
few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi
Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language mod-
els. arXiv preprint arXiv:2210.11416, 2022.
Antonia Creswell and Murray Shanahan. Faithful reasoning using large language models. arXiv
preprint arXiv:2208.14271, 2022.
Yu Gu and Yu Su. Arcaneqa: Dynamic program induction and contextualized encoding for knowl-
edge base question answering. In Proceedings of the 29th International Conference on Computa-
tional Linguistics, pp. 1718–1731, 2022.
Gaole He, Yunshi Lan, Jing Jiang, Wayne Xin Zhao, and Ji-Rong Wen. Improving multi-hop knowl-
edge base question answering by learning intermediate supervision signals. In Proceedings of the
14th ACM international conference on web search and data mining, pp. 553–561, 2021.
10
Published as a conference paper at ICLR 2024
Hangfeng He, Hongming Zhang, and Dan Roth. Rethinking with retrieval: Faithful large language
model inference. arXiv preprint arXiv:2301.00303, 2022.
Ruixin Hong, Hongming Zhang, Hong Zhao, Dong Yu, and Changshui Zhang. Faithful question
answering with monte-carlo planning. ACL2023, 2023.
Jie Huang and Kevin Chen-Chuan Chang. Towards reasoning in large language models: A survey.
ACL 2023, 2023.
Gautier Izacard and ´Edouard Grave. Leveraging passage retrieval with generative models for open
domain question answering. In Proceedings of the 16th Conference of the European Chapter of
the Association for Computational Linguistics: Main Volume, pp. 874–880, 2021.
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang,
Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM
Computing Surveys, 55(12):1–38, 2023.
Jinhao Jiang, Kun Zhou, Xin Zhao, and Ji-Rong Wen. Unikgqa: Unified retrieval and reasoning
for solving multi-hop question answering over knowledge graph. In The Eleventh International
Conference on Learning Representations, 2022.
Jinhao Jiang, Kun Zhou, Wayne Xin Zhao, Yaliang Li, and Ji-Rong Wen. Reasoninglm: Enabling
structural subgraph reasoning in pre-trained language models for question answering over knowl-
edge graph. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language
Processing, pp. 3721–3735, 2023.
Jinhao Jiang, Kun Zhou, Wayne Xin Zhao, Yang Song, Chen Zhu, Hengshu Zhu, and Ji-Rong
Wen. Kg-agent: An efficient autonomous agent framework for complex reasoning over knowl-
edge graph. arXiv preprint arXiv:2402.11163, 2024.
Ming Jin, Shiyu Wang, Lintao Ma, Zhixuan Chu, James Y Zhang, Xiaoming Shi, Pin-Yu Chen,
Yuxuan Liang, Yuan-Fang Li, Shirui Pan, and Qingsong Wen. Time-llm: Time series forecasting
by reprogramming large language models. In International Conference on Learning Representa-
tions, 2024a.
Ming Jin, Yifan Zhang, Wei Chen, Kexin Zhang, Yuxuan Liang, Bin Yang, Jindong Wang, Shirui
Pan, and Qingsong Wen. Position paper: What can large language models tell us about time series
analysis. arXiv preprint arXiv:2402.02713, 2024b.
Michael I Jordan, Zoubin Ghahramani, Tommi S Jaakkola, and Lawrence K Saul. An introduction
to variational methods for graphical models. Machine learning, 37:183–233, 1999.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi
Chen, and Wen-tau Yih.
Dense passage retrieval for open-domain question answering.
In
Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing
(EMNLP), pp. 6769–6781, 2020.
Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish
Sabharwal. Decomposed prompting: A modular approach for solving complex tasks. In The
Eleventh International Conference on Learning Representations, 2022.
Yunshi Lan and Jing Jiang. Query graph generation for answering multi-hop complex questions
from knowledge bases. Association for Computational Linguistics, 2020.
Shiyang Li, Yifan Gao, Haoming Jiang, Qingyu Yin, Zheng Li, Xifeng Yan, Chao Zhang, and Bing
Yin. Graph reasoning for question answering with triplet retrieval. In Findings of the Association
for Computational Linguistics: ACL 2023, 2023.
Ke Liang, Lingyuan Meng, Meng Liu, Yue Liu, Wenxuan Tu, Siwei Wang, Sihang Zhou, Xinwang
Liu, and Fuchun Sun. Reasoning over different types of knowledge graphs: Static, temporal and
multi-modal. arXiv preprint arXiv:2212.05767, 2022.
11
Published as a conference paper at ICLR 2024
Ke Liang, Yue Liu, Sihang Zhou, Wenxuan Tu, Yi Wen, Xihong Yang, Xiangjun Dong, and Xin-
wang Liu. Knowledge graph contrastive learning based on relation-symmetrical structure. IEEE
Transactions on Knowledge and Data Engineering, pp. 1–12, 2023. doi: 10.1109/TKDE.2023.
3282989.
Yixin Liu, Kaize Ding, Qinghua Lu, Fuyi Li, Leo Yu Zhang, and Shirui Pan.
Towards self-
interpretable graph-level anomaly detection. Advances in Neural Information Processing Systems,
36, 2023a.
Yixin Liu, Kaize Ding, Jianling Wang, Vincent Lee, Huan Liu, and Shirui Pan. Learning strong
graph neural networks with weak information. In ACM SIGKDD International Conference on
Knowledge Discovery and Data Mining (KDD), 2023b.
Linhao Luo, Jiaxin Ju, Bo Xiong, Yuan-Fang Li, Gholamreza Haffari, and Shirui Pan. Chatrule:
Mining logical rules with large language models for knowledge graph reasoning. arXiv preprint
arXiv:2309.01538, 2023a.
Linhao Luo, Yuan-Fang Li, Gholamreza Haffari, and Shirui Pan. Normalizing flow-based neural
process for few-shot knowledge graph completion. In The 46th International ACM SIGIR Con-
ference on Research and Development in Information Retrieval, 2023b.
Alexander Miller, Adam Fisch, Jesse Dodge, Amir-Hossein Karimi, Antoine Bordes, and Jason
Weston. Key-value memory networks for directly reading documents. In Proceedings of the 2016
Conference on Empirical Methods in Natural Language Processing, pp. 1400–1409, 2016.
Shirui Pan, Yizhen Zheng, and Yixin Liu. Integrating graphs with large language models: Methods
and prospects. arXiv preprint arXiv:2310.05499, 2023.
Shirui Pan, Linhao Luo, Yufei Wang, Chen Chen, Jiapu Wang, and Xindong Wu. Unifying large
language models and knowledge graphs: A roadmap. IEEE Transactions on Knowledge and Data
Engineering (TKDE), 2024.
Apoorv Saxena, Aditay Tripathi, and Partha Talukdar. Improving multi-hop question answering
over knowledge graphs using knowledge base embeddings. In Proceedings of the 58th annual
meeting of the association for computational linguistics, pp. 4498–4507, 2020.
Apoorv Saxena, Adrian Kochsiek, and Rainer Gemulla. Sequence-to-sequence knowledge graph
completion and question answering. In Proceedings of the 60th Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers), pp. 2814–2828, 2022.
Jiaxin Shi, Shulin Cao, Lei Hou, Juanzi Li, and Hanwang Zhang. Transfernet: An effective and
transparent framework for multi-hop question answering over relation graph. In Proceedings of
the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 4149–4158,
2021.
Devendra Singh, Siva Reddy, Will Hamilton, Chris Dyer, and Dani Yogatama. End-to-end training
of multi-document reader and retriever for open-domain question answering. Advances in Neural
Information Processing Systems, 34:25968–25981, 2021.
Haitian Sun, Bhuwan Dhingra, Manzil Zaheer, Kathryn Mazaitis, Ruslan Salakhutdinov, and
William Cohen. Open domain question answering using early fusion of knowledge bases and
text. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Pro-
cessing, pp. 4231–4242, 2018.
Haitian Sun, Tania Bedrax-Weiss, and William Cohen. Pullnet: Open domain question answering
with iterative retrieval on knowledge bases and text. In Proceedings of the 2019 Conference on
Empirical Methods in Natural Language Processing and the 9th International Joint Conference
on Natural Language Processing (EMNLP-IJCNLP), pp. 2380–2390, 2019.
Jiashuo Sun, Chengjin Xu, Lumingyuan Tang, Saizhuo Wang, Chen Lin, Yeyun Gong, Lionel Ni,
Heung-Yeung Shum, and Jian Guo. Think-on-graph: Deep and responsible reasoning of large
language model on knowledge graph. In The Twelfth International Conference on Learning Rep-
resentations, 2024. URL https://openreview.net/forum?id=nnVO1PvbTv.
12
Published as a conference paper at ICLR 2024
Yawei Sun, Lingling Zhang, Gong Cheng, and Yuzhong Qu.
Sparqa: skeleton-based semantic
parsing for complex questions over knowledge bases. In Proceedings of the AAAI conference on
artificial intelligence, volume 34, pp. 8952–8959, 2020.
Oyvind Tafjord, Bhavana Dalvi, and Peter Clark. Entailer: Answering questions with faithful and
truthful chains of reasoning. In Proceedings of the 2022 Conference on Empirical Methods in
Natural Language Processing, pp. 2078–2093, 2022.
Alon Talmor and Jonathan Berant. The web as a knowledge-base for answering complex questions.
In Proceedings of the 2018 Conference of the North American Chapter of the Association for
Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 641–
651, 2018.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy
Liang, and Tatsunori B Hashimoto.
Stanford alpaca: an instruction-following llama model.
https://github.com/tatsu-lab/stanford_alpaca, 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko-
lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda-
tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
Hongwei Wang, Hongyu Ren, and Jure Leskovec. Relational message passing for knowledge graph
completion. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery &
Data Mining, pp. 1697–1707, 2021.
Jiapu Wang, Boyue Wang, Meikang Qiu, Shirui Pan, Bo Xiong, Heng Liu, Linhao Luo, Tengfei Liu,
Yongli Hu, Baocai Yin, et al. A survey on temporal knowledge graph completion: Taxonomy,
progress, and prospects. arXiv preprint arXiv:2308.02457, 2023a.
Keheng Wang, Feiyu Duan, Sirui Wang, Peiguang Li, Yunsen Xian, Chuantao Yin, Wenge Rong,
and Zhang Xiong. Knowledge-driven cot: Exploring faithful reasoning in llms for knowledge-
intensive question answering. arXiv preprint arXiv:2308.13259, 2023b.
Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim.
Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language
models. arXiv preprint arXiv:2305.04091, 2023c.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H Chi, Sharan Narang, Aakanksha
Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language
models. In The Eleventh International Conference on Learning Representations, 2022.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An-
drew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In International
Conference on Learning Representations, 2021.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny
Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in
Neural Information Processing Systems, 35:24824–24837, 2022.
Xiaohan Xu, Peng Zhang, Yongquan He, Chengpeng Chao, and Chaoyang Yan. Subgraph neigh-
boring relations infomax for inductive link prediction on knowledge graphs. In Proceedings of
the Thirty-First International Joint Conference on Artificial Intelligence, 2022.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan
Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International
Conference on Learning Representations, 2022.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik
Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. arXiv
preprint arXiv:2305.10601, 2023.
13
Published as a conference paper at ICLR 2024
Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, and Jure Leskovec. Qa-gnn:
Reasoning with language models and knowledge graphs for question answering. In Proceedings
of the 2021 Conference of the North American Chapter of the Association for Computational
Linguistics: Human Language Technologies, pp. 535–546, 2021.
Xi Ye, Semih Yavuz, Kazuma Hashimoto, Yingbo Zhou, and Caiming Xiong. Rng-kbqa: Generation
augmented iterative ranking for knowledge base question answering. In Proceedings of the 60th
Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.
6032–6043, 2022.
Wen-tau Yih, Matthew Richardson, Christopher Meek, Ming-Wei Chang, and Jina Suh. The value
of semantic parse labeling for knowledge base question answering. In Proceedings of the 54th
Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp.
201–206, 2016.
Donghan Yu, Sheng Zhang, Patrick Ng, Henghui Zhu, Alexander Hanbo Li, Jun Wang, Yiqun Hu,
William Yang Wang, Zhiguo Wang, and Bing Xiang. Decaf: Joint decoding of answers and logical
forms for question answering over knowledge bases. In The Eleventh International Conference
on Learning Representations, 2022a.
Donghan Yu, Chenguang Zhu, Yuwei Fang, Wenhao Yu, Shuohang Wang, Yichong Xu, Xiang Ren,
Yiming Yang, and Michael Zeng. Kg-fid: Infusing knowledge graph in fusion-in-decoder for
open-domain question answering. In Proceedings of the 60th Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers), pp. 4961–4974, 2022b.
Jing Zhang, Xiaokang Zhang, Jifan Yu, Jian Tang, Jie Tang, Cuiping Li, and Hong Chen. Subgraph
retrieval enhanced model for multi-hop knowledge base question answering. In Proceedings of the
60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),
pp. 5773–5784, 2022.
Xikun Zhang, Antoine Bosselut, Michihiro Yasunaga, Hongyu Ren, Percy Liang, Christopher D
Manning, and Jure Leskovec. Greaselm: Graph reasoning enhanced language models. In Inter-
national conference on learning representations, 2021.
Yuyu Zhang, Hanjun Dai, Zornitsa Kozareva, Alexander Smola, and Le Song. Variational reasoning
for question answering with knowledge graph. In Proceedings of the AAAI conference on artificial
intelligence, volume 32, 2018.
Zicheng Zhao, Linhao Luo, Shirui Pan, Quoc Viet Hung Nguyen, and Chen Gong. Towards few-
shot inductive link prediction on knowledge graphs: A relational anonymous walk-guided neural
process approach. In Joint European Conference on Machine Learning and Knowledge Discovery
in Databases, pp. 515–532, 2023.
Xin Zheng, Miao Zhang, Chunyang Chen, Chaojie Li, Chuan Zhou, and Shirui Pan. Multi-relational
graph neural architecture search with fine-grained message passing. In 2022 IEEE International
Conference on Data Mining (ICDM), pp. 783–792. IEEE, 2022a.
Xin Zheng, Yixin Liu, Zhifeng Bao, Meng Fang, Xia Hu, Alan Wee-Chung Liew, and Shirui
Pan.
Towards data-centric graph machine learning: Review and outlook.
arXiv preprint
arXiv:2309.10979, 2023a.
Xin Zheng, Miao Zhang, Chunyang Chen, Soheila Molaei, Chuan Zhou, and Shirui Pan. Gnneval-
uator: Evaluating gnn performance on unseen graphs without labels.
In Advances in Neural
Information Processing Systems (NeurIPS), 2023b.
Xin Zheng, Miao Zhang, Chunyang Chen, Quoc Viet Hung Nguyen, Xingquan Zhu, and Shirui
Pan. Structure-free graph condensation: From large-scale graphs to condensed graph-free data.
In Advances in Neural Information Processing Systems (NeurIPS), 2023c.
Yizhen Zheng, Shirui Pan, Vincent Lee, Yu Zheng, and Philip S Yu. Rethinking and scaling up
graph contrastive learning: An extremely efficient approach with group discrimination. Advances
in Neural Information Processing Systems, 35:10809–10820, 2022b.
14
Published as a conference paper at ICLR 2024
Yizhen Zheng, Huan Yee Koh, Jiaxin Ju, Anh TN Nguyen, Lauren T May, Geoffrey I Webb, and
Shirui Pan. Large language models for scientific synthesis, inference and explanation. arXiv
preprint arXiv:2310.07984, 2023d.
Yizhen Zheng, He Zhang, Vincent Lee, Yu Zheng, Xiao Wang, and Shirui Pan. Finding the missing-
half: Graph complementary learning for homophily-prone and heterophily-prone graphs. Inter-
national Conference on Machine Learning, 2023e.
A
APPENDIX
A.1
DETAILED DERIVATION OF THE PLANNING MODULE
In this section, we provide a detailed derivation of the planning module. By approximating the Q(z)
with Q(z|a, q, G), the KL divergence can be calculated as
Lplan = DKL(Q(z)∥Pθ(z|q)) = DKL(Q(z|a, q, G)∥Pθ(z|q)),
= Ez∼Q(z|a,q,G)[log Q(z|a, q, G) −log Pθ(z|q)],
= −Ez∼Q(z|a,q,G) log Pθ(z|q) + CONST,
(11)
where the expectations cannot be computed exactly because of the large number of valid relation
paths Z, so we approximate it by using the shortest paths z ∈Z∗⊂Z between eq and ea in KGs
(Zhang et al., 2022). This can be formulated as
Lplan = −
X
z∈Z∗
Q(z|a, q, G) log Pθ(z|q) + CONST.
(12)
Based on equation 3, by assuming a uniform distribution over the set of shortest paths Z∗, we can
rewrite the equation 12 as
Lplan = −1
|Z∗|
X
z∈Z∗
log Pθ(z|q) + CONST,
(13)
where the CONST is omitted in the final optimization since it makes no contributions to the loss.
A.2
DETAILED RELATED WORK
A.2.1
LLM REASONING PROMPT
Many studies have been proposed to harness the reasoning ability of LLMs to handle complex tasks
through prompting (Wei et al., 2022; Wang et al., 2022; Yao et al., 2023; Besta et al., 2023; Pan et al.,
2023; Zheng et al., 2023d; Jin et al., 2024a;b). Chain-of-Thought (Wei et al., 2022) enables LLMs
to generate a reasoning chain that could be helpful to reasoning. Tree of thoughts (Yao et al., 2023)
expands the reasoning chain to a tree structure to explore more reasoning paths. Graph of thoughts
(Besta et al., 2023) further models the reasoning chain as a graph with an aggregation operation to
synergize the reasoning paths. Plan-and-solve (Wang et al., 2023c) prompts LLMs to generate a plan
and execute based on it. DecomP (He et al., 2021) prompts LLMs to decompose the reasoning task
into a series of sub-tasks and solve them step by step. However, the problem of hallucinations and
lack of knowledge affect the faithfulness of the reasoning. ReACT (Yao et al., 2022) treats LLMs
as agents, which interact with the environment to get the latest knowledge for reasoning. To explore
faithful reasoning, Entailer (Tafjord et al., 2022) introduces a verifier to validate the reasoning steps
generated by LLMs. Creswell & Shanahan (2022) present a framework including two LLMs that are
used for selecting and generating reasoning steps, respectively. FAME (Hong et al., 2023) introduces
the Monte-Carlo planning to generate faithful reasoning steps. RR (He et al., 2022) and KD-CoT
Wang et al. (2023b) aim to retrieve relevant knowledge from KGs to produce faithful reasoning
plans for LLMs. Think-on-Graph (Sun et al., 2024) and KG-Agent (Jiang et al., 2024) treat LLMs
as agents to interact with KGs via prompting to get the latest knowledge for reasoning.
15
Published as a conference paper at ICLR 2024
A.2.2
KNOWLEDGE GRAPH QUESTION ANSWERING
Knowledge graphs contain abundant factual knowledge in a structured format, which has attracted
great attention from researchers (Zheng et al., 2022a; 2023a; Pan et al., 2023; Liang et al., 2023).
Knowledge graph reasoning aims to derive new insights based on the existing graph structure and
facts (Liang et al., 2022; Wang et al., 2023a; Luo et al., 2023b; Zhao et al., 2023). Graph neural
networks that effectively capture the structure information have also been widely used in KG rea-
soning Zheng et al. (2022b; 2023e;b;c); Liu et al. (2023b;a). As a typical reasoning task, knowledge
graph question answering (KGQA) aims to obtain answers based on knowledge from KGs, which
can be generally divided into three categories: 1) embedding-based methods, 2) retrieval-augmented
methods, and 3) semantic parsing methods.
Embedding-based methods model the entities and relations in embedding space and design special
model architectures to reason answers. KV-Mem (Miller et al., 2016) adopts a Key-Value memory
network to store triples for reasoning. EmbedKGQA (Saxena et al., 2020) and NSM (He et al.,
2021) utilize the sequential model to mimic the multi-hop reasoning process. QA-GNN (Yasunaga
et al., 2021) and Greaselm (Zhang et al., 2021) further adopt the graph neural network to capture the
graph structure for reasoning. However, these methods need to design different model architectures,
which are not flexible and generalizable.
Retrieval-augmented methods aims to retrieve the relative facts from the KGs to improve the
reasoning performance. Early works adopt the page rank or random walk algorithm to retrieve
subgraphs from KGs for reasoning (Sun et al., 2018; 2019). However, they ignore the semantic
information in questions and lead to noisy retrieval results. Zhang et al. (2022) proposes a relation
paths-based subgraph retrieval, resulting a better retrieval and QA performance. Other lines of
studies retrieving triples from KGs via BM25 (Li et al., 2023) or DPR (Karpukhin et al., 2020; Yu
et al., 2022b) to improve the performance of LLMs. They discard the structure information in KGs
which leads to suboptimal results. Recently, UniKGQA (Jiang et al., 2022) unifies the graph retrieval
and reasoning process into a single model with LLMs, which achieves state-of-the-art performance
on KGQA tasks.
Semantic parsing methods parse the question into a structural query (e.g., SPARQL) which can be
executed by a query engine to get answers (Sun et al., 2020; Lan & Jiang, 2020). ArcaneQA (Gu &
Su, 2022) dynamically generates the query based on results from previous steps. RnG-KBQA (Ye
et al., 2022) first enumerate all possible queries and then rank them to get the final output. These
methods heavily rely on the quality of generated queries. If the query is not executable, no answers
will be generated. DECAF (Yu et al., 2022a) combines semantic parsing and LLMs reasoning to
jointly generate answers, which also reach salient performance on KGQA tasks.
A.3
RETRIEVAL ALGORITHM
Given a question q and a relation path as plan z, we adopt a constrained breadth-first search to
retrieve the reasoning paths. The pseudocode code is presented in Algorithm 1.
We first initialize a queue of current reasoning paths Q with the entities in the question Tq (line 3-5).
Then, we iteratively expand each reasoning path in Q by adding the triples that are connected to
the entities in the queue following the relation in relation path (line 11-19). The reasoning path is
expanded until the length is equal to the length of the relation path. The expanded reasoning path is
added to the set Wz as the final results (line 8-10).
A.4
DATASETS
We adopt two benchmark KGQA datasets: WebQuestionSP (WebQSP)4 (Yih et al., 2016) and Com-
plex WebQuestions (CWQ)5 (Talmor & Berant, 2018) in this work. We follow previous works (Sun
et al., 2018; Jiang et al., 2022) to use the same train and test splits for fair comparison. The statistic
of the datasets are given in Table 6. The statistics of the answer numbers and reasoning hops are
presented in Table 7 and Table 8, respectively.
4https://www.microsoft.com/en-us/download/details.aspx?id=52763
5https://www.tau-nlp.sites.tau.ac.il/compwebq
16
Published as a conference paper at ICLR 2024
Algorithm 1: Retrieve reasoning paths based on relation paths
Input: Question q, relation path z = {r1, r2, . . . , rl}, KG G.
Output: Reasoning paths Wz.
1 Wz ←∅;
2 Q ←Queue();
3 foreach eq ∈Tq do
4
Q.append((eq, [])); // Initialize queue with question entities.
5 end
6 while Q ̸= ∅do
7
(s, wz) ←Q.pop();
8
if len(wz) = len(z) then
9
Wz.append(wz);
10
end
11
if len(wz) < len(z) then
12
r ←z[len(wz) + 1]; // Get relation for next step.
13
foreach (s, r′, t) ∈G do
14
if r′ = r then
15
w′
z.append((s, r, t)); // Expand the reasoning path.
16
Q.append((t, w′
z));
17
end
18
end
19
end
20 end
21 return Wz.;
Table 6: Statistics of datasets.
Datasets
#Train
#Test
Max #hop
WebQSP
2,826
1,628
2
CWQ
27,639
3,531
4
Table 7: Statistics of the number of answers for questions in WebQSP and CWQ.
Dataset
#Ans = 1
2 ≥#Ans ≤4
5 ≥#Ans ≤9
#Ans ≥10
WebQSP
51.2%
27.4%
8.3%
12.1%
CWQ
70.6%
19.4%
6%
4%
Table 8: Statistics of the question hops in WebQSP and CWQ.
Dataset
1 hop
2 hop
≥3 hop
WebQSP
65.49 %
34.51%
0.00%
CWQ
40.91 %
38.34%
20.75%
Table 9: Statistics of MetaQA-3hop datasets.
Datasets
#Train
#Test
#hop
MetaQA-3hop
1,000
1,4274
3
17
Published as a conference paper at ICLR 2024
Table 10: Statistics of constructed knowledge graphs.
KG
#Entities
#Relations
#Triples
Freebase
2,566,291
7,058
8,309,195
Wiki-Movie
43,234
9
133,582
Table 11: The statistics of the instances in the instruction-tuning datasets.
Planing Data
Retrieval-reasoning Data
Interpretability Data
216,006
30,465
2,000
To evaluate the transferability of RoG to other KGs. We further select the MetaQA-3hop dataset
(Zhang et al., 2018) which is based on Wiki-Movies KGs6. We select 1000 samples from the training
split. The statistic of the dataset is presented in Table 9.
Both WebQSP and CWQ can be reasoned based on Freebase KGs7 (Bollacker et al., 2008). To
reduce the size of KGs, following previous works (He et al., 2021; Jiang et al., 2022), we construct
a subgraph of Freebase by extracting all triples that contain within the max reasoning hops of ques-
tion entities in WebQSP and CWQ. Similarly, we construct a subgraph of Wiki-Movies KGs for
MetaQA-3hop. The statistics of constructed KGs are presented in Table 10.
We construct the instruction-tuning dataset using the training split of WebQSP and CWQ datasets.
For planning optimization, we generate data by extracting the shortest paths that connect the ques-
tions and answers, which serve as supervisory signals. In the retrieval-reasoning optimization phase,
we feed both extracted shortest paths together with the questions to predict the answers. To enhance
interpretability, we randomly select 1000 samples from each dataset along with their reasoning paths
and answers. These are then input into ChatGPT to produce interpretive responses, which help em-
power our method in generating results with good explainability. The statistics of final training
datasets are shown in Table 11.
A.5
BASELINES
We compare RoG with 21 baselines grouping into 5 categories: 1) Embedding-based methods, 2)
Retrieval-augmented methods, 3) Semantic parsing methods, 4) LLMs, and 5) LLMs+KGs methods.
The details of each baseline are described as follows.
Embedding-based methods.
• KV-Mem (Miller et al., 2016) adopts a Key-Value memory network to store triples and
perform multi-hop reasoning by iterative operating on the memory.
• EmbedKGQA (Saxena et al., 2020) models the reasoning on KGs as a sequential link pre-
diction problem by using the embedding of entities and questions.
• NSM (He et al., 2021) utilizes the sequential model to mimic the multi-hop reasoning
process.
• TransferNet (Shi et al., 2021) adopts a graph neural network to capture the relevance be-
tween entities and questions for reasoning.
• KGT5 (Saxena et al., 2022) finetunes a sequence-to-sequence framework on KGs and gen-
erates answers based on the input question.
Retrieval-augmented methods.
• GraftNet (Sun et al., 2018) retrieves relevant subgraphs from KGs with entity linking.
• PullNet (Sun et al., 2019) trains a retrieval model composed of a LSTM and a graph neural
network to retrieve a question-specific subgraph.
6https://research.fb.com/downloads/babi
7https://github.com/microsoft/FastRDFStore
18
Published as a conference paper at ICLR 2024
• SR+NSM (Zhang et al., 2022) proposes a relation-path retrieval to retrieve subgraphs for
multi-hop reasoning.
• SR+NSM+E2E (Zhang et al., 2022) further adopts an end-to-end training strategy to jointly
train the retrieval and reasoning modules of SR+NSM.
Semantic parsing methods.
• SPARQL (Sun et al., 2020) presents a novel skeleton grammar to represent the high-level
structure of a complex question with language modes.
• QGG (Lan & Jiang, 2020) generates a query graph for a question by simultaneously adding
constraints and extending relation paths.
• ArcaneQA (Gu & Su, 2022) dynamically generates the query based on results from previ-
ous steps.
• RnG-KBQA (Ye et al., 2022) first enumerates all possible queries and then ranks them to
get the final output.
Large language models (LLMs).
• Flan-T5 (Chung et al., 2022) is an enhanced version of T5 models that is instruction fine-
tuned on mixture of tasks.
• Alpaca (Taori et al., 2023) is based on LLaMA and finetuned on an instruction-following
dataset.
• LLaMA2-Chat (Touvron et al., 2023) is a large language model that is optimized for dia-
logue purposes.
• ChatGPT8 is a powerful closed-source LLM that could follow instructions to conduct com-
plex tasks9.
• ChatGPT+CoT (Wei et al., 2022) uses the Chain-of-thought prompt to improve the reason
ability of ChatGPT.
LLMs+KGs methods.
• KD-CoT Wang et al. (2023b) retrieves relevant knowledge from KGs to generate faithful
reasoning plans for LLMs.
• UniKGQA (Jiang et al., 2022) unifies the graph retrieval and reasoning process into a single
model with LLMs, which achieves state-of-the-art performance on KGQA tasks.
• DECAF (Yu et al., 2022a) combines semantic parsing and LLMs reasoning to jointly gen-
erate answers, which also reach salient performance on KGQA tasks.
A.6
IMPLEMENTATION SETTINGS
For RoG, we use LLaMA2-Chat-7B (Touvron et al., 2023) as the LLM backbone, which is instruc-
tion finetuned on the training split of WebQSP and CWQ as well as Freebase for 3 epochs. The
batch size is set to 4 and the learning rate is set to 2e-5. We use the cosine learning rate scheduler
policy with the warmup ratio set to 0.03. The training is conducted on 2 A100-80G GPUs for 38
hours. During inference, we first adopt the LLM to generate top-K relation paths with the highest
probability as the plans. Then, we adopt the Algorithm 1 to retrieve reasoning paths, which are fed
into the LLM to reason the final answers.
For LLM beelines, we use zero-shot prompting to conduct KGQA, which directly asks LLMs to
answer the question. For other baselines, we directly copy their results reported in UniKGQA (Jiang
et al., 2022) and DECAF (Yu et al., 2022a) for comparisons.
For combining the planning module of RoG with different LLMs, we use the planning module to
generate plans (relation paths), which are executed on KGs to retrieve the reasoning paths. The
retrieved paths are fed into different LLMs during inference by utilizing the reasoning prompts
template shown in Appendix A.10.
8https://openai.com/blog/chatgpt
9Experiments are conducted with the ChatGPT model released between July. to Sept., 2023.
19
Published as a conference paper at ICLR 2024
Table 12: Performance of RoG on MetaQA-3hop.
Strategies
MetaQA-3hop
Hits@1
F1
RoG (train from scratch)
84.81
41.32
RoG (transfer from Freebase)
88.98
50.68
Table 13: Training time comparison.
Method
Training on Freebase
Transferring to Wiki-Movies
RoG
38 hours
2 hours
A.7
ADDITIONAL EXPERIMENT RESULTS
A.7.1
TRANSFERABILITY TO OTHER KGS
We evaluate the transferability of RoG to other KGs. We select the MetaQA-3hop dataset (Zhang
et al., 2018) which is based on Wiki-Movies KGs. We select 1000 samples from the training split
and utilize two training strategies to finetune RoG: 1) training from scratch, where we directly train
RoG from LLaMA2-Chat with 1000 samples; 2) transfer from Freebase, where we conduct a further
finetuning based on RoG trained for Freebase. The results are shown in Table 12. From results, we
can see that transfer from Freebase achieves better performance than training from scratch, which
demonstrates the transferability of RoG to other KGs.
We also compare the training time on Freebase and transferring to Wiki-Movies KGs. From results
shown in Table 13, we can see that the training time on Freebase is 38 hours, while the transferring
time is only 2 hours. This demonstrates the efficiency of transferring RoG to other KGs.
A.7.2
PERFORMANCE WITH DIFFERENT TRAINING DATA
In our experiment, we finetune RoG jointly on the training set of both WebQSP and CWQ datasets to
maximize the ability of RoG for reasoning on Freebase. To fairly compare with other methods (e.g.,
UniKGQA (Jiang et al., 2022)) that are only trained on single dataset, we provide additional results
of the performance of RoG trained on single dataset. From the results shown in Tables 14 and 15,
we can see that RoG trained on single dataset still outperforms the STOA baselines (UniKGQA).
Besides, we can also find that jointly training RoG on multiple datasets can further improve the
performance. In the future, we will try to generate more QA datasets from Freebase to further
improved the reasoning ability of RoG.
A.7.3
PERFORMANCE COMPARISON WITH DIFFERENT FINETUNED LLMS
In Table 1, we report the zero-shot performance of different LLMs. However, RoG is finetuned on
the training split of the QA dataset. To make a fair comparison, we further report the performance
of the LLMs finetuned on the training split of the QA dataset in Table 16. From the results, we can
see that RoG still outperforms the finetuned LLMs.
Table 14: Performance on WebQSP with different training data.
Method
Training Data
Hits@1
F1
UniKGQA
WebQSP
77.2
72,2
RoG
WebQSP
81.5
61.8
RoG
WebQSP+CWQ
85.7
70.8
20
Published as a conference paper at ICLR 2024
Table 15: Performance on CWQ with different training data.
Method
Training Data
Hits@1
F1
UniKGQA
CWQ
51.2
49.1
RoG
CWQ
59.1
52.9
RoG
WebQSP+CWQ
62.6
56.2
Table 16: Performance comparison with different finetuned LLMs (Hits@1).
Method
WebQSP
CWQ
Alpaca-7B (Zero Shot)
51.78
27.44
LLaMA2-Chat-7B (Zero Shot)
64.37
34.60
Alpaca-7B (Finetuned)
74.57
55.98
LLaMA2-Chat-7B (Finetuned)
73.89
53.49
RoG
85.75
62.65
A.7.4
RETRIEVAL COSTS
We present the retrieval time and number of retrieved reasoning paths in Figure 4. From results,
we can see that the retrieval time increases with the number of top-K relation paths. Therefore, we
should select a proper K to balance the retrieval time and the number of retrieved reasoning paths.
In experiments, we set K = 3.
A.7.5
PERFORMANCE ON DIFFERENT HOPS
We present the performance of RoG and its variants on different hops of questions in Table 17.
From results, we can see that RoG achieves better performance than its variants on different hops of
questions, especially questions with more than 3 hops. This demonstrates the importance of relation
paths for improving the reasoning performance of LLMs on complex questions.
A.7.6
PERFORMANCE ON DIFFERENT ANSWER NUMBERS
We also present the performance of RoG and its variants on questions with different numbers of
answers in Table 18. From results, we can see that RoG achieves better performance than its variants
on questions with different numbers of answers. Specifically, with the number of answer increasing,
the performance of RoG w/o planning decreases significantly due to the lack of knowledge from
1
2
3
4
5
Top-K relation paths
0
10
20
30
40
# Retrieved Reasoning Paths
1
2
3
4
5
Top-K relation paths
0
25
50
75
100
125
# Retrieved Reasoning Paths
0.5
1.0
1.5
Retrieval Time (s)
1e−3
WebQSP
1.0
1.5
2.0
2.5
Retrieval Time (s)
1e−3
CWQ
# Retrieved Reasoning Paths
Retrieval Time
Figure 4: Average retrieval time and average number of retrieved reasoning paths w.r.t. the number
of top-K relation paths.
21
Published as a conference paper at ICLR 2024
Table 17: F1 scores of RoG and its variants for different hops of questions.
Methods
WebQSP
CWQ
1 hop
2 hop
≥3 hop
1 hop
2 hop
≥3 hop
RoG
77.03
64.86
-
62.88
58.46
37.82
RoG w/o reasoning
57.06
25.49
-
17.06
34.25
17.07
RoG w/o planning
50.33
51.66
-
31.04
33.93
23.29
Table 18: F1 scores of RoG and its variants for questions with varying numbers of answers.
Methods
WebQSP
CWQ
#Ans = 1
2 ≥#Ans ≤4
5 ≥#Ans ≤9
#Ans ≥10
#Ans = 1
2 ≥#Ans ≤4
5 ≥#Ans ≤9
#Ans ≥10
RoG
67.89
79.39
75.04
58.33
56.90
53.73
58.36
43.62
RoG w/o reasoning
33.49
52.80
58.05
66.01
16.61
27.06
40.10
34.45
RoG w/o planning
55.03
51.08
44.81
27.00
34.08
34.16
31.67
25.21
KGs. Although, RoG w/o reasoning can retrieve more answers to improve the performance. It still
is inferior to RoG due to the lack of reasoning ability to remove the noise.
A.8
CASE STUDIES: RELATION PATHS
We illustrate several examples of relation paths generated by RoG in Table 19.
A.9
CASE STUDIES: INTERPRETABLE RESULTS
We illustrate several examples of interpretable reasoning results generated by RoG in Table 20.
A.10
PROMPTS
Planning module aims to generate faithful relation paths as plans for answering the question. The
instruction template is presented as follows:
Planning Prompt Template
Please generate a valid relation path that can be helpful for answering the following question:
<Question>
where <Question> indicates the question.
The reasoning module takes the question q and a set of reasoning paths Wz to generate answers a.
The instruction template is presented as follows:
Reasoning Prompt Template
Based on the reasoning paths, please answer the given question. Please keep the answer as simple as
possible and return all the possible answers as a list.
Reasoning Paths:
<Reasoning Paths>
Question:
<Question>
where <Reasoning Paths> denotes the retrieved reasoning paths Wz which are formatted as a
series of structural sentences:
e0 →r1 →e1 →· · · →rl →el
. . .
e0 →r1 →e1 →· · · →rl →el.
22
Published as a conference paper at ICLR 2024
To exploit the explanation ability of RoG, we design a new instruction template for the reasoning
module to generate interpretable results. The instruction template is presented as follows:
Explanation Prompt Template
Based on the reasoning paths, please answer the given question and explain why.
Here are some examples:
<Examples>
Reasoning Paths:
<Reasoning Paths>
Question:
<Question>
where the Examples denotes a few-shot human-annotated examples to demonstrate the explana-
tion process.
Table 19: Examples of the generated relation paths.
Question
Top-3 Relation Paths
what does jamaican people speak?
z1 : location.country.languages spoken
z2 : language.human language.countries spoken in
z3 : location.country.official language
where is jamarcus russell from?
z1 : location.location.people born here
z2 : people.person.place of birth
z3 : sports.sports league draft pick.player →sports.sports league draft pick.location
where did edgar allan poe died?
z1 : people.deceased person.place of death
z2 : people.cause of death.people
z3 : people.person.place of birth
what highschool did harper lee go to?
z1 : people.person.education →education.educational institution.students graduates
z2 : education.education.student →education.educational institution.students graduates
z3 : people.person.education →education.education.institutio
what are the songs that justin bieber wrote?
z1 : music.recording.artist
z2 : music.composition.composer
z3 : music.composer.compositions
what are the religions practiced in indonesia?
z1 : people.person.nationality →people.person.religion
z2 : location.statistical region.religions →location.religion percentage.religion
z3 : location.country.languages spoken →religion.religion.languages
Lou Seal is the mascot for the team that last
won the World Series when?
z1 : sports.mascot.team →sports.sports championship event.champion
z2 : sports.mascot.team →sports.sports team.championships
z3 : sports.sports championship event.championship
What type of government is used in the coun-
try with Northern District?
z1 : location.administrative division.first level division of →government.form of government.countries
z2 : location.administrative division.first level division of →location.country.form of government
z3 : administrative division.first level division of →government.form of government.countries
The people from the country that contains
Nord-Ouest Department speak what lan-
guages today?
z1 : location.administrative division.first level division of →language.human language.countries spoken in
z2 : location.administrative division.first level division of →location.country.languages spoken
z3 : base.aareas.schema.administrative area.administrative parent →location.country.languages spoken
What stadium does the team with mascot
named Hank play at?
z1 : sports.mascot.team →sports.sports facility.teams
z2 : sports.sports team.team mascot →sports.sports facility.teams
z3 : sports.mascot.team →sports.sports team.arena stadium
Which popular sports team in Spain, that won
the 2014 Eurocup Finals championship?
z1 : sports.sports team.championships →sports.sports team location.teams
z2 : sports.sports team.location →sports.sports team location.teams
z2 : sports.sports team.location →sports.sports team location.teams
z3 : sports.sports team location.teams
What educational institution with the mascot
named Washington University Bear did Ten-
nessee Williams go to?
z1 : education.educational institution.mascot
z2 : people.person.nationality →location.location.containedby
z3 : education.education.student →education.education.institution
Who is the current head coach of the NFL
squad owned by the Rooney family?
z1 : sports.professional sports team.owner s →american football.football coach.current team head coached
z2 : sports.professional sports team.owner s →american football.football team.current head coach
z3 : american football.football coach.current team head coached
What is the home field of the sports team
whose mascot is named Fredbird?
z1 : sports.mascot.team →sports.sports facility.teams
z2 : sports.sports team.team mascot →sports.sports facility.teams
z3 : sports.mascot.team →sports.sports team.arena stadium
23
Published as a conference paper at ICLR 2024
Table 20: Examples of faithful and interpretable reasoning of RoG.
Question
Lou Seal is the mascot for the team that last won the World Series when?
Answer
2014
Reasoning Paths
Lou
Seal
→
sports.mascot.team
→
San
Francisco
Giants
→
sports.sports championship event.champion →2014 World Series
RoG
Based on the provided reasoning paths, we can determine when the team for which Lou
Seal is the mascot last won the World Series:
1. Lou Seal is associated with the San Francisco Giants.
2. The San Francisco Giants are associated with winning the 2014 World Series.
Therefore, the San Francisco Giants, the team for which Lou Seal is the mascot, last won
the World Series in 2014.
Question
what is my timezone in louisiana?
Answer
Central Time Zone
Reasoning Paths
Louisiana →location.location.partially containedby →Tangipahoa River →geogra-
phy.river.origin →Osyka →location.location.time zones →Central Time Zone
Louisiana
→
location.location.containedby
→
Alexandria
→
loca-
tion.location.containedby →United States of America →location.location.time zones →
Central Time Zone
RoG
Based on the reasoning paths provided, it can be determined that Louisiana is in the Central
Time Zone.
Here’s the explanation:
1. The first reasoning path tells us that Louisiana is partially contained by the Tangipahoa
River, which is a geographical feature. However, it doesn’t directly mention the time zone.
2. The second reasoning path establishes that Louisiana is contained by Alexandria, which
is in turn contained by the United States of America. The United States of America is in
the Central Time Zone.
Therefore, based on these reasoning paths, the time zone in Louisiana is the Central Time
Zone.
Question
Which child of Walt Disney died from lung cancer?
Answer
Sharon Mae Disney
Reasoning Paths
Lung cancer →people.deceased person.cause of death →Sharon Mae Disney
Lung cancer →people.deceased person.cause of death →Walt Disney
Walt Disney →people.deceased person.cause of death →Lung cancer
Walt Disney →people.person.children →Diane Disney Miller
Walt Disney →people.cause of death.people →Circulatory collapse
Walt Disney →people.person.parents →Flora Call Disney
Walt Disney →people.person.parents →Elias Disney
Walt Disney →people.person.parents →Sharon Mae Disney
RoG
Based on the reasoning paths provided, it appears that Sharon Mae Disney is the child of
Walt Disney who died from lung cancer.
The reasoning path ”Walt Disney →people.deceased person.cause of death →Lung can-
cer” directly connects Walt Disney to lung cancer as a cause of death, and since Sharon
Mae Disney is also linked to lung cancer as a cause of death through the path ”Lung can-
cer →people.deceased person.cause of death →Sharon Mae Disney,” it can be concluded
that Sharon Mae Disney is the child of Walt Disney who died from lung cancer.
24