Question:
Which of the following are key techniques used in the proposed model to improve knowledge KG-to-text generation?
A. A Planning Scorer to select the most relevant linearized sequence from all possible triple orders.
B. A graph structure-aware module that only considers first-order neighbors to capture KG topology.
C. Relative Distance Encoding, which is incorporated as a bias to better capture the intricate topology of the KG, including indirectly connected nodes.
D. Contrastive learning to help the model distinguish between similar input KGs and similar entities within the same KG.

Answer: ACD

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 8693–8703
December 6-10, 2023 ©2023 Association for Computational Linguistics
Structure-aware Knowledge Graph-to-Text Generation with Planning
Selection and Similarity Distinction
Feng Zhao, Hongzhi Zou, Cheng Yan
Natural Language Processing and Knowledge Graph Lab,
School of Computer Science and Technology,
Huazhong University of Science and Technology, Wuhan, China
{zhaof, hongzhizou, yancheng}@hust.edu.cn
Abstract
The knowledge graph-to-text (KG-to-text)
generation task aims to synthesize coherent
and engaging sentences that accurately con-
vey the complex information derived from
an input knowledge graph.
One of the pri-
mary challenges in this task is bridging the
gap between the diverse structures of the KG
and the target text, while preserving the de-
tails of the input KG. To address this, we
propose a novel approach that efﬁciently in-
tegrates graph structure-aware modules with
pre-trained language models. Unlike conven-
tional techniques, which only consider direct
connections between ﬁrst-order neighbors, our
method delves deeper by incorporating Rela-
tive Distance Encoding as a bias within the
graph structure-aware module. This enables
our model to better capture the intricate topol-
ogy information present in the KG. To fur-
ther elevate the ﬁdelity of the generated text,
Planning Selection and Similarity Distinction
are introduced. Our approach ﬁlters the most
relevant linearized sequences by employing
a planning scorer, while simultaneously dis-
tinguishing similar input KGs through con-
trastive learning techniques. Experiments on
two datasets demonstrate the superiority of our
model.
1
Introduction
Knowledge graph (KG) is a structured data repre-
sentation form that contains rich knowledge infor-
mation and is more convenient for processes such
as information retrieval and reasoning. Although
KGs facilitate computational processes, it is difﬁ-
cult for humans to intuitively understand the con-
tent in KGs, so the proposed KG-to-text genera-
tion task aims to produce correct descriptive text
for the input KG. KG-to-text has various applica-
tions, like question-and-answer (Pal et al., 2019)
and dialogue systems (Zhou et al., 2018). Figure 1
shows a KG and its corresponding text. The main
Adare
Manor
1862
James
Pain
1700
completion date
Description Text:
James Pain was the architect of the Adare Manor building
which construction started in 1700 and completed in 1862 .
Linearized Sequence:
<H>Adare Manor <R> architect [T] James Pain
<H>Adare Manor <R> building start date [T] 1700
<H>Adare Manor <R> completion date [T] 1862
Figure 1: The KG and its corresponding text. The enti-
ties in the text are marked in color. The order of triples
in the linearized sequence can control the accuracy of
the generated text.
challenges for the task are the different struc-
tures of the input KG and output texts as well as
the need to ensure generated description texts can
completely and accurately cover the KG informa-
tion, including all entities and relations.
Beneﬁtting from the powerful capabilities of
pre-trained language models (PLMs), methods
based on PLMs (Ke et al., 2021) are currently
achieving outstanding results on KG-to-text gener-
ation tasks. Since there is a structural gap between
the graph structure of KGs and the sequence in-
put requirements demanded by PLMs, an impor-
tant step is to linearize the given KG into an in-
put sequence. The results of many previous works
have shown that the ﬁnal generated texts are signif-
icantly inﬂuenced by the triple orders of linearized
sequences. Many existing works typically use pre-
deﬁned heuristic rules such as breadth-ﬁrst search
(BFS) (Li et al., 2021), or create a content plan-
ner using multistep prediction methods to predict
of the triple order (Zhao et al., 2020). Each step of
these multistep prediction processes may have pre-
diction bias, which results in the ﬁnal sequence or-
8693
der easily deviating signiﬁcantly from the ground-
truth order. Considering this problem, we train an
efﬁcient planning scorer to score the generated re-
sults of all linearized sequences corresponding to
KG, and pick the sequence with the highest score
to be the ﬁnal input, thus enabling the bias caused
by multistep predictions to be avoided.
The linearized sequence may ignore the graph
structure information in the corresponding KG,
such as the connectivity of nodes and the direc-
tion of edges. Some works (Colas et al., 2022;
Ke et al., 2021) have improved PLMs by adding
graph structure-aware modules to every layer of
the transformer encoder. These graph-aware mod-
ules only consider the directly connected entities
in one triple, i.e., the ﬁrst-order neighbors in the
KG. During the speciﬁc text generation process,
only the information within each triple is consid-
ered, while the connections of the entities between
triples are ignored. Therefore, to fully utilize the
topological structure information of a KG, we con-
sider the edges in the KG as nodes to compute
the shortest path distance among nodes and add
it to the weight computation as a bias. Addition-
ally, to enable better distinction of similar input
KGs and similar entities in the same KG, we use
contrastive learning to enable the transformer en-
coder to differentiate the input sequences at a ﬁner-
grained level.
We use two datasets to test our
model: WebNLG and DART, and it outperforms
the previous models. Our contributions are three-
fold:
• We propose an efﬁcient score control-based
planning scorer to generate remarkably accurate
linearized sequences for text generation.
• Our approach involves capturing and distin-
guishing the information of a KG using ﬁne-
grained control methods,
including relative
distance-based structure-aware control and con-
trast learning-based sequence representation dis-
tinction.
• Results from two datasets demonstrate the su-
periority of our model compared to other meth-
ods, and our approach has the potential to signif-
icantly improve the quality of text generation.
2
Related Work
2.1
KG-to-Text Generation
To capture the KG structural information, many re-
cent works on KG-to-text generation encode the
graph structure directly using graph neural net-
works (GNNs) (Guo et al., 2019; Zhao et al.,
2020; Ribeiro et al., 2020b; Li et al., 2021)
or graph-transformers (Schmitt et al., 2020) and
then decode into texts. DUALENC (Zhao et al.,
2020) feeds the input KG into two GNN en-
coders for order planning and sentence gener-
ation.
Graformer (Schmitt et al., 2020) intro-
duces a model that combines relative position in-
formation to compute self-attention.
Other ap-
proaches (Wang et al., 2021b; Liu et al., 2022; Guo
et al., 2020; Ribeiro et al., 2020a) ﬁrst linearize
KG into sequences and then feed them into the
sequence-to-sequence (Seq2Seq) model for gener-
ating desired texts. In this paper, we employ the
Seq2Seq model with a planning selector to control
linearized sequence orders.
2.2
Sequence Order Generation
Existing works (Zhao et al., 2020) have shown
that the linearized order of the given triples has
an effect on the generated text’s quality.
Previ-
ous works mainly use graph traversal (Li et al.,
2021) or multistep prediction (Su et al., 2021;
Liu et al., 2022; Zhao et al., 2020) methods for
triple order generation. (Li et al., 2021) uses the
relation-biased BFS (RBFS) strategy to traverse
and linearize KGs into sequences.
(Zhao et al.,
2020) uses the content planner to select one of the
remaining unvisited triples at each step until all
triples have been visited. These graph traversal-
based approaches are difﬁcult to handle when fac-
ing isolated points that are not connected to other
points. Methods based on multistep prediction can
be incorrect at each step of the chain decisions.
Inspired by previous work (Kertkeidkachorn and
Takamura, 2020), we use a PLM based efﬁcient
planning scorer to control the order generation pro-
cess.
2.3
Pre-trained Language Model
PLMs such as T5 (Raffel et al., 2020) and
BART (Lewis et al., 2020) have achieved superior
performance on KG-to-text generation tasks. Ex-
isting works for solving this task using PLMs fall
into two main categories. One category does not
change the internal structures of PLMs, but rather
optimizes the ﬁne-tuning process by introducing
other training tasks.
(Li et al., 2021) utilizes the
KG reconstruction task and the entity copying task
to ﬁne-tune the BART. The other category adjusts
the internal structures of PLMs by incorporating
8694
new modules in encoder or decoder.
(Ke et al.,
2021; Colas et al., 2022) incorporate the graph
structure-aware module into the encoder of PLMs.
However, they only consider information of di-
rectly connected ﬁrst-order neighbors and ignore
the connection information of indirectly connected
neighbors. Inspired by (Schmitt et al., 2020), we
aggregate the relative distance information into the
graph structure-aware module, thus exploiting the
information of indirectly connected neighbors.
2.4
Sequence Embedding with Contrastive
Learning
Due to the power of contrastive learning exhib-
ited in image representation (Radford et al., 2021),
many approaches (Gao et al., 2021; Wang et al.,
2021a; Reimers and Gurevych, 2019) have ap-
plied it to sentence embedding as well. Sentence-
BERT (Reimers and Gurevych, 2019) presents
a modiﬁcation of BERT, which employs the
Siamese network to derive ﬁxed-sized vectors for
input sentences and employs a similarity mea-
sure to ﬁnd semantically similar sentences. Sim-
CSE (Gao et al., 2021) presents the unsupervised
methods that apply different hidden dropout masks
and the supervised methods that use additional de-
scription datasets to generate positives and nega-
tives. Similar to the above sentence embedding
approaches, in the KG-to-text generation task we
embed linearized sequences of triples using con-
trastive learning to distinguish similar KGs and
similar entities.
3
Approaches
Our method not only ensures that the generated
text and the target text have the same information,
but also enables the model to capture both topolog-
ical information and speciﬁc node content infor-
mation of the graph for the KG-to-text generation
task. To accomplish the ﬁrst goal, we train a plan-
ning scorer that can select the best linearized se-
quence by scoring all possible input KGs. For the
second goal, we ﬁrst add the graph structure-aware
module to the BART encoder. Then, we calculate
the cosine similarity of the sequence embeddings
from the last layer of the encoder to generate a new
loss. Figure 2 represents our model architecture
and the following sections describe the contents
of these modules.
3.1
Problem Formulation
The aim is to generate accurate text to describe the
input KG. The input KG consists of some triples
and G = {< h, r, t > |h, t ∈E, r ∈R}, where
E and R are sets of entities and relations, respec-
tively. Following (Ke et al., 2021), we linearize
the input KG as Glinear = (w1, w2, · · · , wm),
where m is the number of tokens. The target is to
generate the text T = (t1, t2, · · · , tn), which give
an accurate and complete description of the infor-
mation in the input KG.
3.2
Planning Scorer
The purpose of the planning scorer is to compute
the scores of all linearized KG sequences to select
the highest scoring sequence. Previous methods
incorporated GCN-based representation of graphs,
but this process does not mainly affect the lin-
earization order of the triples and instead severely
affects the training efﬁciency in the ﬁrst stage. We
therefore propose a planning scorer with a simple
structure to further speed up the training efﬁciency
of the model while obtaining the correct lineariza-
tion sequences.
The planning scorer is also di-
vided into two steps: 1) Plan Generation and 2)
Plan Evaluation.
3.2.1
Plan Generation
We generate all triple orders as needed by Plan
generation. Each graph in the WebNLG has no
more than seven triples, so we can generate all pos-
sible triple orders by exhaustive enumeration. Fig-
ure 3 shows a KG with three triples and the six cor-
responding kinds of linearized sequences, where
<H>, <R>, and <T> identify the head entity, rela-
tion, and tail entity. For KGs with the number of
triples more than seven in the DART dataset, we
then randomly sampled only 5,000 possible triplet
orders as the result of the planning generation.
3.2.2
Plan Evaluation
Planning evaluation performs evaluations for each
generated linearized sequence. The structure of
the Plan Generation consists of two modules: Lin-
earization Representation and Plan Score.
Linearization Representation maps each to-
ken in the input linearized sequence into a high-
dimensional vector.
As shown in Figure 2, we
employ BART to achieve the linealization rep-
resentation of the input sequence.
Getting the
plan p, we ﬁrst obtain the linearized sequence
L = (w1, w2, · · · , wn), in which w1, w2, · · · , wn
8695
Text generation
2）Text generation
1）Planning Selection
Constractive loss
James Pain was the architect of the Adare Manor building
which construction started in 1700 and completed in 1862 .
<H>Adare Manor <R> architect [tail] James Pain
<H>Adare Manor <R> building start date [tail] 1700
<H>Adare Manor <R> completion date [tail] 17862
BART
Average Pool
Linearized Sequence
<H>Adare Manor <R> architect [tail] James Pain
<H>Adare Manor <R> building start date [tail] 1700
<H>Adare Manor <R> completion date [tail] 17862
FNN
Reference
James Pain was the architect of the
Adare Manor building which
construction started in 1700 and
completed in 1862 .
BART
bleu
Plan
Socre
loss
Planning Scorer
Decoder
Graph-aware
Encoder
Linearized Sequence:
<H>Adare Manor <R> architect <T> James Pain
<H>Adare Manor <R> building start date <T> 1700
<H>Adare Manor <R> completion date <T> 17862
Linearized Sequence:
<H>Adare Manor <R> architect <T> James Pain
<H>Adare Manor <R> building start date <T> 1700
<H>Adare Manor <R> completion date <T> 17862
Reference:
James Pain was the architect of the
Adare Manor building which
construction started in 1700 and
completed in 1862.
James Pain was the architect of the Adare Manor building
which construction started in 1700 and completed in 1862.
Figure 2: Overview of our model. During the planning selection phase, the planning scorer ﬁrst generates a
linearized sequence of KGs and then computes the resulting score. The BLEU values of the generated texts
obtained using references as inputs are used as the labels to train the planning scorer. During the text generation
phase, the highest scoring linearized sequences screened by the planning scorer are sequentially fed into the graph
structure-aware encoder and the decoder to ﬁnish the text generation process.
Adare
Manor
James
Pain
1862
1700
completion date
Linearized Sequence1:
<H>Adare Manor <R> architect [T] James Pain
<H>Adare Manor <R> completion date [T] 1862
<H>Adare Manor <R> building start date [T] 1700
Linearized Sequence2:
<H>Adare Manor <R> architect [T] James Pain
<H>Adare Manor <R> building start date [T] 1700
<H>Adare Manor <R> completion date [T] 1862
Linearized Sequence3:
[<H>Adare Manor <R> completion date <T> 1862
<H>Adare Manor <R> architect <T> James Pain
<H>Adare Manor <R> building start date <T> 1700
Linearized Sequence4:
<H>Adare Manor <R> completion date <T> 1862
<H>Adare Manor <R> building start date <T> 1700
<H>Adare Manor <R> architect <T> James Pain
Linearized Sequence5:
<H>Adare Manor <R> building start date <T> 1700
<H>Adare Manor <R> completion date <T> 1862
<H>Adare Manor <R> architect <T> James Pain
Linearized Sequence6:
<H>Adare Manor <R> building start date <T> 1700
<H>Adare Manor <R> architect <T> James Pain
<H>Adare Manor <R> completion date <T> 1862
Figure 3: A KG with three relations and all of its corre-
sponding linearized sequences.
represent each token.
Then we feed the lin-
earized sequence L into BART and take the em-
bedding of the decoder’s last hidden layer X
= (x1, x2, · · · , xn) as the representation result.
Finally the Linearization Representation C is
learned using a mean pooling layer by the follow-
ing equation:
C = 1
n
n
∑
i=0
xi,
(1)
where the linearized sequence has n tokens and xi
denotes the embedding of the i-th token.
Plan Score is to estimate the BLEU score based
on the Linearization Representation. Here we feed
the Linearization Representation C to a feedfor-
ward neural network (FNN), and the output is the
score:
by = CWF NN + b,
(2)
where WFNN is a linear matrix and b is the bias.
During the training phase, the linearized se-
quences are fed into BART to generate new texts.
And the generated texts and the target texts are
used to calculate the BLEU metrics as the target
scores y. The optimized loss function is as follows:
Lplan =
∑
s∈S
∑
p∈P (s)
|ys −bys,p|,
(3)
where S denotes the corpus and for sample s in
the corpus, P(s) denotes the set of generated order
plans.
During the testing phase, all possible planning
sequences for each sample KG in the dataset
are fed into the planning scorer to compute the
scores, and the linearized sequences with the high-
est scores are kept in the ﬁle for subsequent gener-
ation tasks.
3.3
Graph Structure-Aware Model
The linearized sequences are used as the inputs for
text generation. We add the graph structure-aware
module to each layer of the transformer encoder,
and also use the relative distance as a bias when
computing the self-attention weights.
Global Attention.
Each layer in the trans-
former encoder starts with a Self-Attention mod-
ule, which captures the global relationships be-
tween all tokens of linearized sequences. Assum-
ing that the input linearized sequence is X, the
8696
self-attention is computed as follows:
αi = σ
(XiW Q(XW K)⊤
√
d
)
,
(4)
Vi =
n
∑
j=1
αij(XjW V ),
(5)
where W V , W Q and W K are learned parameters.
d represents the dimension of word vectors. σ()
denotes the softmax function.
Graph Aware Attention. Global attention cap-
tures the global relations between all tokens, but
ignores the connectivity among entities and rela-
tions in the KG. To further capture the connectiv-
ity between entities and relations in the KG, in-
cluding those not directly connected, we add the
structure-aware module that incorporates relative
distances behind each Global Attention layer. First
the mean pooling is used to generate the represen-
tation Xp ∈Rm×d of every component from the
output V ∈Rn×d of Global Attention:
Xp = pooling(V ),
(6)
where m denotes the sum of the numbers of en-
tities and relations.
n is the number of tokens
from the linearized sequence. Then we compute
the representation Xg
l of each entity and relation
by Graph Structure-Aware Attention:
αg
i =σ
(Xp
i W QS(XpW KS+qiW KC)⊤
√
d
+γ(R)
)
, (7)
e
Xg =
m
∑
j=1
αg
ij(Xp
i W V S + qiW V C),
(8)
where W QS, W KS, W KC, W V S and W V C are
the weight matrices to be learned. q ∈Rm×m de-
notes the adjacency matrix that records the connec-
tivity among entities and relations, and the value q
is either 1 or 0. γ(R) ∈Rm×m denotes the relative
distance encoding matrix.
Following
(Schmitt
et al., 2020), we consider the relations of KG as
graph’s nodes and compute the path distance be-
tween each pair of nodes to generate the relative
distance matrix R:
Rij =









∞,
if δ(ni, nj) = ∞
and δ(nj, ni) = ∞
−δ(nj, ni),
if δ(ni, nj) > δ(nj, ni)
δ(ni, nj),
if δ(ni, nj) ≤δ(nj, ni)
,
(9)
where δ(ni, nj) denotes the shortest length from
ni to nj. Finally, the graph structure-aware rep-
resentation e
Xg is added to the self-attentive repre-
sentation V after a residual layer:
eV = gather( e
Xg) + V,
(10)
where gather() redistributes the m-dimensional
node representation e
Xg onto the n-dimensional
token representation.
Finally, after the encoder-
decoder, for the input linearized sequence Glinear,
the text generation loss is formulated as follows:
LT = −
n
∑
i=1
logP(wj|w1, · · · , wj −1; Glinear),
(11)
where P is the predicted probability of each token.
3.4
Similarity Distinction
For cases of similar input KGs or similar entities
in one KG, the generated texts are often confused
and erroneous. Therefore we reduce the similar-
ity of different sequence representations by con-
trastive learning. First we use different sequences
in the same batch as simple negative samples for
the differentiation of similar input KGs and each
linearized KG as its own positive sample. Taking
the embeddings H from the encoder’s last hidden
layer as inputs, we ﬁrst obtain the representation
ri for each sequence i using a mean pooling layer.
We assume that all sequences within a batch are
simple negative samples of each other and then
compute the contrastive loss by cosine similarity
as follows:
LC = −
∑
i∈P
log
(
esim(ri,ri)/τ
∑
j∈J esim(ri,rj)/τ
)
,
(12)
where P denotes the corpus and J denotes the set
of training samples in the batch. To distinguish
entities in the same KG, we generate some new
sequences as hard negative samples for each se-
quence in a batch. Speciﬁcally, we randomly re-
place each entity in the linearized sequence by
other entities with the 15% probability to obtain
hard negative samples. All simple negative sam-
ples in the same batch and the generated hard neg-
ative samples will be used together in the compu-
tation of the contrastive loss in Eq. (12).
At last, the total loss Ltotal consists of the text
generation loss LT and the contrastive learning
loss LC:
Ltotal = LT + λLC,
(13)
where λ is the combination coefﬁcient.
4
Experiments
4.1
Datasets
WebNLG (Gardent et al., 2017) is a frequently
used dataset.
A sample in the dataset contains
8697
Table 1: Results on WebNLG. #Param represents the number of training parameters for models with PLMs. Pre-
task represents if models have been pre-trained on additional tasks. ‡ and ♯indicate that the results are reprinted
from (Ke et al., 2021) and (Colas et al., 2022). SOTA-NPT indicates the state-of-the-art (SOTA) model without
any pre-training.
Model
#Param
Pre-task
BLEU
ROUGE
METEOR
SOTA-NPT♯(Shimorina and Gardent, 2018)
-
No
61.00
71.00
42.00
BART-base♯(Ribeiro et al., 2020a)
140M
No
64.55
75.13
46.51
T5-base♯(Ribeiro et al., 2020a)
220M
No
64.42
74.77
46.58
KGPT♯(Chen et al., 2020)
177M
Yes
64.11
74.57
46.30
JointGT (BART)♯(Ke et al., 2021)
160M
Yes
65.92
76.10
47.15
GAP‡ - Me,r + γ (Colas et al., 2022)
153M
No
66.20
76.36
46.77
Ours
161M
No
66.53
76.54
47.39
one to seven triples corresponding to one to ﬁve
texts. We randomly select one text at a time from
one sample as the ﬁnal target text and replace the
underscores with spaces. Following
(Ke et al.,
2021), we use the version 2.0 and the numbers of
KG-text pairs in the training/validation/testing are
34,352/4,316/4,224.
DART (Nan et al., 2021) is an open-domain
dataset with KGs extracted from Wikipedia tables
or incorporated from other datasets like Cleaned
E2E (Novikova et al., 2017).
The numbers of
KG-text pairs in the training/validation/testing are
30,348/2,759/5,097.
4.2
Implementation Details
For the Planning Scorer, we employ BART-base
for Linearization Representation. In the FNN, we
utilize 2 hidden layers and the ReLU activation.
The Planning Scorer is trained with the learning
rate of 0.001, the batch size of 16, the epoch of 10
and Adam (Kingma and Ba, 2015) to optimize the
parameters.
For text generation, the hyperparameters are set
as follows: batch size: 100, epoch: 60, learning
rate: 2e-5 and optimizer: Adam. The τ in the
contrastive learning is 0.05. The maximum input
length of Linearized sequences is 256. The total
loss is optimized by Eq. 13 and λ is set to 0.5.
When generating text in the inference time, the
beam search size is 5.
Following previous work, the evaluation met-
rics for experiments on WebNLG are ME-
TEOR (Banerjee and Lavie, 2005), ROUGE-
L (Lin, 2004) and BLEU-4 (Papineni et al., 2002).
For DART, besides BLEU and METEOR we use
two additional metrics, MoverScore (Zhao et al.,
2019) and BLEURT (Sellam et al., 2020). With
the BLEU metrics produced on the validation sets
Table 2:
Results on DART. Seven other models are
compared with our model. ‡ indicates that the results
are reprinted from (Liu et al., 2022).
Model
BLEU
METEOR
MoverScore
BLEURT
Seq2Seq-Att
29.66
0.27
0.31
-0.13
T5-base
49.21
0.40
0.53
0.43
T5-large
50.66
0.40
0.54
0.44
BART-base
47.11
0.38
0.51
0.37
BART-large
48.56
0.39
0.52
0.41
JointGT‡
54.24
0.44
0.64
0.59
S-OSC‡
62.01
0.43
0.64
0.49
Ours
62.67
0.44
0.65
0.49
of both datasets, we select the best performing
model.
4.3
Main Results
The WebNLG results displayed in Table 1 rep-
resent that our model outperforms other models
in overall performance including three metrics,
i.e., BLEU, METEOR, and ROUGE. Regarding
the BLEU metric on WebNLG, our model outper-
forms the SOTA Seq2Seq model without any pre-
training by 5.53%. Compared with BART and T5,
our models improve by 1.98% and 2.11%, respec-
tively. Both JointGT and KGPT utilize additional
pre-training tasks such as graph-text alignment
and text or graph reconstruction. The BLEU im-
provements of 2.42% over KGPT and 0.61% over
JointGT show that our model performs well with-
out additional pre-training tasks. Compared with
GAP, which considers only directly connected
ﬁrst-order neighbors in the graph structure-aware
module, the improvement of 0.33% on BLEU and
0.62% on METEOR shows that our model better
captures the input graph’s topological structure.
The results on DART are displayed in Table 2.
The large gaps between "Seq2Seq-Att" (Nan et al.,
2021) and the other models on all metrics demon-
8698
Table 3:
Ablation study for different modules on
WebNLG.
Model
BLEU
METEOR
ROUGE
full model
66.53
47.39
76.54
w/o PS
65.86
47.20
76.27
w/o RDE
65.82
47.13
76.25
w/o DK
66.17
47.41
76.29
strate the power of PLMs for KG-to-text genera-
tion tasks. Compared to the method using only
PLMs (e.g. T5-large), our model has an improve-
ment of 12.01% on BLEU, 0.04% on METEOR,
0.11% on MoverScore and 0.05% on BLEURT.
The 8.43% improvement achieved in the BLEU
metric also shows that our model makes more efﬁ-
cient use of the graph structure. We observe im-
provements of 0.66% on BLEU, 0.01% on ME-
TEOR and 0.01% on MoverScore, over S-OSC. To
summarize, our model generates more ﬂuent and
accurate texts than the other models.
5
Analysis
5.1
Ablation Study
In additional experiments on WebNLG, we evalu-
ated the impact of the Planning Scorer, relative dis-
tance encoding in the graph structure-aware mod-
ule and the distinction of KGs on the performance
of our model. Besides the full model, we also con-
struct the following three variations: without Plan-
ning Scorers (w/o PS) and just relying on orders
of the input triples, without relative distance en-
coding in the graph structure-aware module (w/o
RDE) and without distinction of KGs (w/o DK).
The results in Table 3 demonstrate that all three
modules impact the performance of our model on
BLEU, METEOR and ROUGE metrics.
By re-
moving the planning scorer (w/o PS) and the rel-
ative distance coding in the graph structure-aware
module (w/o RDE), the BLEU metrics drop by
0.67% and 0.71%, respectively, which reﬂects the
importance of the linearized orders and the infor-
mation abtained from nodes that are not directly
connected. Removing the distinction of KGs (w/o
DK), the small reductions of BLEU and ROUGE
metrics indicate that contrastive learning can im-
prove the accuracy of generated texts to some ex-
tent.
5.2
Linearized Order of Triples
To conﬁrm our planning scorer’s effectiveness, we
compare it with the RBFS multistep prediction ap-
Table 4: Results of different methods. PS denotes Plan-
ning Scorer. RBFS and RDFS (Li et al., 2021) denote
the relation-biased BFS and the relation-biased DFS,
respectively. RS denotes the random sorting order.
Methods
BLEU
METEOR
ROUGE
PS
66.53
47.39
76.54
RBFS
66.30
47.28
76.34
RDFS
65.78
47.04
76.02
RS
58.22
46.17
73.09
Table 5:
BLEU scores for the different models on
DART with different graph sizes.
Model
#Triples
1-3
4-6
≥7
JointGT
75.46
56.14
47.36
w/o PS
73.37
57.50
50.10
full model
73.98
60.44
54.30
proach and the random sorting order on WebNLG.
Table 4 displays the results of different approaches
to generate orders of triples. The large drop-off be-
tween the random sorting order (RS) and the other
methods on the three metrics suggests that lin-
earized orders of triples signiﬁcantly impact the ef-
fectiveness of generated texts. The planning scorer
outperforms RBFS and RDFS in three metrics in-
dicating the advantage of evaluating the linearized
order of triples from a holistic perspective.
5.3
KG Size
To verify the effectiveness of the graph structure-
aware module with relative distance encoding, we
classify DART into "1-3", "4-6" and "≥7" ac-
cording to the number of triples and compare the
BLEU metrics for generating texts. Table 5 dis-
plays the BLEU metric results of JointGT, our
model without the Planning Scorer (w/o PS), and
our full model under different conditions. From
the top two results, we can observe that in the "1-
3" case, JointGT outperforms our model without
the Planning Scorer (w/o PS) by 2.09%, which
indicates that for small KG inputs, relative dis-
tance coding may present redundant information
leading to poor results. With the increase of the
triples’ number, the results of JointGT are gradu-
ally overtaken by "w/o PS", which indicates that
the relative distance coding can obtain more infor-
mation about the graph structures of larger graphs
with more indirectly connected nodes. The bottom
two results show that the role of Planning scorer
becomes more important as the number of input
triples increases.
8699
Table 6: Examples of the generated texts and linearized KGs produced by JointGT and our model on WebNLG
and DART, where bold represents entities in KGs.
Real
Knowledge
Graph
Andrew White
(musician)
2003
active years
start year
Universal Records
(defunct record label)
record label
Kaiser Chiefs
Marry Banilow
associated musical
artist
①
②
③
⑤
④
associated musical
artist
Lee County, Alabama
Auburn, Alabama
IS_PART_OF
Washington, D.C.
ETHNIC_GROUP
African Americans
United States
COUNTRY
①
②
③
⑤
④
STATE
Alabama
CAPITAL
⑥
Reference
Musician Andrew White began in 2003
and is associated with artists Marry
Banilow and the Kaiser Chiefs . He was
signed to the defunct Universal records .
The United States, where the capital is
Washington DC, includes the ethnic group
of African Americans. Auburn is located
in the country and is part of Lee County
in Alabama.
JointGT
Linearized
KG
2⃝→1⃝→4⃝→5⃝→3⃝
1⃝→2⃝→3⃝→5⃝→4⃝→6⃝
Generated
Text
The musician Andrew White started his
career in 2003 and is associated with the
musical artist Marry Banilow . He was
once signed to the defunct Records label
but now plays for Kaiser Chiefs .
Auburn, Alabama is in Lee County,
Alabama which is in Alabama in the United
States where African Americans are an ethnic
group and the capital is Washington DC.
Ours
Linearized
KG
2⃝→1⃝→4⃝→3⃝→5⃝
5⃝→6⃝→4⃝→1⃝→2⃝→3⃝
Generated
Text
Andrew White started his career in 2003
and is associated with the musical artist
Marry Banilow and Kaiser Chiefs. He
was once signed to the now defunct
Universal Records .
The United States where the capital is
Washington DC, has the ethnic group of
African Americans. Auburn is part of
Lee County which is in Alabama of the
United States.
89
90
91
92
93
94
95
96
50
100
150
200
Entity Coverage
Batch Size
without hard samples
with hard samples
Figure 4: Entity coverage without hard negative sam-
ples and with hard negative samples under different
batch sizes.
5.4
Entity Coverage
We explore the effect of contrastive learning on dif-
ferentiating similar KGs and similar entities by cal-
culating the entity coverage in the generated texts
over the entities of input KGs. Figure 4 reports the
variation curves of entity coverage with and with-
out hard negative samples under different batch
sizes. It can be observed that the entity coverage
increases with increasing batch size. The hard neg-
ative samples are helpful to improve the entity cov-
erage, but the effect becomes smaller as the batch
size increases. We set the batch size to 100 be-
cause the entity coverage changes very little after
the batch size reaches 100.
5.5
Qualitative Analysis
Table 6 shows examples of generated texts from
our model and JointGT on WebNLG and DART.
From the WebNLG example we can see that
JointGT reverses the position of node 3 and node
5, resulting in incorrect inference results of "once"
and "now" in the generated text.
With the cor-
rect linearization order, our model makes an effec-
tive distinction between node 3 and node 4, which
have the same relation. On DART, the long en-
tity chains in the input KG lead to many attributive
clauses and unclear expressions in the text gener-
ated by JointGT. The linearized order generated by
our model is not exactly correct, but under the in-
ﬂuence of the graph structure-aware module with
relative distance encoding and entity distinction,
the generated text is able to correctly and concisely
describe the input KG.
6
Conclusion
The paper presents a two-step training model for
KG-to-text generation including planning scorer
and text generation.
In the ﬁrst step the plan-
ning scorer ﬁrst obtains the linearized KG order
with the highest score. Then the second step uses
a graph structure-aware module with relative dis-
tance encoding and KG distinction by contrastive
learning to ensure that the model captures the topo-
logical information and speciﬁc entity features in
the input KG. Additional experiments and qualita-
tive analyses indicate our model outperforms the
existing KG-to-text approaches on two datasets.
8700
Limitations
Since the Planning Scorer needs to score all possi-
ble linearized sequences of the input KG, the pro-
cessing becomes less efﬁcient as the number of the
input triples increases. Therefore our model is not
suitable for handling long text or paragraph gener-
ation tasks. Because of the large amount of param-
eters in PLMs, our model also consumes a large
amount of GPU resources when performing com-
parative learning. Four Tesla V100 GPUs are used
to train our model, and the maximum batch size
that can be set is 312. Finally, our model can accu-
rately describe entities and relations in the KG, but
lacks reasoning power. For example, the triples
(A,husband,B) and (B,mother,C) as inputs, it can-
not generate "A is the father of C". These above
limitations are also the direction of our continuing
research in the future.
Acknowledgements
This work was supported in part by National Nat-
ural Science Foundation of China under Grants
No.62072203.
References
Satanjeev Banerjee and Alon Lavie. 2005. METEOR:
An automatic metric for MT evaluation with im-
proved correlation with human judgments. In Pro-
ceedings of the Workshop on Intrinsic and Ex-
trinsic Evaluation Measures for Machine Transla-
tion and/or Summarization@ACL 2005, Ann Arbor,
Michigan, USA, June 29, 2005, pages 65–72.
Wenhu Chen, Yu Su, Xifeng Yan, and William Yang
Wang. 2020.
KGPT: Knowledge-grounded pre-
training for data-to-text generation. In Proceedings
of the 2020 Conference on Empirical Methods in
Natural Language Processing, EMNLP 2020, On-
line, November 16-20, 2020, pages 8635–8648.
Anthony Colas, Mehrdad Alvandipour, and Daisy Zhe
Wang. 2022.
GAP: A Graph-aware Language
Model Framework for Knowledge Graph-to-Text
Generation.
In Proceedings of the 29th Inter-
national Conference on Computational Linguistics,
COLING 2022, Gyeongju, Republic of Korea, Octo-
ber 12-17, 2022, pages 5755–5769.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021.
SimCSE: Simple contrastive learning of sentence
embeddings. In Proceedings of the 2021 Conference
on Empirical Methods in Natural Language Pro-
cessing, EMNLP 2021, Virtual Event / Punta Cana,
Dominican Republic, 7-11 November, 2021, pages
6894–6910.
Claire Gardent, Anastasia Shimorina, Shashi Narayan,
and Laura Perez-Beltrachini. 2017. The WebNLG
challenge: Generating text from RDF data. In Pro-
ceedings of the 10th International Conference on
Natural Language Generation, INLG 2017, Santi-
ago de Compostela, Spain, September 4-7, 2017,
pages 124–133.
Qipeng Guo, Zhijing Jin, Ning Dai, Xipeng Qiu, Xi-
angyang Xue, David Wipf, and Zheng Zhang. 2020.
P2: A Plan-and-Pretrain Approach for Knowledge
Graph-to-Text Generation.
In Proceedings of the
3rd International Workshop on Natural Language
Generation from the Semantic Web (WebNLG+),
pages 100–106, Dublin, Ireland (Virtual).
Zhijiang Guo, Yan Zhang, Zhiyang Teng, and Wei Lu.
2019. Densely Connected Graph Convolutional Net-
works for Graph-to-Sequence Learning. Trans. As-
soc. Comput. Linguistics, 7:297–312.
Pei Ke, Haozhe Ji, Yu Ran, Xin Cui, Liwei Wang, Lin-
feng Song, Xiaoyan Zhu, and Minlie Huang. 2021.
JointGT: Graph-Text Joint Representation Learning
for Text Generation from Knowledge Graphs.
In
Findings of the Association for Computational Lin-
guistics: ACL/IJCNLP 2021, Online Event, August
1-6, 2021, volume ACL/IJCNLP 2021 of Findings
of ACL, pages 2526–2538.
Natthawut Kertkeidkachorn and Hiroya Takamura.
2020.
Text-to-Text Pre-Training Model with Plan
Selection for RDF-to-Text Generation. In Proceed-
ings of the 3rd International Workshop on Natu-
ral Language Generation from the Semantic Web
(WebNLG+), pages 159–166, Dublin, Ireland (Vir-
tual).
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A
method for stochastic optimization.
In 3rd Inter-
national Conference on Learning Representations,
ICLR 2015, San Diego, CA, USA, May 7-9, 2015,
Conference Track Proceedings.
Mike
Lewis,
Yinhan
Liu,
Naman
Goyal,
Mar-
jan Ghazvininejad, Abdelrahman Mohamed, Omer
Levy, Veselin Stoyanov, and Luke Zettlemoyer.
2020. BART: Denoising Sequence-to-Sequence Pre-
training for Natural Language Generation, Transla-
tion, and Comprehension.
In Proceedings of the
58th Annual Meeting of the Association for Compu-
tational Linguistics, ACL 2020, Online, July 5-10,
2020, pages 7871–7880. Association for Computa-
tional Linguistics.
Junyi Li, Tianyi Tang, Wayne Xin Zhao, Zhicheng Wei,
Nicholas Jing Yuan, and Ji-Rong Wen. 2021. Few-
shot Knowledge Graph-to-Text Generation with Pre-
trained Language Models. In Findings of the Associ-
ation for Computational Linguistics: ACL/IJCNLP
2021, Online Event, August 1-6, 2021, volume
ACL/IJCNLP 2021 of Findings of ACL, pages 1558–
1568. Association for Computational Linguistics.
8701
Chin-Yew Lin. 2004. Rouge: A package for automatic
evaluation of summaries.
In Text Summarization
Branches Out, pages 74–81.
Jin Liu, Chongfeng Fan, Fengyu Zhou, and Huijuan
Xu. 2022.
Syntax Controlled Knowledge Graph-
to-Text Generation with Order and Semantic Con-
sistency. In Findings of the Association for Com-
putational Linguistics: NAACL 2022, Seattle, WA,
United States, July 10-15, 2022, pages 1278–1291.
Association for Computational Linguistics.
Linyong Nan, Dragomir R. Radev, Rui Zhang, Am-
rit Rau, Abhinand Sivaprasad, Chiachun Hsieh, Xi-
angru Tang, Aadit Vyas, Neha Verma, Pranav Kr-
ishna, Yangxiaokang Liu, Nadia Irwanto, Jessica
Pan, Faiaz Rahman, Ahmad Zaidi, Mutethia Mu-
tuma, Yasin Tarabar, Ankit Gupta, Tao Yu, Yi Chern
Tan, Xi Victoria Lin, Caiming Xiong, Richard
Socher, and Nazneen Fatema Rajani. 2021. DART:
Open-domain structured data record to text genera-
tion. In Proceedings of the 2021 Conference of the
North American Chapter of the Association for Com-
putational Linguistics: Human Language Technolo-
gies, NAACL-HLT 2021, Online, June 6-11, 2021,
pages 432–447.
Jekaterina Novikova, Ondrej Dusek, and Verena Rieser.
2017. The E2E dataset: New challenges for end-to-
end generation. In Proceedings of the 18th Annual
SIGdial Meeting on Discourse and Dialogue, Saar-
brücken, Germany, August 15-17, 2017, pages 201–
206.
Vaishali Pal, Manish Shrivastava, and Irshad Bhat.
2019. Answering naturally: Factoid to full length
answer generation. In Proceedings of the 2nd Work-
shop on New Frontiers in Summarization, pages 1–9,
Hong Kong, China. Association for Computational
Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-
Jing Zhu. 2002. Bleu: A method for automatic eval-
uation of machine translation. In Proceedings of the
40th Annual Meeting of the Association for Compu-
tational Linguistics, July 6-12, 2002, Philadelphia,
PA, USA, pages 311–318.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya
Ramesh, Gabriel Goh, Sandhini Agarwal, Girish
Sastry, Amanda Askell, Pamela Mishkin, Jack Clark,
Gretchen Krueger, and Ilya Sutskever. 2021. Learn-
ing transferable visual models from natural lan-
guage supervision. In Proceedings of the 38th In-
ternational Conference on Machine Learning, ICML
2021, 18-24 July 2021, Virtual Event, volume 139 of
Proceedings of Machine Learning Research, pages
8748–8763.
Colin Raffel, Noam Shazeer, Adam Roberts, Kather-
ine Lee, Sharan Narang, Michael Matena, Yanqi
Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the
Limits of Transfer Learning with a Uniﬁed Text-to-
Text Transformer. J. Mach. Learn. Res., 21:140:1–
140:67.
Nils Reimers and Iryna Gurevych. 2019.
Sentence-
BERT: Sentence embeddings using siamese BERT-
Networks. In Proceedings of the 2019 Conference
on Empirical Methods in Natural Language Pro-
cessing and the 9th International Joint Conference
on Natural Language Processing, EMNLP-IJCNLP
2019, Hong Kong, China, November 3-7, 2019,
pages 3980–3990.
Leonardo F. R. Ribeiro, Martin Schmitt, Hinrich
Schütze, and Iryna Gurevych. 2020a. Investigating
Pretrained Language Models for Graph-to-Text Gen-
eration. CoRR, abs/2007.08426.
Leonardo F. R. Ribeiro, Yue Zhang, Claire Gardent,
and Iryna Gurevych. 2020b.
Modeling Global
and Local Node Contexts for Text Generation from
Knowledge Graphs. Trans. Assoc. Comput. Linguis-
tics, 8:589–604.
Martin Schmitt, Leonardo F. R. Ribeiro, Philipp Dufter,
Iryna Gurevych, and Hinrich Schütze. 2020. Mod-
eling Graph Structure via Relative Position for Bet-
ter Text Generation from Knowledge Graphs. CoRR,
abs/2006.09242.
Thibault Sellam, Dipanjan Das, and Ankur P. Parikh.
2020.
BLEURT: Learning robust metrics for text
generation. In Proceedings of the 58th Annual Meet-
ing of the Association for Computational Linguistics,
ACL 2020, Online, July 5-10, 2020, pages 7881–
7892.
Anastasia Shimorina and Claire Gardent. 2018. Han-
dling rare items in data-to-text generation. In Pro-
ceedings of the 11th International Conference on
Natural Language Generation, Tilburg University,
the Netherlands, November 5-8, 2018, pages 360–
370.
Yixuan Su, David Vandyke, Sihui Wang, Yimai Fang,
and Nigel Collier. 2021. Plan-then-Generate: Con-
trolled Data-to-Text Generation via Planning.
In
Findings of the Association for Computational Lin-
guistics: EMNLP 2021, Virtual Event / Punta Cana,
Dominican Republic, 16-20 November, 2021, pages
895–909. Association for Computational Linguis-
tics.
Dong Wang, Ning Ding, Piji Li, and Haitao Zheng.
2021a. CLINE: Contrastive learning with semantic
negative examples for natural language understand-
ing. In Proceedings of the 59th Annual Meeting of
the Association for Computational Linguistics and
the 11th International Joint Conference on Natural
Language Processing, ACL/IJCNLP 2021, (Volume
1: Long Papers), Virtual Event, August 1-6, 2021,
pages 2332–2342.
Qingyun Wang, Semih Yavuz, Xi Victoria Lin, Heng
Ji, and Nazneen Fatema Rajani. 2021b.
Stage-
wise Fine-tuning for Graph-to-Text Generation. In
Proceedings of the ACL-IJCNLP 2021 Student Re-
search Workshop, ACL 2021, Online, JUli 5-10,
2021, pages 16–22. Association for Computational
Linguistics.
8702
Chao
Zhao,
Marilyn
A.
Walker,
and
Snigdha
Chaturvedi. 2020.
Bridging the Structural Gap
Between Encoding and Decoding for Data-To-Text
Generation.
In Proceedings of the 58th Annual
Meeting of the Association for Computational Lin-
guistics, ACL 2020, Online, July 5-10, 2020, pages
2481–2491. Association for Computational Linguis-
tics.
Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Chris-
tian M. Meyer, and Steffen Eger. 2019. MoverScore:
Text generation evaluating with contextualized em-
beddings and earth mover distance. In Proceedings
of the 2019 Conference on Empirical Methods in
Natural Language Processing and the 9th Interna-
tional Joint Conference on Natural Language Pro-
cessing, EMNLP-IJCNLP 2019, Hong Kong, China,
November 3-7, 2019, pages 563–578.
Hao Zhou, Tom Young, Minlie Huang, Haizhou Zhao,
Jingfang Xu, and Xiaoyan Zhu. 2018.
Com-
monsense knowledge aware conversation generation
with graph attention. In Proceedings of the Twenty-
Seventh International Joint Conference on Artiﬁcial
Intelligence, IJCAI 2018, July 13-19, 2018, Stock-
holm, Sweden, pages 4623–4629.
8703


Question:
The reason this work can effectively perform well in few-shot scenarios is largely due to:
A. Unified the training objectives of pre-training and downstream tasks.
B. Using graphs to model tables.
C. Only fine-tuning a subset of parameters.
D. Conducting sufficient training during the pre-training phase.

Answer: A

To address the above two challenges, we propose a new framework called TabPrompt. First, we incorporate soft prompts (Li and Liang, 2021) into the framework and devise a prompt-based learning method suitable for tabular data. Recently, the new paradigm of “prompt-based learning” has attracted extensive attention due to its remarkable performance in few-shot scenarios (Liu et al., 2021a). Prompt-based learning enables more effective knowledge transfer by making downstream TU tasks more compatible with the pre-training task. By leveraging it, TabPrompt is capable of effectively tackling the challenges posed by few-shot TU. Second, we employ a novel Graph Contrastive Learning (Graph CL) to encode tabular data during pre-training. GNNs are well-suited for processing data with topology (Kipf and Welling, 2016), making them an ideal choice for capturing the topological information of tabular data. In addition, to encode the intrinsic structure knowledge, CL is one the most effective and popular pretext tasks (Zhang et al., 2020). In this regard, TabPrompt introduces a graph constructing method that fully considers the internal relationship between cells, which enables TabPrompt to better learn the topological structure information of the table during pretraining. To evaluate TabPrompt, we conduct extensive experiments in few-shot scenarios on three public datasets by comparing TabPrompt against several strong baselines. The results of outperforming all the baselines demonstrate the strength of our method in few-shot table understanding tasks.

Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7373–7383
December 6-10, 2023 ©2023 Association for Computational Linguistics
TabPrompt: Graph-based Pre-training and Prompting for Few-shot Table
Understanding
Rihui Jin1,2∗, Jianan Wang3∗, Wei Tan3, YongRui Chen1,2, Guilin Qi1,2, Wang Hao3†
1 School of Computer Science and Engineering, Southeast University, Nanjing, China
2 Key Laboratory of New Generation Artificial Intelligence Technology and Its
Interdisciplinary Applications (Southeast University), Ministry of Education, China
3 Alibaba Group
220212061@seu.edu.cn, {nanxi.wjn, celia.tw}@alibaba-inc.com,
{yrchen, gqi}@seu.edu.cn, wanghao.wh@alibaba-inc.com
Abstract
Table Understanding (TU) is a crucial aspect of
information extraction that enables machines
to comprehend the semantics behind tabular
data. However, existing methods of TU cannot
deal with the scarcity of labeled tabular data.
In addition, these methods primarily focus on
the textual content within the table, disregard-
ing the inherent topological information of the
table. This can lead to a misunderstanding of
the tabular semantics. In this paper, we pro-
pose TabPrompt, a new framework to tackle
the above challenges. Prompt-based learning
has gained popularity due to its exceptional per-
formance in few-shot learning. Thus, we intro-
duce prompt-based learning to handle few-shot
TU. Furthermore, Graph Contrastive Learning
(Graph CL) demonstrates remarkable capabili-
ties in capturing topological information, mak-
ing Graph Neural Networks an ideal method
for encoding tables. Hence, we develop a novel
Graph CL method tailored to tabular data. This
method serves as the pretext task during the
pre-training phase, allowing the generation of
vector representations that incorporate the ta-
ble’s topological information. The experimen-
tal results of outperforming all strong baselines
demonstrate the strength of our method in few-
shot table understanding tasks.
1
Introduction
Abundant tabular data resources are readily avail-
able nowadays (Cafarella et al., 2008).
How-
ever, the extensive knowledge within these cor-
pora remains largely untapped because most tables
are designed to be “human-friendly” rather than
“machine-friendly” (Dong et al., 2019b). Conse-
quently, to extract knowledge stored within these
vast tables, it is crucial to enable machines to com-
prehend the semantics of tabular data, referred to as
Table Understanding (TU). Furthermore, TU is fun-
damental in numerous subsequent tasks, including
∗R. Jin and J. Wang contributed equally to this work.
† Corresponding author.
Knowledge Base Augmentation (Bhagavatula et al.,
2015) and Table QA (Zhang and Balog, 2020).
Early methods involved manual feature construc-
tion designed for specific table datasets (Chen and
Cafarella, 2014) or utilizing CNN or LSTM ar-
chitectures (Nishida et al., 2017). These methods
demonstrate good performance only when there is
an ample amount of manually labeled training data
available. However, labeling tabular data requires
high labor costs due to the inherent characteristics
of tables, such as their two-dimensionality and flex-
ible layouts. Consequently, these methods struggle
to cope with the challenge posed by the scarcity of
labeled tabular data.
More recently, researchers have explored the
application of pre-trained language models to the
task of TU (Wang et al., 2020b; Deng et al., 2020;
Herzig et al., 2020). During the pre-training phase,
the model learns optimal initialization parameters
through self-supervised training on many unlabeled
tabular data, enabling it to achieve excellent per-
formance after fine-tuning downstream tasks. Al-
though self-supervised pre-training reduces the bur-
den of manual annotation to some extent, it still
requires a substantial amount of labeled table data
during the fine-tuning process. Thus, the existing
“pre-training, fine-tuning” methods also remain in-
sufficient in addressing the scarcity of labeled tab-
ular data when performing TU tasks. In other
words, existing methods struggle to perform well
in few-shot TU.
On the other hand, tables are structured data that
encompass not only textual content but also topo-
logical information related to their layout. How-
ever, the existing methods sometimes misunder-
stand the tabular semantics resulting from dis-
regarding the topological information within
tables. For example, many pre-training models
(Herzig et al., 2020) flatten a table and concatenate
its contents row by row into one-dimensional plain
texts as the input of the encoder. These methods
7373
only focusing on textual content overlook the topo-
logical information of the table, as cells in the same
column typically have semantic commonalities. As
depicted in Fig. 1 (a), cells in the second column
belong to the language category.
To address the above two challenges, we pro-
pose a new framework called TabPrompt. First, we
incorporate soft prompts (Li and Liang, 2021) into
the framework and devise a prompt-based learn-
ing method suitable for tabular data. Recently,
the new paradigm of “prompt-based learning” has
attracted extensive attention due to its remark-
able performance in few-shot scenarios (Liu et al.,
2021a). Prompt-based learning enables more effec-
tive knowledge transfer by making downstream TU
tasks more compatible with the pre-training task.
By leveraging it, TabPrompt is capable of effec-
tively tackling the challenges posed by few-shot
TU. Second, we employ a novel Graph Contrastive
Learning (Graph CL) to encode tabular data during
pre-training. GNNs are well-suited for process-
ing data with topology (Kipf and Welling, 2016),
making them an ideal choice for capturing the topo-
logical information of tabular data. In addition,
to encode the intrinsic structure knowledge, CL is
one the most effective and popular pretext tasks
(Zhang et al., 2020). In this regard, TabPrompt
introduces a graph constructing method that fully
considers the internal relationship between cells,
which enables TabPrompt to better learn the topo-
logical structure information of the table during pre-
training. To evaluate TabPrompt, we conduct ex-
tensive experiments in few-shot scenarios on three
public datasets by comparing TabPrompt against
several strong baselines. The results of outperform-
ing all the baselines demonstrate the effectiveness
of TabPrompt in the few-shot scenarios.
The contributions of this paper are summarized
in the following points:
• We apply prompt-based learning to the TU
task to tackle the scarcity of labeled tabular
data. To the best of our knowledge, this is the
first attempt to introduce prompt learning into
the field of TU.
• To obtain vector representations that incorpo-
rate the topological information of tables, we
introduce a novel Graph CL method as the
pretext task to pre-train the encoder GNN.
• To evaluate TabPrompt, we conduct experi-
ments on publicly available datasets, focusing
on two specific few-shot TU sub-tasks. In
both tasks, the results of TabPrompt outper-
forming all baselines underscore its superior-
ity in handling few-shot TU scenarios.
2
RELATED WORKS
Graph Neural Network (GNN). GNN architec-
tures, such as GCN (Kipf and Welling, 2016) and
GIN (Xu et al., 2018), have gathered substantial
interest among researchers for their remarkable ca-
pability to handle real-world data containing inher-
ent topological structure. Recognizing that tables
inherently embody topological information, sev-
eral studies (Du et al., 2021; Wang et al., 2021)
have explored the application of GNNs in table-
related research fields. However, their methods of
constructing the tabular graph fail to fully consider
the topological relationships between different cell
types, resulting in the misunderstanding of tabular
semantics.
Pre-training and Fine-tuning. Since the introduc-
tion of BERT, the “pre-training and fine-tuning”
paradigm has gained significant attention.
Re-
searchers have adapted this paradigm to TU by
designing customized pretext tasks tailored to tabu-
lar data (Wang et al., 2020b; Iida et al., 2021; Deng
et al., 2020). While these methods effectively lever-
age unlabeled data during pre-training, they still
rely on a substantial amount of labeled data in the
fine-tuning stage to achieve optimal performance.
As a result, these methods struggle to handle few-
shot TU scenarios where only a limited amount
of labeled data is available. Additionally, these
methods often flatten the table into a sequential
input, disregarding the inherent topology of the ta-
ble. This can lead to the loss of crucial topological
information during the modeling process.
Other Deep Learning-based TU. In the early
stages of table understanding (TU) research, the
emphasis was primarily on Cell Entity Linking
(Ibrahim et al., 2016; Hassanzadeh et al., 2015;
Efthymiou et al., 2017; Bhagavatula et al., 2015).
These methods often relied on pre-defined ontolo-
gies and external knowledge bases, which limited
their versatility. However, more recent work has
shifted towards addressing broader TU tasks such
as Table Cell Classification (Ghasemi-Gol et al.,
2019; Sun et al., 2021) and Table Type Classifi-
cation (Eberius et al., 2015; Nishida et al., 2017).
These methods typically utilize manual features or
early neural architectures like LSTM, which need a
large quantity of manually labeled data for training.
7374
rank
language
total
enrollments percentage
Years
Net lending/
borrowing
Primary
balance
1
Spanish
822,985
52.20%
1995
-7.3
3.8
2
French
206,426
13.10%
1996
-6.6
4.5
3
German
94,264
6%
1997
-3.0
6.2
4
American Sign
78,829
5%
1998
-3.0
4.9
5
Italian
78,368
5%
1999
-1.8
4.6
index
6
Japanese
66,605
4.20%
2000
-1.3
4.8
indexName
7
Chinese
51,582
3.30%
2001
-3.4
2.7
value
8
Arabic
23,974
2%
2002
-3.1
2.4
valueName
Net lending/borrowing & primary balance in Italy
List of most commonly learned foreign languages in the U.S.
(a) A sample table in TURL
(b) A sample table in WebSheet
Figure 1: A subtask of TU: Cell Type Classification. Cells of different colors represent different cell types.
ISBN
312424094
Authors Name
Size
Waist
Buy PbNation T-Shirts
ISBN-13
9780312424091
Raadik, Tarmo
Small
28" - 30"
About Us
Pages
224
Raadik, Tarmo A.
Medium
30" - 32"
FAQ
Publisher
Picador USA
Raas-Rothschild, A
Large
32" - 35"
Staff
Published
2004
Raasch, Beverly
XLarge
36" - 39"
Members
Language
English
Rabaa'i, Ahmad A.
XXLarge
40" - 44"
Leaderboard
Alibris ID
12873728485
Rabago, D.
(b) list table
(d) non-data table
(a) entity table
(c) relational table
Figure 2: A subtask of TU: Table Type Classification. Examples of table types from WCC.
Prompt-based Learning. The main idea of the
new paradigm “pre-train, prompt, and predict” (Liu
et al., 2021a) is to reformalize the target task to look
more like the pretext task to use better what the
model has already learned. An increasing number
of novel prompt-based learning methods have been
proposed, encompassing various forms of prompts,
such as soft prompts (Li and Liang, 2021). How-
ever, it is worth noting that there is currently no
research available on the development of prompts
specifically designed for tabular data.
3
PROPOSED METHOD
In this section, we first introduce the preliminar-
ies of TU sub-tasks relevant to our work and then
describe TabPrompt in detail. In this section, we
begin by introducing the preliminaries of TU sub-
tasks relevant to our work.
Subsequently, we
present a comprehensive description of TabPrompt.
3.1
Preliminaries
Given a table T = {ci,j|0 ≤i < N, 0 ≤j < M}
where N is the number of rows, M is the number
of columns, and ci,j is the cell located in the ith
row and jth column.
Cell Type Classification (CTC). This sub-task
of TU involves the identification of the cell type
for each cell ci,j in a table T.
CTC has been
widely studied by many works in which different
taxonomies of cell types were used (Wang et al.,
2020b; Du et al., 2021; Sun et al., 2021). In our
work, we adopt the taxonomy used in Dong et al.
2019a and expand it by including an additional type
for a comprehensive comparison, as shown in Fig.
1. Definitions of four cell types are as follows: A
value represents a basic unit describing the content
within a table. A valueName serves as a summary
unit of value cells in the same column. An index
is utilized to index value cells. An indexName
acts as a summary unit for index cells in the same
column.
Table Type Classification (TTC). This task in-
volves table-level categorization, requiring ma-
chines to classify tables based on certain taxonomy.
Similar to CTC, there are multiple taxonomies in
this task. We follow the taxonomy introduced in
Wang et al. 2020b as illustrated in Fig. 2. The
taxonomy consists of four types: relational, entity,
list, and non-data.
3.2
An Overview of TabPrompt
An overview of our framework is shown in Fig. 3
First, TabPrompt transforms the tabular data into
graph data, taking into account the topological rela-
tionships between cells during the graph construc-
tion. Second, TabPrompt utilizes the tabular Graph
CL as the pretext task to pre-train the encoder GNN.
Lastly, TabPrompt is trained on a limited amount
of labeled data to tune soft prompts. This is accom-
plished by reformulating the objectives of CTC and
7375
A Tabular Graph
A Table
Pre-tained
GNN
Pre-training
Few-shot
TU Tasks
Transforming
Prompting
Tuned
Frozen
Tuned
Frozen
Soft
Prompts
Soft
Prompts
Soft
Prompts
Figure 3: An Overview of TabPrompt.
TTC, aiming to bridge the gap with the pre-trained
objective.
3.3
Construct the Tabular Graph
Existing methods that utilize GNNs often consider
adjacent table cells or cells with similar strings as
neighboring nodes in the graph (Du et al., 2021;
Wang et al., 2021). However, these methods fail
to adequately capture the internal topological re-
lationships between nodes of different types. In
order to address this limitation, we adopt a more
sophisticated method of constructing graph data.
We establish connections between cells that exhibit
specific topological relationships. This transforma-
tion method aligns with the fundamental assump-
tion of our tabular Graph CL method, which posits
that connected nodes should have closer relation-
ships and display higher similarity in their vector
representations.
According to our observations of tabular data,
we identify several characteristics of table cells: 1)
Cell pairs within the same column exhibit higher
levels of relatedness compared to cell pairs across
different columns. 2) Cells within the header row of
a table tend to have similar levels of dependencies.
3) Each individual cell within a merged cell is the
same type. 4) The string within the header cell
often serves as a summary description for the non-
header cells in the same column. Based on these
patterns, we establish links between cells in the
following situations: 1) Adjacent cells within the
same column. 2) Adjacent cells within the header
row. 3) Cells split from merged cells. 4) Cells
within the header row and non-header cells within
the same column.
3.4
Tabular Graph Pre-training
In this section, we initially introduce our tabular
Graph CL employed for pre-training, followed by
a detailed discussion of the pre-training process.
...
Sample Triplets
A Tabular Graph
Sampling
Sampling
Tabular CL
neg
vneg
v
Tabular CL
neg
v
Training
[
]
[
]
[
]
[
]
[
]
Figure 4: Tabular Graph Contrastive Learning.
Tabular Graph Contrastive Learning. After ta-
bles are transformed into graphs, our graph CL
tailored to tabular data can be carried out. The pro-
cess of our tabular graph CL method is shown in
Fig. 4.
A tabular graph of table T can be defined as
Tg = (V, E), where V is the set of nodes repre-
senting cells C in table T. Let xij ∈Rd denote the
embedding of node vij ∈V . xij consists of the
output of the string in the cell cij through BERT
(Devlin et al., 2019) and some handcrafted features
that are general on tabular data. We observe that
the information carried by surrounding cells may
be helpful in determining the identity of the central
cell. For example, if a cell is surrounded by nu-
meric cells, the cell is also likely to be numeric. We
define Bij as the embedding of cell cij combining
with neighbors’ information. The formula of Bij
is as follows:
Bij = readout({xij, XN(xij)}),
(1)
where N(xij) is the cell set containing adjacent
cells of xij. XN(xij) is the embedding set of cells
in N(xij). Note that Bg is the vector representa-
tion of a whole tabular graph Tg. The choice of
readout function is flexible such as sum pooling,
mean pooling, or concatenation.
Our tabular Graph CL method follows the as-
sumption that the distance of connected cell pairs
7376
should be closer than that of unconnected cell pairs
in the embedding space. Formally, given a sample
triplet of cells (cab, cpos, cneg) that (cab, cpos) ∈E
and (cab, cneg) /∈E, the vector distance between
the former cell pair should be smaller than that of
the latter as follows:
dis(Bab, Bpos) < dis(Bab, Bneg),
(2)
where dis is the reciprocal of the cosine distance
function.
Pre-Training. The strength of pre-trained models
lies in their ability to leverage a vast amount of
unlabeled data. Similarly, in our tabular Graph CL,
we capitalize on its label-free nature by employing
it as the pretext task for model pre-training. During
this phase, the model learns to assign similar vector
representations to two highly related nodes. After
pre-training, we can determine whether there is a
high degree of correlation between two cells by
measuring the vector distance between them.
Before initiating the pre-training process of the
model, it is necessary to prepare the training data.
The training data comprises triples that consist of
nodes along with their corresponding positive and
negative nodes. Formally, given a tabular graph Tg,
we sample a pair of positive and negative cells for
every cell in Tg to form a triplet (cij, cpos, cneg) for
CL. To construct a sample set S for pre-training, we
sample triplets from every graph Tg in the tabular
graph set TG. We employ GNN as the pre-trained
encoder with its parameters Wi. The objective
function of pre-training is as follows:
Z(ij, pos, neg) = exp(dis(Bij, Bpos)/τ) (3)
+ exp(dis(Bij, Bneg)/τ),
Lpre(Wi) =
X
(cij,cpos,cneg)∈S
(cij,cpos)∈E
(cij,cneg)/∈E
ln exp(dis(Bij, Bpos)/τ)
Z(ij, pos, neg)
, (4)
where τ is a temperature hyperparameter often used
in CL to adjust the distribution shape.
When pre-training ends, the encoder GNN is
utilized to handle downstream TU tasks with the
pre-trained parameters Wi.
3.5
Tabular Graph Prompting
Prompting-based learning aims to bridge the sig-
nificant gap between pre-training and downstream
tasks, which allows the model to transfer the knowl-
edge acquired during pre-training more effectively.
In this section, we introduce the process of reformu-
lating two TU tasks to enhance their compatibility
with the pre-training objective, as well as the de-
tails for tuning the soft prompts. Fig. 5 shows the
process of prompting for CTC.
Prompt Addition. In the prompting stage, a key
step is to reformulate the raw input by designing
the appropriate prompt addition (Liu et al., 2021a).
For instance, in machine translation, the input “I
love you.” is reformulated as “English: I love you.
French: [Z]” with [Z] denoting an answer slot. Nev-
ertheless, applying such templates to tabular data
is challenging due to the interdependence of text
within adjacent cells. Therefore, we develop a cus-
tomized prompt addition specifically designed for
tabular data.
As mentioned above, our pre-trained model is
inclined to give closer vector representations to
highly related cells when presented with a table as
input. Consequently, the machine can determine
whether two nodes share the same type based on
the distance between their embeddings. Following
this idea, we create proxy cells for each cell type,
with their vector representations calculated as the
mean of the vector representations of the same type
of nodes in the few-shot training set. For each cell
in a table, the machine compares its vector repre-
sentation with those of all proxy nodes to determine
its type. Similar prompt addition is also employed
for the TTC task, the added proxy node represents
the type of the table, and its vector is calculated as
the mean of the BTg of tables of the same type.
Prompt tuning. In the prompt-based training pro-
cess, there is a potential risk of overfitting in few-
shot experimental settings if all parameters are
tuned (Dong et al., 2021). To mitigate this, we
freeze the pre-trained parameters Wi and introduce
the soft prompt (Fang et al., 2022). This involves
employing the learnable prompts pc and pt for the
CTC and TTC tasks, respectively. We perform the
element-wise multiplication between the vector of
each node and the corresponding soft prompt. This
operation can be seen as assigning weights to each
element of the vector. By tuning the learnable soft
prompts, the model can assign larger weights to el-
ements that are more relevant to the specific tasks,
thereby enhancing its performance.
Formally, given the CTC dataset DC
t containing
a labeled set of tabular graphs T
′
g where each cell
cij is associated with a label yC
cij, we denote the
label set for the CTC task as Y C = {yC
0 , ..., yC
4 }.
7377
index
indexName
value
valueName
Tuned
Frozen
A proxy node
Tuned
Frozen
A proxy node
CTC
the min of
Pre-tained
GNN
Pre-tained
GNN
Pre-tained
GNN
Figure 5: Prompting for CTC.
The formulas for training pc are as follows:
BC
ij = readout({pc ⊙xuv|xuv ∈Nx(ij)}), (5)
Lpro(pc) =
X
Tg∈DC
t
(cij,yC
cij )∈T ′
G
ln
exp(dis(BC
ij, ByC
cij )/τ)
P4
u=0 exp(dis(BC
ij, ByC
u )/τ)
,
(6)
where ByC
u is the vector of the proxy node repre-
senting the table cell labeled as yC
u . Similarly, the
TTC dataset DT
t contains a set of labeled tabular
graph T
′
g = (V, E, yT
g ) where yT
g is the label of its
table type. The label set of our TTC task is denoted
as Y T = {yT
0 , ..., yT
4 }. The formulas for training
pt are as follows:
BT
g = readout({pt ⊙xuv|xuv ∈V }),
(7)
Lpro(pt) =
X
(T
′
g,yT
g )∈DT
t
ln
exp(dis(BT
g , ByT
g )/τ)
P4
u=0 exp(dis(BT
g , ByT
u )/τ)
,
(8)
where ByTu is the vector of the proxy node repre-
senting the table labeled as yT
u .
4
EXPERIMENTS
In this section, we conduct experiments on public
datasets to evaluate the performance of TabPrompt
in both the CTC and TTC tasks.
4.1
Datasets
Although our method can be applied to tables from
other documents (CSV sheets, PDF documents,
etc.), we focus on web tabular data in this paper
since they are easily accessible and easy to parse.
We employ four datasets in our work. 1) TabEL
(Bhagavatula et al., 2015) contains around 1.6M
tables extracted from Wikipedia pages. 2) TURL
(Deng et al., 2020) selected 670k relational tables
from TabEL and annotated the column number cor-
responding to the subject column of each relational
table. 3) WebSheet (Dong et al., 2019a) contains
about 3k tables with cell type labels which were se-
lected and manually annotated from web-crawled
Websheet (only 50 of them publicly available). 4)
WCC (Ghasemi-Gol and Szekely, 2018) is a sam-
ple of July 2015 Common Crawl and is annotated
with the web table taxonomy introduced by Crestan
and Pantel 2011. The taxonomies adopted in these
datasets differ slightly from the taxonomy used in
this paper. We comprehensively describe the cor-
respondence between the various taxonomies and
other details in the appendix A.
4.2
Baselines
We compare TabPrompt with strong baselines
for CTC and TTC to verify the effectiveness of
TabPrompt.
We employ the following baselines for CTC.
TCC-Embd (Ghasemi-Gol et al., 2019) is a tab-
ular cell classification method with pre-trained
CBOW&Skip-gram cell embeddings (Mikolov
et al., 2013). PSL (Sun et al., 2021) reformalizes
the CTC task into block detection, which aims to
detect the data blocks in the table. TabularNet
(Du et al., 2021) utilizes a homogeneous graph
constructed out of the WordNet (Fellbaum, 2000)
knowledge base and adopts GCN as the encoder
and LSTM. TableFormer (Yang et al., 2022) is guar-
anteed to be invariant to row and column order
perturbations. Because it is specifically for Table
QA and Table Verification, it means that it cannot
handle the TU directly. To adapt it for TU, we treat
these models as encoders that output embeddings
of table cells and stack layers of downstream classi-
fiers (including an MLP layer and a pooling layer)
on top of the encoders. In that case, these meth-
ods can be compared with our method. FewTPT
(Liu et al., 2022) is a Chinese tabular language
7378
model, thus, we pre-train it using 570k Wiki tables
in English from scratch. While the "Table Classi-
fication" mentioned in FewTPT centres on identi-
fying header domains within tables, distinct from
the focus of TTC, we employ a similar processing
methods as in the case of TableFormer to facilitate
handling the TU addressed in this paper. TUTA
(Wang et al., 2020b) is a pre-training model with
tree-based transformers for TU, which achieves
SOTA performance among existing methods.
We employ the following baselines for TTC.
DWTC (Eberius et al., 2015) train a Random For-
est model through manually engineered features.
TabNet (Nishida et al., 2017) utilizes a hybrid deep
neural network architecture of LSTM and CNN.
TabVec (Ghasemi-Gol and Szekely, 2018) is an
unsupervised method to embed tables based on
table-level manual features. Additionally, when
considering TTC, TableFormer and FewTPT can
also be assessed by applying stacked downstream
classifiers. In addition to CTC, TUTA (Wang et al.,
2020b) also performs well on TTC.
4.3
Settings and parameters
The embedding of a node in a tabular graph is com-
posed of semantic information and manual features.
The embedding of semantic information is obtained
by feeding the string of the node into Sentence-
BERT (Reimers and Gurevych, 2019) and then
performing dimensionality reduction. The manual
features used in our work are introduced in Crestan
and Pantel 2011, which have strong versatility on
tabular data. We present details of the manual fea-
tures in Appendix B. To be fair, the manual features
used in TabPrompt are also used by all baselines.
We employ the macro-F1 score and the standard de-
viation as the evaluation metric in our experiments,
commonly used in other TU works.
In the pre-training phase, we employ a 3-layer
GIN as the backbone and set the hidden dimension
as 64. We randomly sample 5k tables from TabEL
and utilize the tabular Graph CL method mentioned
above as the pretext task to pre-train the parameters
of the backbone. The details of hyper-parameters
can be found in Appendix C.
Following a typical k-shot classification setting
(Liu et al., 2021b; Zhou et al., 2019; Wang et al.,
2020a), we generate a series of few-shot down-
stream tasks of CTC and TTC for model training,
validation, and testing.
For CTC, we conduct this downstream task on
Table 1: Macro-F1 evaluation on CTC.
Methods
TURL
WebSheet
1-shot
3-shot
1-shot
3-shot
TCC-Embd
26.72 ± 6.27
31.14 ± 8.51
19.62 ± 12.54
21.42 ± 12.39
PSL
29.23 ± 7.55
39.72 ± 7.43
15.53 ± 10.11
18.55 ± 13.18
TabularNet
34.41 ± 7.47
45.14 ± 4.77
20.64 ± 12.92
29.44 ± 12.56
TableFormer
33.68 ± 6.33
45.78 ± 5.88
21.74 ± 8.95
27.75 ± 7.15
FewTPT
33.21 ± 10.22
44.71 ± 8.13
22.93 ± 9.75
28.41 ± 8.42
TUTA
35.73 ± 6.81
47.32 ± 9.48
27.65 ± 13.25
34.77 ± 9.84
TabPrompt
46.82 ± 8.80
57.79 ± 9.01
30.78 ± 10.25
35.96 ± 9.76
two datasets, i.e., TURL and WebSheet. On each
dataset, we generate ten z-shot Cell Type Classi-
fication tasks for training and validation. In each
task, z tables are randomly sampled for both train-
ing and validation, where z ∈{1, 3}. Additionally,
we randomly generate 20 tables for testing from
the remaining not sampled for training and vali-
dation. After conducting the experiments, we cal-
culate the mean macro-F1 score of the ten testing
results, along with the standard deviation. For TTC,
following a similar process of CTC, we randomly
generate 10 z-shot (z ∈{3, 5}) table classification
tasks (i.e., sample z tables per class in one task)
from WCC and the remaining tables not sampled
for testing.
4.4
Performance Evaluation
In this section, we analyze the results of exper-
iments of CTC and TTC conducted on public
datasets, respectively.
CTC. Based on the results of the few-shot CTC
presented in Table 1, we draw the following con-
clusions.
Firstly, the superior performance of
TabPrompt over all baselines underscores the effec-
tiveness of our proposed framework. This result
proves that prompt-based learning is well-suited for
addressing few-shot TU scenarios. Secondly, de-
spite TUTA having a higher number of parameters
pre-trained with more data, TabPrompt achieves
better performance. This finding highlights the sig-
nificance of bridging the gap between pre-training
and downstream tasks, as it enhances the effec-
tiveness of knowledge transfer from pre-training
to downstream tasks.
Thirdly, TabPrompt out-
performs TabularNet, indicating that considering
the topological information within the table sig-
nificantly aids the model in better comprehending
the table’s semantics. Fourthly, TableFormer and
FewTPT exhibit inferior performance compared to
TUTA despite all being pre-trained. This is mainly
because neither are designed for the TU task, lead-
ing to the absence of modules tailored to TU tasks.
7379
Table 2: Macro-F1 evaluation on TTC.
Methods
WCC
3-shot
5-shot
DWTC
26.12 ± 9.71
31.52 ± 10.01
TabNet
14.53 ± 13.76
16.35 ± 10.17
TabVec
31.43 ± 8.63
49.12 ± 10.26
TableFormer
42.14 ± 6.73
50.17 ± 6.18
FewTPT
44.97 ± 7.13
52.72 ± 6.42
TUTA
46.12 ± 8.73
54.45 ± 8.27
TabPrompt
50.75 ± 7.18
57.41 ± 5.14
TTC. We present the results of few-shot TTC in Ta-
ble 2. First, the consistently superior performance
of TabPrompt once again demonstrates the effec-
tiveness of our proposed framework. Second, as
both CTC and TTC share the same parameters of
the pre-trained model, the superior performance of
GraphPrompt on both types of tasks further sup-
ports the notion that our tabular Graph CL enables
the models to learn better vector representations.
Performance with different shots. From Table
1 and Table 2, it is evident that the performance
of all models improves as the number of samples
increases. Therefore, we study the trends in per-
formance with varied shots. We have Method A,
Method B, and our method tested with different
shots on the TURL dataset in Fig 6. The graph
shows that in the case of low shots, TabPrompt
consistently outperforms other methods. Although
TabPrompt is surpassed when it comes to 50 shots
(typically beyond 600 table cells), the amount of
data is beyond the scope of our target scenario.
25
40
55
70
85
1
3
5
1 0
2 0
5 0
TabularNet
TUTA
TabPrompt
Figure 6: Impact of varied shots on CTC.
4.5
Ablation study
To analyze the individual contribution of each com-
ponent in TabPrompt, we conduct the following
ablation study:
• w/o cl: When constructing graph data without
Table 3: Ablation results.
CTC
CTC
TTC
(TUTA)
(WebSheet)
(WCC)
3-shot
3-shot
5-shot
TabPrompt
57.79 ± 9.01
35.96 ± 9.76
57.41 ± 5.14
w/o cl
46.12 ± 10.48
31.52 ± 9.01
54.52 ± 8.76
w/o sp
55.74 ± 11.16
33.25 ± 9.87
56.14 ± 9.11
w/o pr
49.15 ± 9.38
30.18 ± 10.47
52.32 ± 13.5
taking into account the topological informa-
tion of the table, link all adjacent cells simply.
• w/o sp: Downstream tasks are performed
without tuning the parameters of the soft
prompt.
• w/o pr: Using fine-tuning instead of prompt
tuning to train the model in downstream tasks.
Results of ablation experiments are shown in Ta-
ble 3. It is evident that TabPrompt’s performance
significantly deteriorates when using a vanilla
method to construct graph data. This can be at-
tributed to the fact that the vanilla method dramati-
cally diminishes the effectiveness of tabular Graph
CL. It confirms that considering the topological
information within tables aids machines in compre-
hending the semantics of tables more effectively.
In our previous analysis, we hypothesize that the
soft prompt can assign more weight to distinguish
features during the prompting process. We observe
that when the parameters of the soft prompt are not
tuned, the performance of TabPrompt experiences
a slight decrease. This observation confirms our
hypothesis.
When we replace the prompt tuning method with
fine-tuning, we observe a decrease in the model’s
performance across all three datasets. This out-
come further reinforces that prompt tuning can
effectively enhance the model’s performance in
scenarios with limited training tabular data.
4.6
Case Study
To facilitate an intuitive interpretation, we present
a typical case in the CTC testing data that demon-
strates the functionality of tabular Graph CL.
Fig. 7 shows two classification results for a ta-
ble. Fig. 7 (a) represents a correct CTC result,
whereas the classification results in Fig. 7 (b) have
some flaws. It highlights that misunderstanding
may result from neglecting the topological struc-
ture information within the table, such as the cells
in one column tending to be a category as well
as that cells in the header rows being more likely
7380
to be a valueName rather than a value. For in-
stance, the misclassified cell “July 1780 Typhoon”
should have a stronger relationship with the up-
per and lower cells, while the relationship with the
right cell “1780” is relatively weak despite both
containing the textual content of “1780”.
indexName
index
valueName
value
year
fatalities
1881
300,000
1975
229,000
1780
100,000
1922
60,000
1912
50,000
year
fatalities
1881
300,000
1975
229,000
1780
100,000
1922
60,000
1912
50,000
typhoon
" Haiphong "
Deadliest Pacific typhoons
(a) CTC results of TabPrompt
" Haiphong "
Nina
July 1780 Typhoon
" Swatow "
"China"
typhoon
(b) CTC results of TabPrompt w / o  cl
Nina
July 1780 Typhoon
" Swatow "
"China"
Figure 7: A real case for CTC.
5
CONCLUSION
In this paper, we proposed a new framework
TabPrompt. To tackle the scarcity of labeled tab-
ular data and capture the untapped topological in-
formation, we resorted to prompt-based learning
and Graph CL, respectively. The experimental re-
sults of outperforming all baselines demonstrate
the effectiveness of TabPrompt in few-shot TU.
In future work, we aim to extend our method to
handle domain-specific tables and more complex
table structures, further enhancing its capabilities.
In addition, we introduce a multimodal technique
that leverages both text and image data within the
table to enhance the understanding of its semantics.
Limitations
First, the input of TabPrompt must be a machine-
parsable data format, such as HTML files with
<table> tags or CSV files. TabPrompt does not
encompass the processes of locating or extracting
tables from images or original web pages. Sec-
ond, TabPrompt cannot handle tables with overly
complex layouts, such as tables with images or too
many empty cells. Third, TabPrompt performs
poorly on small-sized tables due to the limited
amount of text information available within these
tables.
Acknowledgement
This work is partially supported by National
Nature Science Foundation of China under No.
U21A20488. We thank the Big Data Computing
Center of Southeast University for providing the
facility support on the numerical calculations in
this paper.
References
Chandra Bhagavatula, Thanapon Noraset, and Doug
Downey. 2015. Tabel: Entity linking in web tables.
In International Workshop on the Semantic Web.
Michael J. Cafarella, Alon Y. Halevy, Daisy Zhe Wang,
Eugene Wu, and Yang Zhang. 2008. Webtables: ex-
ploring the power of tables on the web. Proc. VLDB
Endow., 1:538–549.
Zhe Chen and Michael J. Cafarella. 2014. Integrating
spreadsheet data via accurate and low-effort extrac-
tion. Proceedings of the 20th ACM SIGKDD interna-
tional conference on Knowledge discovery and data
mining.
Eric Crestan and Patrick Pantel. 2011. Web-scale table
census and classification. In Web Search and Data
Mining.
Xiang Deng, Huan Sun, Alyssa Lees, You Wu, and
Cong Yu. 2020. Turl: Table understanding through
representation learning. ArXiv, abs/2006.14806.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and
Kristina Toutanova. 2019. Bert: Pre-training of deep
bidirectional transformers for language understand-
ing. ArXiv, abs/1810.04805.
Haoyu Dong, Shijie Liu, Zhouyu Fu, Shi Han, and
Dongmei Zhang. 2019a. Semantic structure extrac-
tion for spreadsheet tables with a multi-task learning
architecture.
Haoyu Dong, Shijie Liu, Shi Han, Zhouyu Fu, and
D. Zhang. 2019b. Tablesense: Spreadsheet table
detection with convolutional neural networks. In
AAAI Conference on Artificial Intelligence.
Xinhsuai Dong, Anh Tuan Luu, Min Lin, Shuicheng
Yan, and Hanwang Zhang. 2021. How should pre-
trained language models be fine-tuned towards adver-
sarial robustness? In Neural Information Processing
Systems.
Lun Du, Fei Gao, Xu Chen, Ran Jia, Junshan Wang, Shi
Han, and Dongmei Zhang. 2021. Tabularnet: A neu-
ral network architecture for understanding semantic
structures of tabular data. Proceedings of the 27th
ACM SIGKDD Conference on Knowledge Discovery
& Data Mining.
7381
Julian Eberius, Katrin Braunschweig, Markus Hentsch,
Maik Thiele, Ahmad Ahmadov, and Wolfgang
Lehner. 2015. Building the dresden web table cor-
pus: A classification approach. 2015 IEEE/ACM 2nd
International Symposium on Big Data Computing
(BDC), pages 41–50.
Vasilis Efthymiou,
Oktie Hassanzadeh,
Mariano
Rodriguez-Muro, and Vassilis Christophides. 2017.
Matching web tables with knowledge base entities:
From entity lookups to entity embeddings. In Inter-
national Workshop on the Semantic Web.
Taoran Fang, Yunchao Zhang, Yang Yang, and Chun-
ping Wang. 2022. Prompt tuning for graph neural
networks. ArXiv, abs/2209.15240.
Christiane D. Fellbaum. 2000. Wordnet : an electronic
lexical database. Language, 76:706.
Majid Ghasemi-Gol, Jay Pujara, and Pedro A. Szekely.
2019. Tabular cell classification using pre-trained
cell embeddings. 2019 IEEE International Confer-
ence on Data Mining (ICDM), pages 230–239.
Majid Ghasemi-Gol and Pedro A. Szekely. 2018. Tab-
vec: Table vectors for classification of web tables.
ArXiv, abs/1802.06290.
Oktie Hassanzadeh,
Michael J. Ward,
Mariano
Rodriguez-Muro, and Kavitha Srinivas. 2015. Un-
derstanding a large corpus of web tables through
matching with knowledge bases: an empirical study.
In Organizational Memories.
Jonathan Herzig, Pawel Krzysztof Nowak, Thomas
Müller, Francesco Piccinno, and Julian Martin Eisen-
schlos. 2020. Tapas: Weakly supervised table parsing
via pre-training. In Annual Meeting of the Associa-
tion for Computational Linguistics.
Yusra Ibrahim, Mirek Riedewald, and Gerhard Weikum.
2016. Making sense of entities and quantities in web
tables. Proceedings of the 25th ACM International
on Conference on Information and Knowledge Man-
agement.
Hiroshi Iida, Dung Ngoc Thai, Varun Manjunatha, and
Mohit Iyyer. 2021. Tabbie: Pretrained representa-
tions of tabular data. In North American Chapter of
the Association for Computational Linguistics.
Thomas Kipf and Max Welling. 2016. Semi-supervised
classification with graph convolutional networks.
ArXiv, abs/1609.02907.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning:
Optimizing continuous prompts for generation. Pro-
ceedings of the 59th Annual Meeting of the Associa-
tion for Computational Linguistics and the 11th Inter-
national Joint Conference on Natural Language Pro-
cessing (Volume 1: Long Papers), abs/2101.00190.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang,
Hiroaki Hayashi, and Graham Neubig. 2021a. Pre-
train, prompt, and predict: A systematic survey of
prompting methods in natural language processing.
ACM Computing Surveys, 55:1 – 35.
Ruixue Liu, Shaozu Yuan, Aijun Dai, Lei Shen, Tian-
gang Zhu, and Meng Chen. 2022. Few-shot table
understanding: A benchmark dataset and pre-training
baseline. In International Conference on Computa-
tional Linguistics.
Zemin Liu, Yuan Fang, Chenghao Liu, and Steven C. H.
Hoi. 2021b. Relative and absolute location embed-
ding for few-shot node classification on graph. In
AAAI Conference on Artificial Intelligence.
Tomas Mikolov, Kai Chen, Gregory S. Corrado, and
Jeffrey Dean. 2013. Efficient estimation of word
representations in vector space. In International Con-
ference on Learning Representations.
Kyosuke Nishida, Kugatsu Sadamitsu, Ryuichiro Hi-
gashinaka, and Yoshihiro Matsuo. 2017. Understand-
ing the semantic structures of tables with a hybrid
deep neural network architecture. In AAAI Confer-
ence on Artificial Intelligence.
Nils Reimers and Iryna Gurevych. 2019. Sentence-bert:
Sentence embeddings using siamese bert-networks.
In Conference on Empirical Methods in Natural Lan-
guage Processing.
Kexuan Sun, Harsha Rayudu, and Jay Pujara. 2021. A
hybrid probabilistic approach for table understanding.
In AAAI Conference on Artificial Intelligence.
Fei Wang, Kexuan Sun, Muhao Chen, Jay Pujara, and
Pedro A. Szekely. 2021. Retrieving complex tables
with multi-granular graph representation learning.
Proceedings of the 44th International ACM SIGIR
Conference on Research and Development in Infor-
mation Retrieval.
Ning Wang, Minnan Luo, Kaize Ding, Lingling Zhang,
Jundong Li, and Qinghua Zheng. 2020a. Graph few-
shot learning with attribute matching. Proceedings
of the 29th ACM International Conference on Infor-
mation & Knowledge Management.
Zhiruo Wang, Haoyu Dong, Ran Jia, Jia Li, Zhiyi Fu,
Shi Han, and Dongmei Zhang. 2020b. Tuta: Tree-
based transformers for generally structured table pre-
training. Proceedings of the 27th ACM SIGKDD
Conference on Knowledge Discovery & Data Mining.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie
Jegelka. 2018. How powerful are graph neural net-
works? ArXiv, abs/1810.00826.
Jingfeng Yang, Aditya Gupta, Shyam Upadhyay,
Luheng He, Rahul Goel, and Shachi Paul. 2022.
Tableformer: Robust transformer modeling for table-
text encoding. In Annual Meeting of the Association
for Computational Linguistics.
Shichang Zhang, Ziniu Hu, Arjun Subramonian, and
Yizhou Sun. 2020. Motif-driven contrastive learning
of graph representations. In AAAI Conference on
Artificial Intelligence.
7382
Shuo Zhang and Krisztian Balog. 2020.
Web table
extraction, retrieval, and augmentation: A survey.
ACM Trans. Intell. Syst. Technol., 11:13:1–13:35.
Fan Zhou, Chengtai Cao, Kunpeng Zhang, Goce Tra-
jcevski, Ting Zhong, and Ji Geng. 2019.
Meta-
gnn: On few-shot node classification in graph meta-
learning. Proceedings of the 28th ACM International
Conference on Information and Knowledge Manage-
ment.
A
Correspondence with Different
Taxonomies and More Details
In this section, we comprehensively describe the
correspondence between the various taxonomies.
The annotation taxonomy employed in TURL
focuses on annotating the subject column within
tables. For instance, in Fig. 1 (a), the subject
column of the table is the second column titled
“language”. The correspondence between the tax-
onomy of TURL and ours is illustrated in Fig. 8.
The taxonomy of this WebSheet is nearly identical
to ours, except that they do not explicitly annotate
unlabeled cells with value. Regarding the WCC
dataset, we classify the tables categorized as Ma-
trix into the category of Relational tables. This
decision is made due to the absence of clear bound-
aries between these two types, as they share similar
cell types.
To ensure data quality, we applied filters to re-
move certain tables. This includes tables with ir-
regular layouts, where rows do not have the same
number of cells as other rows, as well as tables
with pictures, huge tables, and repeated tables. The
filtering process was implemented through some
rules.
√
×
√
indexName
index
×
valueName
value
is located in header rows?
is located in the
 subject column?
Figure 8: Manual features.
B
Manual Features
The main manual features used in our work are
listed in Fig 9.
C
Hyper-parameters
The semantic features of nodes are first output
by Roberta and then reduced dimensions to 512
through PCA. We employ a 3-layer GIN architec-
ture as the backbone whose hidden dimension is set
For CTC
the number of row the current cell is located in
the number of column the current cell is located in
whether the string of the current cell
is unique in the current column
the string type of the current cell
whether the current cell is from the merged cell
For TTC
the number of rows of the current table
the number of columns of the current table
Figure 9: Manual features.
as 1024. The activation function of GIN is ReLu.
For pre-training, we set the learning rate as 0.01,
the weight decay as 1e-5, and the dropout as 0.5.
We set the readout function as mean pooling.
7383