Question:
Which datasets contain hierarchical tables?
A. IM-TQA
B. WTQ (Raw)
C. WTQ (Flatten)
D. HiTab

Answer: ABD

To evaluate HeGTa, we selected a variety of datasets that are widely studied and easy to parse, including WCC (Ghasemi-Gol and Szekely 2018), IM-TQA (Zheng et al. 2023), HiTab (Cheng et al. 2021b), WTQ (Raw) along with its header-flatten version WTQ (Flatten) (Pasupat and Liang 2015) and TabFact (Chen et al. 2020), each pertinent to CTC, TTC, or TQA, respectively.

Statistics for these datasets are presented in Table 1, illustrating the types of annotations, the primary domains covered, and the proportion of complex tables. Given the focus on few-shot TU scenarios, we only list the size of test sets. Comprehensive details of the dataset annotations and preprocessing are available in Appendix.

Datasets
Annotation Type
Table Info
CTC
TTC
TQA
# Test Tables
% Complex Tables
# QA pairs
Main Domains
IM-TQA
✓
✓
✓
153
47.71
627
Manufacturing
WCC
✓
371
–
–
General
HiTab
✓
✓
3597
92.88
1584
Crime, Health
WTQ (Flatten)
✓
2108
0.00
4344
General
WTQ (Raw)
✓
2108
14.80
4344
General
TabFact
✓
1695
0.00
12779
General

3) WTQ (Raw) represents the unprocessed version of WTQ (Flat), preserving the complexity of hierarchical table headers.

HeGTa: Leveraging Heterogeneous Graph-enhanced Large Language Models for
Few-shot Complex Table Understanding
Rihui Jin1,2, Yu Li1,2, Guilin Qi1,7*, Nan Hu1,2, Yuan-Fang Li3, Jiaoyan Chen4, Jianan Wang5,
Yongrui Chen1,2, Dehai Min1,2, Sheng Bi6
1School of Computer Science and Engineering, Southeast University, China
2Key Laboratory of New Generation Artificial Intelligence Technology and its Interdisciplinary Applications
(Southeast University), Ministry of Education, China
3Monash University 4The University of Manchester 5Alibaba Group
6Law and Innovation Lab, Law School, Southeast University, China
7State Key Laboratory for Novel Software Technology, Nanjing University, China
{ari king, yuli 11, gqi, nanhu, yrchen, zhishanq, bisheng}@seu.edu.cn, yuanfang.li@monash.edu,
jiaoyan.chen@manchester.ac.uk, jn wang2@outlook.com
Abstract
Table Understanding (TU) has achieved promising advance-
ments, but it faces the challenges of the scarcity of manually
labeled tables and the presence of complex table structures.
To address these challenges, we propose HeGTa, a hetero-
geneous graph (HG)-enhanced large language model (LLM)
designed for few-shot TU tasks. This framework aligns struc-
tural table semantics with the LLM’s parametric knowledge
through soft prompts and instruction tuning. It also addresses
complex tables with a multi-task pre-training scheme by in-
corporating novel multi-granularity self-supervised HG pre-
text tasks. We empirically demonstrate the effectiveness of
HeGTa, showing that it outperforms the SOTA for few-shot
complex TU on several benchmarks.
1
Introduction
Table Understanding (TU) seeks to learn informative em-
beddings of tables containing inherently tabular semantics,
often formatted in ways not easily interpretable by ma-
chines, as shown in Fig. 1. This endeavor enhances machine
performance across a range of table-related tasks, such as
Table QA (Wang et al. 2024), Cell Type Classification (Jia
et al. 2023) and Table Type Classification (Lu et al. 2024).
Yet, in real-world scenarios, TU faces the challenges of a
lack of sufficient human annotations and the presence of
complex table structures, which diminishes the effective-
ness and applicability of existing frameworks. As for the first
challenge, the data-hungry nature of existing frameworks re-
sults in diminished performance in few-shot TU scenarios,
where only several samples are annotated. Although some
studies (Dong et al. 2022; Herzig et al. 2020) have adopted
pre-training with encoder-only architectures (Devlin et al.
2019) to alleviate the annotation burden, these solutions
still require considerable amounts of labeled data for task-
specific fine-tuning. Regarding the second challenge, while
*Corresponding author.
Copyright © 2025, Association for the Advancement of Artificial
Intelligence (www.aaai.org). All rights reserved.
Hierarchical
2
1
0
Amount incurred
in the current period
project
0
1
cost
income
36,829,903.92
53,196,521.18
Main business
2
2,591,042.45
4,868,204.64
Other businesses
3
Amount incurred
in the current period
Project
cost
income
36,829,903.92
53,196,521.18
Main
business
2,591,042.45
4,868,204.64
Other
businesses
TU Model
Q: How much is the income of
the other businessesof the
company’s current amount?
A: Cell(3, 1) or 4,868,204.64
Cell Type Classification
Table Type Classification
Table QA
Table Type: Relational type
Embeddings
Parallel
Dependency
Specific Table-related Tasks
( lack of sufficient manual labels )
Complex Table
Figure 1: Few-shot complex Table Understanding. Complex
tables contain intricate cell-to-cell relationships, including
dependency, hierarchical, and parallel ones.
current frameworks attempt to capture structural informa-
tion in tables through the use of position embeddings (Wang
et al. 2020c) or by modeling tables as graphs (Jia et al. 2023;
Wang et al. 2021), these methods are effective for simple
tables, but wane with complex tables. This shortfall arises
because the cell-to-cell relationships in complex tables are
more intricate than those in simple tables, as shown in Fig. 1.
Fortunately, recent advancements have introduced tech-
niques that can be integrated into the TU framework to
arXiv:2403.19723v2  [cs.CL]  15 Dec 2024
tackle the challenges: 1) Large Language Models (LLMs)
have shown remarkable effectiveness in managing few-shot
tasks involving data in other modalities, such as vision (Liu
et al. 2023) and time series (Jin et al. 2023a). This is
achieved through instruction tuning and soft prompts, which
seek to align the semantic spaces of the other modality en-
coder with that of the LLM, a method that can also be ap-
plied to tabular data. 2) Self-supervised heterogeneous graph
(HG) pre-training (Ma, Liu, and Zuo 2023; Yang et al. 2022)
enables models to navigate multifaceted relationships within
data, such as complex intercell relationships in Fig. 1, using
substantial volumes of unlabeled data.
We propose HeGTa, an HG-enhanced LLM framework
for the few-shot complex TU. To enable HeGTa to model
intricate relationships within complex tables, HeGTa begins
with modeling the tabular data with the HG and then pro-
cessing it through a tabular graph encoder to generate vec-
tors imbued with structural information. Additionally, to en-
sure that the LLM achieves comparable performance in ta-
ble tasks to its performance in natural language (NL) tasks,
especially in few-shot scenarios, we align the representation
spaces of the tabular graph encoder and the LLM via instruc-
tion tuning. Specifically, we integrate table vectors contain-
ing structural information as soft prompts within the LLM
inputs and innovatively design three multi-granularity self-
supervised tasks tailored for both tables and LLMs for pre-
training. After pre-training specific parameters of HeGTa
via these self-supervised tasks, the HeGTa can capitalize on
the LLM’s exceptional generalization capability to adapt to
downstream tasks with minimal data samples. To validate
the performance of HeGTa in few-shot complex TU, we con-
duct extensive comparative experiments with existing pow-
erful baselines on nine publicly available datasets for three
specific table-related tasks in few-shot scenarios. The exper-
imental results show that HeGTa exceeds the current SOTA
for the few-shot complex TU across multiple benchmark
datasets. In summary, our main contributions are as follows.
• To the best of our knowledge, we are the first to pro-
pose a framework to align table semantics with NLs’ to
empower the LLM to perform on tables with the same
few-shot proficiency as it does with NLs.
• To improve the framework’s ability to complex TU, we
propose a refined way to convert tables into HGs and
design three novel multi-granularity self-supervised HG
pre-training tasks tailored for tabular data and LLMs.
• We conduct extensive experiments on nine publicly avail-
able datasets, and the experimental results show that
HeGTa exceeds the current SOTA for few-shot complex
TU across multiple benchmark datasets.
2
Related Work
2.1
Graph-based Table Understanding
Many studies (Du et al. 2021; Jin et al. 2023b) have con-
verted tables into graphs and utilized graph encoders to cap-
ture tables’ inherent topological information. These frame-
works include employing homogeneous graphs or utilizing a
basic node-linking strategy that connects cells exclusively to
their adjacent counterparts. Consequently, such frameworks
underperform when dealing with complex table structures.
2.2
LLM-based Textual Table Understanding
Following the success of LLMs in NL tasks, some ef-
forts (Zhang et al. 2023a; Chen 2022; Sui et al. 2024) have
extended their application to table-related tasks. Although
these frameworks leverage the exceptional generalization
capacity of LLMs to achieve SOTA performance in some
few-shot tasks, they resort to simply converting the table into
a row-by-row NL format as the input. This process leads to
a loss of the tables’ intrinsic topological information.
2.3
LLM-based Symbolic Table Understanding
Inspired by Program-of-Thoughts(Chen et al. 2022), several
studies (Cao et al. 2023; Cheng et al. 2022; Wang et al.
2024; Zhang et al. 2023b; Ye et al. 2023) have adapted
semantic parsing techniques, traditionally applied to rela-
tional database tables, to general Table QA tasks. This in-
volves transforming the table into a format interpretable by
query languages, such as SQL, and subsequently utilizing
LLMs to generate a symbolic program to retrieve the answer.
This methodology represents the current SOTA in Table QA.
However, its architectural limitations restrict its applicability
to other table-related tasks.
2.4
Bert-like Encoder-only Table Understanding
Since the rise in popularity of pre-training models like
BERT (Devlin et al. 2019), there has been considerable
effort devoted to designing specialized encoding methods
for tabular data and unique pre-text objectives for pre-
training (Yin et al. 2020; Cheng et al. 2021a; Jia et al. 2023).
Despite their utilization of self-supervised training, these
methods still require a substantial amount of labeled data
during fine-tuning for downstream tasks. Additionally, while
they incorporate positional embedding into serialized tabular
data, they do not effectively capture topological information.
3
Task Definition
Given a table T = {ci,j|0 ≤i < N, 0 ≤j < M} where N
is the number of rows, M is the number of columns, and ci,j
is the cell located in the ith row and jth column. Merged
cells, characterized by a row span or column span greater
than 1, are prevalent in complex tables. For instance, the cell
labeled ”project” in the top-left corner of the table shown
in Fig. 3 has a row span of 2. We assign the coordinates of
such merged cells based on the position of the top-left cell
before merging. Hence, the coordinate of the ”project” cell
is designated as (0,0).
Specific sub-TU tasks for Evaluation
Cell Type Classification (CTC) involves identifying the
type yc of each cell ci,j within a table T, where yc can be-
long to a basic taxonomy Yc = {header cell, data cell} or
a more complex one, varying across datasets.
Table Type Classification (TTC) is a table-level catego-
rization task that requires models to classify the table T ac-
cording to a specific taxonomy Yt.
Table QA (TQA) demands that the model produce an an-
swer ya in response to a natural language question q, with
table T serving as the reference for deriving the answer ya.
4
METHODOLOGY
This section explains three phases of HeGTa: tabular HG
construction and two tuning stages, as shown in Fig. 2.
4.1
Tabular Heterogeneous Graph Construction
Given that HGs are more proficient at capturing diverse re-
lationships compared to homogeneous graphs, we employ
HGs with heuristic node-linking rules to model the struc-
ture of complex tables effectively. The subsequent subsec-
tions detail the process of converting tabular data into HGs,
including the creation of nodes and the heuristic rules for
establishing edges between nodes. The process of creating a
tabular HG is shown in Fig. 3.
Four Tabular Node Types
TABLE, ROW, Header CELL,
and Data CELL nodes are denoted as green, red, yellow, and
blue nodes in Fig. 3. 1) The TABLE node represents the con-
tent described by the table, facilitating the table-level tasks.
2) The ROW node signifies the information contained within
a row, aiding in the prediction of the row’s type during self-
supervised training. 3) The Header CELL node denotes cells
located in the header row, identifying column schemas or
categories. 4) The Data CELL node represents cells in data
rows, meaning the actual data entries of the table.
Creating Tabular Nodes
First, CELL nodes are created
for each cell in the table, with each node denoted as ci,j,
where i and j represent the cell’s coordinates in the original
table. If a CELL node is located in the table’s header section
or is part of a merged cell spanning the entire table width,
it is viewed as a Header Cell; otherwise, it is identified as a
Data Cell. The algorithm for determining the header section
is elaborated in Appendix. Subsequently, ROW nodes are
created to match the number of rows in the table, along with
a single TABLE node.
Initializing Tabular Node Embeddings
An initializa-
tion vector is required for each node in the HG. For
Header and Data CELL nodes, we employ the output of
S-BERT (Reimers and Gurevych 2019) applied to the text
within each cell to obtain their initialization vectors. In the
case of ROW nodes ri, we initialize a vector by concatenat-
ing the text from the cells in the ith row and inputting this
concatenated text into the S-BERT. TABLE nodes serve to
represent the table’s content itself. In a human’s view, a ta-
ble’s content can be inferred by examining the cells in the
headers. Consequently, we opt to concatenate the text from
the cells in the headers and to obtain the embedding.
Adding Edges
To enhance the machine’s understanding
of the table’s semantics, we develop the following heuris-
tic rules to link nodes: 1) A TABLE node, representing the
whole table, should be linked to all CELL nodes to encap-
sulate the global semantics. 2) A ROW node derives infor-
mation from cells within the same row, so the ROW node ri
should be linked to each CELL node ci∗. 3) A strong corre-
lation exists between the semantics of Data CELLs and their
corresponding Header CELLs within the same column, so
they should be linked. 4) Data CELLs located in the same
column exhibit a stronger relational bond compared to those
in different columns. Consequently, adjacent Data CELLs
within a column are interconnected. 5) All Header CELLs
should be interconnected because the interpretation of re-
lationships between Data CELLs across columns necessi-
tates the semantic understanding of their respective Header
CELLs. For instance, as demonstrated in Fig. 3, determin-
ing the relationship between cell c2,0 and cell c2,1 requires
an examination of their Header CELLs. This examination
reveals that “53,196,521.18” represents the income from the
“main business” project.
Edge types are categorized based on the nodes they link,
as follows: Table-Header, Table-Data, Header-Row, Data-
Row, Header-Data, Data-Data, and Header-Header edges.
4.2
Stage 1: Self-supervised Instruction Tuning
Illustrated in Fig. 2, HeGTa utilizes the tabular output of the
HG encoder as soft prompts (Li and Liang 2021), which are
part of the LLM input. The weights of both modules are
tuned through self-supervised instruction tuning to align the
vector representation spaces of the two modules. This sub-
section provides details of the pre-training process, which
involves three different granularity self-supervised tasks.
Tabular HG Encoder
Following the conversion of the ta-
ble into a tabular HG, HeGTa introduces a heterogeneous
GNN (HGNN) (Hu et al. 2020) as the encoder for tabular
HG data. The encoder takes a tabular HG as input and gen-
erates vector representations for the tabular nodes as output.
The HGNN employs a message-passing mechanism to col-
lect semantic and topological information from neighboring
cell nodes within each layer, considering various edge types.
Multi-granularity
Self-supervised
Instruction
Tasks
Drawing inspiration from the methodologies in LLaVA (Liu
et
al.
2023),
Time-LLM
(Jin
et
al.
2023a),
and
GraphGPT (Tang et al. 2023), we develop a method to
effectively align the vector spaces of two distinct modalities
of data: tables and NLs. We introduce the soft prompt
technique as a bridge between the two encoders and focus
on fine-tuning the tabular graph encoder and the LoRA (Hu
et al. 2021) module within the LLM. This process, designed
to be lightweight, enables the LLM to grasp the topological
information of tables through semantic instruction tuning.
Instruction tuning (Wang et al. 2022; Ouyang et al. 2022),
a technique that merges fine-tuning with prompt learn-
ing, significantly enhances the generalization capabilities of
LLMs. Therefore, before the model-tuning process, we pre-
format the self-supervised training data into an instruction-
based QA format. Examples of training data for instruction
tuning are illustrated in Fig. 4.
To align the LLM with the HG encoder, the HeGTa
incorporates additional tokens into the vocabulary: <tab-
ular node>, <table start> and <table end>. The token
<tabular node> serves as a placeholder for table tasks
within the instruction text, allowing for the substitution with
Large Language Model
Other Layers
Large Language Model
Embedding Layer
...
...
...
Raw Token Embeddings

Tabular
HG
Encoder
Tabular HG
Table
Tuned
Frozen
...
...
...
Replaced Token Embeddings
Input
[Instruction Texts]
[Instruction Texts]
<Table Token Placeholders>
Stage 2: Fine-tuning
Few-shot Specific Tasks
（CTC, TTC, TQA)
Stage 1: Self-supervised Pre-Training
Task:
Table Row
Classification
Task:
Table Cell
Matching
Task:
Table Context
Generation
LoRA Module
Figure 2: An overview of HeGTa framework. HeGTa processes <table, instruction> as an input. First, the table is converted
into an HG and processed by a Tabular HG encoder to generate a vector for each tabular node, while the LLM transforms
instruction texts into initial token embeddings. Subsequently, the HG encoder’s outputs serve as soft prompts for the LLM,
enabling the replacement of placeholder embeddings with actual tabular node vectors. The modified embedding sequence is
then processed by the remaining LLM layers. Throughout Stage 1 and Stage 2, only the weights of red components are tuned.
actual table node vectors post-processing by the LLM’s Em-
bedding layer. The quantity of these placeholders matches
the number of table nodes relevant to the current task. The
tokens <table start> and <table end> signify the begin-
ning and ending delimiters of the table placeholders.
The forward propagation in HeGTa begins with the input
of a <Table, Instruction> pair, where the instruction text
passes through the LLM’s embedding layer, and the table is
transformed into a Tabular HG before being processed by
the HGNN. The LLM embedding layer assigns embeddings
to each token, creating a sequence, while the HGNN’s out-
put provides aggregated vector representations for each table
node. Subsequently, HeGTa replaces the embeddings corre-
sponding to the table token placeholders within the sequence
with the actual node vectors. This adjusted embedding se-
quence is then fed into the LLM’s remaining layers.
To enhance the HeGTa’s understanding of tables, we in-
troduce three tasks tailored for the graph-enhanced LLM,
each varying in granularity: Table Row Classification, Ta-
ble Cell Matching, and Table Context Generation. These
tasks are designed to integrate table representations into the
LLM’s semantic space. The complexity of these tasks in-
creases progressively, aiming to incrementally improve the
model’s ability to comprehend semantic information. De-
tailed descriptions of each task are provided below.
Table Row Classification (TRC). The objective of this task
is to train the model to accurately identify the category
of each table row. Utilizing the vector representations of
Row nodes provided by the HG encoder, the model dis-
cerns whether a node corresponds to a Header Row or a Data
Row. The categorization of Row nodes is automated by an
algorithm elaborated in Appendix, eliminating the need for
manual intervention and thus qualifying as a self-supervised
training process. In the dataset we utilize, the labeling accu-
racy of the algorithm reaches 97.6%. This task facilitates the
model’s initial grasp of coarse-grained information about the
structure of table rows.
Table Cell Matching (TCM). The task involves supply-
ing the model with vector representations of each cell node
alongside a list of shuffled cell contents. The model’s objec-
tive is to correctly pair each cell node vector with its cor-
responding original text. For instance, referring to the table
depicted in Fig. 3, the model needs to align the node vec-
tor of c1,1 with the string ”income” within the list correctly.
This training task enables the model to discern the seman-
tic information encapsulated within the cell contents based
on the graph node vectors. Essentially, it aligns the semantic
space of the HG encoder and the LLM.
Table Context Generation (TCG). This task aims to enable
the model to generate context surrounding table data, utiliz-
ing the vector representations from the Table node. This task
facilitates the model’s learning of the table’s global semantic
information, proving beneficial for tasks requiring a compre-
hensive understanding of the table as an entity.
Examples of datasets in instruction format for the three
tasks are provided in Fig. 4.
2
1
0
Amount incurred
in the current period
project
0
cost
income
1
36,829,903.92
53,196,521.18
Main business
2
2,591,042.45
4,868,204.64
Other businesses
3
0,0
c
0,1
c
1,1
c
1,2
c
2,0
c
3,0
c
2,1
c
2,2
c
3,1
c
3,2
c
0r
1r
2r
3r
t
c
Figure 3: Table-to-heterogeneous graph conversion. Node types are color-coded: Table (green), Row (red), Data Cell (blue),
and Header Cell (yellow). Edge types are similarly color-coded, with bidirectional edges shown as undirected lines. Some edges
are omitted for clarity.
Q: Given a sequence of table CELL tokens <table_start> <tabular_node>...< tabular_node> <table_end> that constitute a table. Each
table CELL token contains the content within the CELL. The following is a list of CELL contents:
0) "income", 1) "Float Value: 36,829,903.92", 2) "Other businesses", … , 9) "cost". Please reorder the list of CELLs according to the
order of table CELL tokens (i.e., complete the matching of table CELL tokens and CELL contents).
A: Based on the given 8 table CELL tokens and the list of table CELL contents, we obtain the matching of table CELL tokens and contents
as follows: Table CELL token 0 corresponds to "project" … Table CELL token 9 corresponds to "Float Value: 2,591,042.45".
Q: Given a sequence of table ROW tokens <table_start> <tabular_node>...< tabular_node> <table_end> that constitute a table. Each
table has two ROW types: Header-Row and Data-Row. Please classify each table ROW according to the table ROW tokens.
A: Based on the given 4 table ROW tokens, we can classify them as follows: Table ROW token 0 corresponds to Header-Row. Table ROW
token 1 corresponds to Header-Row. Table ROW token 2 corresponds to Data-Row. Table ROW token 3 corresponds to Data-Row.
Q: Given a tabular TABLE token <table_start> <tabular_node> <table_end> that represents a table. The TABLE token contains the
content within the table. Please generate the context according to the tabular TABLE token.
A: Based on the given TABLE CELL token, the context about the table is: The income and cost of the main business of the company's
current amount is "53,196,521.18", "36,829,903.92"
Figure 4: Examples of three self-supervised instruction tuning datasets, each tailored for distinct tasks: Table Row Classification,
Table Cell Matching, and Table Context Generation.
4.3
Stage 2: Task-specific Instruction Fine-tuning
Following the completion of the self-supervised pre-training
tasks in Stage 1, the HeGTa has successfully aligned the rep-
resentation spaces of the HG encoder and the LLM. This
alignment, combined with the intrinsic capabilities of the
LLM, empowers HeGTa to effectively comprehend the topo-
logical nuances of complex tables. Consequently, when ap-
plied to a specific downstream task (i.e., CTC, TTC, TQA),
HeGTa requires only a small number of training samples to
grasp the expected answer format for the downstream task
and reorganize its pre-existing knowledge into the suitable
output format. Example prompts of downstream tasks can
be found in Appendix. When fine-tuning in Stage 2, the
model adopts the same tuning strategy as in Stage 1, with all
parameters frozen except for those of the HG Encoder. To
enhance the reasoning capabilities of HeGTa, we annotated
the reasoning process with Chain of Thought (CoT) (Wei
et al. 2022) for k-shot TQA samples. Inspired by Auto-
CoT (Zhang et al. 2022), we first clustered the table samples
in the test set, and then clustered the corresponding ques-
tions within each table cluster. Finally, we selected the cen-
tral TQA samples from each question cluster for annotation.
This approach ensures the diversity of contextual examples
with a limited number of annotations.
5
Experiments
5.1
Datasets
To evaluate HeGTa, we selected a variety of datasets that are
widely studied and easy to parse, including WCC (Ghasemi-
Gol and Szekely 2018), IM-TQA (Zheng et al. 2023),
HiTab (Cheng et al. 2021b), WTQ (Raw) along with its
header-flatten version WTQ (Flatten) (Pasupat and Liang
2015) and TabFact (Chen et al. 2020), each pertinent to
CTC, TTC, or TQA, respectively.
Statistics for these datasets are presented in Table 1, illus-
trating the types of annotations, the primary domains cov-
ered, and the proportion of complex tables. Given the focus
on few-shot TU scenarios, we only list the size of test sets.
Comprehensive details of the dataset annotations and pre-
processing are available in Appendix.
5.2
Baselines
We compare HeGTa with several strong baselines to ver-
ify its effectiveness. These baselines can be categorized into
four groups according to their frameworks. ForTap (Cheng
et al. 2021a) and GetPt (Jia et al. 2023) emulate BERT, de-
vising specific pretext tasks tailored for tabular data to pre-
train the Transformer encoder (Devlin et al. 2019). Tabular-
Net (Du et al. 2021) and TabPrompt (Jin et al. 2023b) in-
corporate a graph encoder. Binder (Cheng et al. 2022), Aug-
Codex (Cao et al. 2023) and COTable (Wang et al. 2024)
leverage LLMs to generate symbolic programs, which are
then executed to obtain the final answer. Compared to sym-
bolic methods, TableLlama (Zhang et al. 2023a) and Table-
CoT (Chen 2022) leverage the textual understanding capa-
bilities of LLMs to directly address tabular tasks.
Datasets
Annotation Type
Table Info
CTC
TTC
TQA
# Test Tables
% Complex Tables
# QA pairs
Main Domains
IM-TQA
✓
✓
✓
153
47.71
627
Manufacturing
WCC
✓
371
–
–
General
HiTab
✓
✓
3597
92.88
1584
Crime, Health
WTQ (Flatten)
✓
2108
0.00
4344
General
WTQ (Raw)
✓
2108
14.80
4344
General
TabFact
✓
1695
0.00
12779
General
Table 1: Dataset Statistics. “✓” indicates the type of annotation in the dataset that has this task in it. “# Test Tables”, “%
Complex Tables” and “# QA pairs” columns show the number of tables in the test set, the percentage of complex tables and the
numbers of QA pairs, respectively.
Model Type
Models
CTC
TTC
TQA
HiTab IM–TQA WCC IM–TQA WTQ (Flat) WTQ (Raw) TabFact HiTab IM–TQA
Bert-like
ForTap
56.33
47.34
46.14
53.40
27.33
26.52
28.67
24.88
29.08
Encoder-based GetPt
58.45
49.67
48.62
56.45
23.69
23.34
25.03
23.34
22.07
Graph-based
TabularNet
54.14
45.62
46.01
56.41
19.18
18.58
20.81
22.34
20.05
TabPrompt
63.56
46.77
50.31
54.34
17.71
14.77
19.56
20.20
14.14
LLM-based
(Symbolic)
Binder
–
–
–
–
43.16
40.05
59.04
44.11
41.25
Aug-Codex
–
–
–
–
33.32
32.28
57.07
45.87
42.16
COTable
–
–
–
–
44.38
42.97
59.76
49.68
43.06
LLM-based
(Textual)
TableCoT
53.12
49.76
45.57
53.80
31.42
32.12
52.01
47.55
45.21
TableLlama 60.15
53.40
51.56
57.55
38.88
39.50
–
–
48.82
HeGTa
66.39
59.58
55.32
58.46
45.12
45.96
60.61
51.56
50.73
Table 2: Overall evaluation results on three TU tasks with best bolded and runner-up underlined. ‘–’ indicates that the current
framework cannot handle the current task due to some limitations. 1
5.3
Implementation Details
General Setup Across All Tasks. We employ Vicuna-7B-
v1.5 (Chiang et al. 2023) as the base LLM and a 2-layer Het-
erogeneous Graph Transformer (HGT) (Wang et al. 2020a)
as our tabular HG encoder whose hidden dimension is set
as 1024. The initial vectors of nodes in HG are obtained by
S-BERT (Reimers and Gurevych 2019), whose dimension is
768. We integrate a LoRA (Hu et al. 2021) module to the
embedding layer of the LLM. At any tuning stage, HeGTa
exclusively tunes the parameters of the HGT and the LoRA
module. This lightweight configuration allows our model to
be trained on a single 4090 GPU. We assemble the tables
from training and validation sets of the datasets mentioned
above along with a selection of TURL (Deng et al. 2020)
corpus as a comprehensive dataset consisting of 200k tables
for use in the pre-training stage.
General Setup across Evaluation Tasks. Following a typ-
ical k-shot setup (Liu et al. 2021; Wang et al. 2020b) to
validate the HeGTa’s performance on few-shot TU, we ran-
domly generate five 8-shot tasks for both training and val-
idation on each dataset (i.e., 8 train/dev tables for CTC, 8
1For a fair comparison, all LLM-based methods use the open-
source LLaMA2-7B (Touvron et al. 2023) as the underlying model.
More details of baselines are shown in Appendix.
train/dev tables per class for TTC and 8 train/dev QA pairs
for TQA). Each training task, paired with a corresponding
validation task, is utilized to fine-tune the models optimally
for subsequent testing. Table 2 presents the average perfor-
mance on the test set across the five tasks.
Evaluation Metrics. We adopt Macro-F1, Macro-F1 and
accuracy as the evaluation metric for CTC, TTC and TQA,
respectively. Given the generative structure of LLMs, re-
lying solely on exact matches might inaccurately catego-
rize some correct responses (e.g., adding an answer with
an additional period at the end). Therefore, we incorporate
the semantic-match evaluator (Cheng et al. 2022) for TQA,
which preprocesses the outputs prior to evaluation.
5.4
Results and Analysis
Table 2 compares the performance of different methods on
all datasets described in § 5.1. From the results, We have the
following observations: 1) HeGTa outperformed all base-
lines across 9 datasets, demonstrating its superiority over
both non-LLM methods and LLMs with equivalent param-
eters. Additionally, as shown in Appendix, HeGTa demon-
strates competitive performance in embedding-related tasks
(i.e., CTC and TTC) even compared to SOTA GPT-4. 2)
LLM-based symbolic methods excel in TQA but are inher-
Module
CTC
TTC
TQA
HiTab
IM–TQA
WCC
IM–TQA
WTQ (Flat)
WTQ (Raw)
TabFact
HiTab
IM–TQA
w/o TRC
-2.65
-1.53
+1.02
-1.40
-1.10
-1.34
-0.89
-0.78
+1.19
w/o TCM
-4.50
-4.88
-3.22
-2.44
-2.31
-2.71
-2.10
-1.67
-2.45
w/o TCG
+0.11
+1.19
-1.61
-1.73
-2.21
-2.54
-2.57
-0.30
-2.58
w/o HG
-3.28
-1.61
-0.24
-0.60
-1.80
-1.75
-1.68
-1.01
-1.31
w/o hl
-2.25
+0.90
+0.42
-0.75
-0.74
-0.85
-0.79
-0.20
+0.56
w/o CoT
–
–
–
–
-2.31
-2.35
-2.57
-1.98
-1.31
Table 3: Ablation results on all datasets. “w/o TRC”, “w/o TCM”, and “w/o TCG”: HeGTa pre-trained without the specified
objective. “w/o HG”: HeGTa pre-trained with homogeneous graphs. “w/o hl”: HeGTa pre-trained with linking all neighboring
cells directly instead of the heuristic linking strategy.
ently incapable of handling other table tasks due to their ar-
chitectural design. Moreover, methods with 7B parameters
struggle with reasoning and generating precise, executable
code compared to those based on ChatGPT. For further de-
tails, see Appendix. 3) WTQ (Raw) represents the unpro-
cessed version of WTQ (Flat), preserving the complexity of
hierarchical table headers. The significant performance drop
of other methods on WTQ (Raw) highlights HeGTa’s abil-
ity to effectively handle relational tables with complex struc-
tures. 4) Compared to HeGTa, other LLM-based frameworks
underperform due to the loss of table topological informa-
tion, even with enhancements like TableLlama’s extensive
training on linearized tables or TableCoT’s use of HTML, a
format better suited for LLMs (Sui et al. 2024). This under-
scores the effectiveness of introducing GNNs in overcoming
the limitations of linearized table representations, thereby
unlocking the full potential of LLMs.
Strengths and Weaknesses of LLM-Based Models
All
QA samples of HiTab can be classified into lookup-based
reasoning (L-R) and computation-based reasoning (C-R)
(e.g., sum, max, count). Table 4 presents the performance
of three LLM-based methods on two types of QA samples.
TableCoT shows superior performance in L-R compared to
C-R, aligning with the conclusion (Chen et al. 2022) that
LLMs struggle with numerical calculations. COTable lever-
ages SQL to mitigate the shortcomings of LLMs in numeri-
cal reasoning, leading to the highest accuracy in C-R. How-
ever, the hierarchical structure causes a decline in SQL per-
formance. HeGTa addresses the issue of structural informa-
tion loss of linearized tables. As a result, HeGTa achieves
notable improvements, particularly in L-R samples. Addi-
tionally, without restricting to the few-shot setting, i.e., tun-
ing the model with the entire training set, HeGTa consis-
tently maintains top performance, as shown in Appendix.
5.5
Ablation Study
We conduct ablation experiments to evaluate each compo-
nent’s effectiveness, as shown in Table 3.
We begin by examining the individual improvements con-
tributed by each of the three self-training tasks. To evaluate
their specific impact, we pre-train three models, each omit-
ting one of the objectives: TRC, TCM, or TCG. These three
pre-training objectives improve the LLM’s understanding
L-R
C-R
Total
#
%
#
%
#
%
# Test set
1133 100.00 451 100.00 1584 100.00
TableCoT
612
54.02
195
43.24
806
47.55
COTable
578
51.02
265
58.76
842
49.68
HeGTa
669
59.05
206
45.68
874
51.56
Table 4: Further results of the three methods on HiTab. “#”
represents the number of correctly answered samples, and
“%” denotes the proportion of correctly answered samples.
of tabular structure. The TCM task most effectively aligns
the tabular modality with the LLM’s semantic space. The
TRC task enhances performance on relational tables, while
the TCG task boosts TQA performance but slightly reduces
CTC performance.
We conducted ablation studies on CoT and our tabular
graph construction method by removing the CoT text, con-
verting tables into homogeneous rather than heterogeneous
graphs, and replacing the heuristic linking (hl) strategy with
a naive rule that links all neighboring cells. The results
clearly demonstrate the significant impact of CoT on TQA
tasks. The introduction of HGs benefits all tasks, with partic-
ularly notable improvements in TU with hierarchical struc-
tures. However, compared to the benefits of HGs, the hl strat-
egy shows less pronounced gains for non-relational tables,
possibly because the naive rule is more effective for aggre-
gating and transmitting information in these tables.
6
Conclusion
In this paper, we introduce a novel framework, HeGTa, tai-
lored for few-shot complex TU. The effectiveness of HeGTa
is validated across multiple tabular datasets, accompanied
by an in-depth ablation study to examine the impact of each
component. In future work, we plan to expand HeGTa’s ap-
plicability to tables featuring more diverse layouts and to
further improve HeGTa’s performance in TQA tasks by in-
tegrating the program-aided paradigm into our approach.
7
Acknowledgments
This work is partially supported by National Nature Science
Foundation of China under No. 62476058, by the project
”Key Laboratory of rich-media Digital Publishing Content
Organization and Knowledge Service Open Fund-Research
on Knowledge-enhanced Training Techniques of Large Lan-
guage Model” No. ZD2024-04/01. We thank the Big Data
Computing Center of Southeast University for providing the
facility support on the numerical calculations in this paper.
References
Cao, Y.; Chen, S.; Liu, R.; Wang, Z.; and Fried, D. 2023.
API-Assisted Code Generation for Question Answering on
Varied Table Structures. ArXiv, abs/2310.14687.
Chen, W. 2022. Large Language Models are few(1)-shot
Table Reasoners. ArXiv, abs/2210.06710.
Chen, W.; Ma, X.; Wang, X.; and Cohen, W. W. 2022.
Program of thoughts prompting: Disentangling computa-
tion from reasoning for numerical reasoning tasks. arXiv
preprint arXiv:2211.12588.
Chen, W.; Wang, H.; Chen, J.; Zhang, Y.; Wang, H.; Li, S.;
Zhou, X.; and Wang, W. Y. 2020. TabFact: A Large-scale
Dataset for Table-based Fact Verification. In International
Conference on Learning Representations.
Cheng, Z.; Dong, H.; Cheng, F.; Jia, R.; Wu, P.; Han, S.; and
Zhang, D. 2021a. FORTAP: Using Formulas for Numerical-
Reasoning-Aware Table Pretraining. In Annual Meeting of
the Association for Computational Linguistics.
Cheng, Z.; Dong, H.; Wang, Z.; Jia, R.; Guo, J.; Gao, Y.;
Han, S.; Lou, J.-G.; and Zhang, D. 2021b. HiTab: A Hier-
archical Table Dataset for Question Answering and Natural
Language Generation. ArXiv, abs/2108.06712.
Cheng, Z.; Xie, T.; Shi, P.; Li, C.; Nadkarni, R.; Hu, Y.;
Xiong, C.; Radev, D. R.; Ostendorf, M.; Zettlemoyer, L.;
Smith, N. A.; and Yu, T. 2022. Binding Language Models
in Symbolic Languages. ArXiv, abs/2210.02875.
Chiang, W.-L.; Li, Z.; Lin, Z.; Sheng, Y.; Wu, Z.; Zhang, H.;
Zheng, L.; Zhuang, S.; Zhuang, Y.; Gonzalez, J. E.; et al.
2023.
Vicuna: An open-source chatbot impressing gpt-4
with 90%* chatgpt quality. See https://vicuna. lmsys. org
(accessed 14 April 2023).
Deng, X.; Sun, H.; Lees, A.; Wu, Y.; and Yu, C. 2020.
TURL: Table Understanding through Representation Learn-
ing. SIGMOD Rec., 51: 33–40.
Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019.
BERT: Pre-training of Deep Bidirectional Transformers for
Language Understanding. ArXiv, abs/1810.04805.
Dong, H.; Cheng, Z.; He, X.; Zhou, M.; Zhou, A.; Zhou, F.;
Liu, A.; Han, S.; and Zhang, D. 2022. Table pre-training: A
survey on model architectures, pre-training objectives, and
downstream tasks. arXiv preprint arXiv:2201.09745.
Du, L.; Gao, F.; Chen, X.; Jia, R.; Wang, J.; Han, S.; and
Zhang, D. 2021.
TabularNet: A Neural Network Archi-
tecture for Understanding Semantic Structures of Tabular
Data. Proceedings of the 27th ACM SIGKDD Conference
on Knowledge Discovery & Data Mining.
Ghasemi-Gol, M.; and Szekely, P. A. 2018.
TabVec:
Table Vectors for Classification of Web Tables.
ArXiv,
abs/1802.06290.
Herzig, J.; Nowak, P. K.; M¨uller, T.; Piccinno, F.; and Eisen-
schlos, J. M. 2020. TaPas: Weakly Supervised Table Parsing
via Pre-training. ArXiv, abs/2004.02349.
Hu, E. J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang,
S.; Wang, L.; and Chen, W. 2021. Lora: Low-rank adaptation
of large language models. arXiv preprint arXiv:2106.09685.
Hu, Z.; Dong, Y.; Wang, K.; and Sun, Y. 2020. Heteroge-
neous graph transformer. In Proceedings of the web confer-
ence 2020, 2704–2710.
Jia, R.; Guo, H.; Jin, X.; Yan, C.; Du, L.; Ma, X.; Stankovic,
T.; Lozajic, M.; Zoranovic, G.; Ilic, I.; Han, S.; and Zhang,
D. 2023. GetPt: Graph-enhanced General Table Pre-training
with Alternate Attention Network. Proceedings of the 29th
ACM SIGKDD Conference on Knowledge Discovery and
Data Mining.
Jin, M.; Wang, S.; Ma, L.; Chu, Z.; Zhang, J. Y.; Shi, X. L.;
Chen, P.-Y.; Liang, Y.; Li, Y.-F.; Pan, S.; and Wen, Q. 2023a.
Time-LLM: Time Series Forecasting by Reprogramming
Large Language Models. ArXiv, abs/2310.01728.
Jin, R.; Wang, J.; Tan, W.; Chen, Y.; Qi, G.; and Hao, W.
2023b. TabPrompt: Graph-based Pre-training and Prompt-
ing for Few-shot Table Understanding. In Conference on
Empirical Methods in Natural Language Processing.
Li, X. L.; and Liang, P. 2021.
Prefix-Tuning: Optimiz-
ing Continuous Prompts for Generation.
Proceedings of
the 59th Annual Meeting of the Association for Computa-
tional Linguistics and the 11th International Joint Confer-
ence on Natural Language Processing (Volume 1: Long Pa-
pers), abs/2101.00190.
Liu, H.; Li, C.; Wu, Q.; and Lee, Y. J. 2023. Visual Instruc-
tion Tuning. ArXiv, abs/2304.08485.
Liu, Z.; Fang, Y.; Liu, C.; and Hoi, S. C. 2021. Relative
and absolute location embedding for few-shot node classifi-
cation on graph. In Proceedings of the AAAI conference on
artificial intelligence, 4267–4275.
Lu, W.; Zhang, J.; Zhang, J.; and Chen, Y. 2024. Large lan-
guage model for table processing: A survey. arXiv preprint
arXiv:2402.05121.
Ma, S.; Liu, J.-w.; and Zuo, X. 2023. Self-supervised learn-
ing for heterogeneous graph via structure information based
on metapath. Applied Soft Computing, 143: 110388.
Ouyang, L.; Wu, J.; Jiang, X.; Almeida, D.; Wainwright,
C. L.; Mishkin, P.; Zhang, C.; Agarwal, S.; Slama, K.; Ray,
A.; Schulman, J.; Hilton, J.; Kelton, F.; Miller, L. E.; Simens,
M.; Askell, A.; Welinder, P.; Christiano, P. F.; Leike, J.; and
Lowe, R. J. 2022. Training language models to follow in-
structions with human feedback. ArXiv, abs/2203.02155.
Pasupat, P.; and Liang, P. 2015. Compositional Semantic
Parsing on Semi-Structured Tables. In Annual Meeting of
the Association for Computational Linguistics.
Reimers, N.; and Gurevych, I. 2019. Sentence-BERT: Sen-
tence Embeddings using Siamese BERT-Networks. In Pro-
ceedings of the 2019 Conference on Empirical Methods in
Natural Language Processing. Association for Computa-
tional Linguistics.
Sui, Y.; Zhou, M.; Zhou, M.; Han, S.; and Zhang, D. 2024.
Table meets llm: Can large language models understand
structured table data? a benchmark and empirical study. In
Proceedings of the 17th ACM International Conference on
Web Search and Data Mining, 645–654.
Tang, J.; Yang, Y.; Wei, W.; Shi, L.; Su, L.; Cheng, S.; Yin,
D.; and Huang, C. 2023. GraphGPT: Graph Instruction Tun-
ing for Large Language Models. ArXiv, abs/2310.13023.
Touvron, H.; Martin, L.; Stone, K. R.; Albert, P.; Alma-
hairi, A.; Babaei, Y.; Bashlykov, N.; Batra, S.; Bhargava,
P.; Bhosale, S.; Bikel, D. M.; Blecher, L.; Ferrer, C. C.;
Chen, M.; Cucurull, G.; Esiobu, D.; Fernandes, J.; Fu, J.; Fu,
W.; Fuller, B.; Gao, C.; Goswami, V.; Goyal, N.; Hartshorn,
A. S.; Hosseini, S.; Hou, R.; Inan, H.; Kardas, M.; Kerkez,
V.; Khabsa, M.; Kloumann, I. M.; Korenev, A. V.; Koura,
P. S.; Lachaux, M.-A.; Lavril, T.; Lee, J.; Liskovich, D.; Lu,
Y.; Mao, Y.; Martinet, X.; Mihaylov, T.; Mishra, P.; Moly-
bog, I.; Nie, Y.; Poulton, A.; Reizenstein, J.; Rungta, R.;
Saladi, K.; Schelten, A.; Silva, R.; Smith, E. M.; Subrama-
nian, R.; Tan, X.; Tang, B.; Taylor, R.; Williams, A.; Kuan,
J. X.; Xu, P.; Yan, Z.; Zarov, I.; Zhang, Y.; Fan, A.; Kam-
badur, M.; Narang, S.; Rodriguez, A.; Stojnic, R.; Edunov,
S.; and Scialom, T. 2023. Llama 2: Open Foundation and
Fine-Tuned Chat Models. ArXiv, abs/2307.09288.
Wang, F.; Sun, K.; Chen, M.; Pujara, J.; and Szekely, P. 2021.
Retrieving complex tables with multi-granular graph repre-
sentation learning. In Proceedings of the 44th International
ACM SIGIR Conference on Research and Development in
Information Retrieval, 1472–1482.
Wang, K.; Shen, W.; Yang, Y.; Quan, X.; and Wang, R.
2020a.
Relational Graph Attention Network for Aspect-
based Sentiment Analysis. In ACL.
Wang, N.; Luo, M.; Ding, K.; Zhang, L.; Li, J.; and Zheng,
Q. 2020b. Graph few-shot learning with attribute matching.
In Proceedings of the 29th ACM International Conference
on Information & Knowledge Management, 1545–1554.
Wang, Y.; Kordi, Y.; Mishra, S.; Liu, A.; Smith, N. A.;
Khashabi, D.; and Hajishirzi, H. 2022. Self-Instruct: Align-
ing Language Models with Self-Generated Instructions. In
Annual Meeting of the Association for Computational Lin-
guistics.
Wang, Z.; Dong, H.; Jia, R.; Li, J.; Fu, Z.; Han, S.; and
Zhang, D. 2020c. TUTA: Tree-based Transformers for Gen-
erally Structured Table Pre-training.
Proceedings of the
27th ACM SIGKDD Conference on Knowledge Discovery
& Data Mining.
Wang, Z.; Zhang, H.; Li, C.-L.; Eisenschlos, J. M.; Perot,
V.; Wang, Z.; Miculicich, L.; Fujii, Y.; Shang, J.; Lee, C.-
Y.; et al. 2024.
Chain-of-table: Evolving tables in the
reasoning chain for table understanding.
arXiv preprint
arXiv:2401.04398.
Wei, J.; Wang, X.; Schuurmans, D.; Bosma, M.; Xia, F.;
Chi, E.; Le, Q. V.; Zhou, D.; et al. 2022.
Chain-of-
thought prompting elicits reasoning in large language mod-
els. Advances in neural information processing systems, 35:
24824–24837.
Yang, Y.; Guan, Z.; Wang, Z.; Zhao, W.; Xu, C.; Lu, W.; and
Huang, J. 2022. Self-supervised heterogeneous graph pre-
training based on structural clustering. Advances in Neural
Information Processing Systems, 35: 16962–16974.
Ye, Y.; Hui, B.; Yang, M.; Li, B.; Huang, F.; and Li, Y.
2023.
Large language models are versatile decomposers:
Decomposing evidence and questions for table-based rea-
soning. In Proceedings of the 46th International ACM SI-
GIR Conference on Research and Development in Informa-
tion Retrieval, 174–184.
Yin, P.; Neubig, G.; tau Yih, W.; and Riedel, S. 2020.
TaBERT: Pretraining for Joint Understanding of Textual and
Tabular Data. ArXiv, abs/2005.08314.
Zhang, T.; Yue, X.; Li, Y.; and Sun, H. 2023a. TableLlama:
Towards Open Large Generalist Models for Tables. ArXiv,
abs/2311.09206.
Zhang, Y.; Henkel, J.; Floratou, A.; Cahoon, J.; Deep, S.;
and Patel, J. M. 2023b. ReAcTable: Enhancing ReAct for
Table Question Answering. ArXiv, abs/2310.00815.
Zhang, Z.; Zhang, A.; Li, M.; and Smola, A. 2022. Auto-
matic chain of thought prompting in large language models.
arXiv preprint arXiv:2210.03493.
Zheng, M.; Hao, Y.; Jiang, W.-J.; Lin, Z.; Lyu, Y.; She,
Q.; and Wang, W. 2023. IM-TQA: A Chinese Table Ques-
tion Answering Dataset with Implicit and Multi-type Table
Structures. In Annual Meeting of the Association for Com-
putational Linguistics.


Question:
What is the primary advantage of the MATEval framework over traditional LLM evaluation methods in assessing open-ended text generation?
A. It uses multiple LLMs to evaluate the text, ensuring unbiased results.
B. It employs a multi-agent discussion system that integrates self-reflection and chain-of-thought strategies, leading to more accurate and stable evaluations.
C. It employs a multi-agent discussion system that integrates self-reflection and chain-of-thought strategies, eliminating the need for human intervention.
D. It employs multiple LLM agents with self-reflection and Chain-of-Thought reasoning, enabling it to assess semantic coherence and logical flow beyond surface-level metrics.

Answer: B

Recent advancements in generative Large Language Models (LLMs) have been remarkable, however, the quality of the text generated by these models often reveals persistent issues. Evaluating the quality of text generated by these models, especially in open-ended text, has consistently presented a significant challenge. Addressing this, recent work has explored the possibility of using LLMs as evaluators. While using a single LLM as an evaluation agent shows potential, it is filled with significant uncertainty and instability. To address these issues, we propose the MATEval: A “Multi-Agent Text Evaluation framework” where all agents are played by LLMs like GPT-4. The MATEval framework emulates human collaborative discussion methods, integrating multiple agents’ interactions to evaluate open-ended text. Our framework incorporates self-reflection and Chain-of-Thought (CoT) strategies, along with feedback mechanisms, enhancing the depth and breadth of the evaluation process and guiding discussions towards consensus, while the framework generates comprehensive evaluation reports, including error localization, error types and scoring. Experimental results show that our framework outperforms existing open-ended text evaluation methods and achieves the highest correlation with human evaluation, which confirms the effectiveness and advancement of our framework in addressing the uncertainties and instabilities in evaluating LLMs-generated text. Furthermore, our framework significantly improves the efficiency of text evaluation and model iteration in industrial scenarios.

MATEval: A Multi-Agent Discussion Framework
for Advancing Open-Ended Text Evaluation
Yu Li1,⋆, Shenyu Zhang1,⋆, Rui Wu2, Xiutian Huang2, Yongrui Chen1,
Wenhao Xu2(), Guilin Qi1 (), and Dehai Min1
1 Southeast University, Nanjing, China
{yuli 11, shenyuzhang, yrchen, gqi, zhishanq}@seu.edu.cn
2 Ant Group, Hangzhou, China
{guli.wr, xiutian.hxt, hao.xuwh}@antgroup.com
Abstract. Recent advancements in generative Large Language Models
(LLMs) have been remarkable, however, the quality of the text generated
by these models often reveals persistent issues. Evaluating the quality of
text generated by these models, especially in open-ended text, has con-
sistently presented a significant challenge. Addressing this, recent work
has explored the possibility of using LLMs as evaluators. While using a
single LLM as an evaluation agent shows potential, it is filled with sig-
nificant uncertainty and instability. To address these issues, we propose
the MATEval: A “Multi-Agent Text Evaluation framework” where
all agents are played by LLMs like GPT-4. The MATEval framework
emulates human collaborative discussion methods, integrating multiple
agents’ interactions to evaluate open-ended text. Our framework incorpo-
rates self-reflection and Chain-of-Thought (CoT) strategies, along with
feedback mechanisms, enhancing the depth and breadth of the evaluation
process and guiding discussions towards consensus, while the framework
generates comprehensive evaluation reports, including error localization,
error types and scoring. Experimental results show that our framework
outperforms existing open-ended text evaluation methods and achieves
the highest correlation with human evaluation, which confirms the effec-
tiveness and advancement of our framework in addressing the uncertain-
ties and instabilities in evaluating LLMs-generated text. Furthermore,
our framework significantly improves the efficiency of text evaluation
and model iteration in industrial scenarios.
Keywords: Multi-Agent · Large Language Models · Text Evaluation
1
Introduction
Evaluating the text generated by large language models (LLMs) has long been
a challenging task, Traditional manual evaluation methods are not only time-
consuming and laborious but also expensive [2]. Although methods like BLEU
[14], Rouge [10], and METEOR [1] have achieved success in scenarios such as
⋆Equal Contributors.
arXiv:2403.19305v2  [cs.CL]  15 Apr 2024
2
Y. Li et al.
machine translation, these automated evaluation methods are limited in the
context of open-ended text generation [11]. Recently, LLMs have been used as
evaluators such as G-Eval [4], but these methods exhibit unstable and uncer-
tain evaluation effects [15] [16]. Even certain collaboration frameworks could
alleviate this problem by employing multi-agent discussion, e.g. ChatEval [3],
however, the current methods of multi-agent collaboration remain limited to
simple interactions, without fully harnessing the potential for agents’ thinking
and planning. Additionally, reaching a consensus within multi-agent discussion
frameworks continues to be a challenging issue. Furthermore, traditional text
evaluation models typically provide only a score without explaining it, making
it difficult for reviewers to trust the reliability of these scores. They still need
to manually identify errors, obviously slowing down the collection of bad cases
and, consequently, affecting the pace of model iteration in industrial scenarios.
To address the above challenges, this paper introduces a Multi-Agent Text
Evaluation Framework (MATEval). In this framework, we simulate the human
collaborative process in evaluating texts generated by LLMs and propose a novel
multi-agent discussion strategy. This strategy integrates self-reflection [13] and
Chain-of-Thought (CoT) [17] concepts, as self-reflection focuses on understand-
ing the depth of issues but may lead to rigid thinking. Meanwhile, strategies
based on the CoT emphasize the refinement of problems but may lack in-depth
analysis of specific issues. Therefore, we combine the two approaches by guiding
agents through prompts to decompose evaluation questions and focus on only
one sub-question in each discussion round. During each round of the discussion,
agents engage in self-reflection, considering peer inputs to enrich issue compre-
hension. This approach strengthens agents’ self-assessment and critical thinking,
broadening their evaluation scope for open-ended text and aligning results more
closely with human evaluations.
Furthermore, our framework introduces a feedback mechanism at the end of
each discussion round to evaluate the quality and efficiency of the discussions,
encouraging agents to reach a consensus. The comprehensive evaluation report
generated by our framework details error types, specific locations, in-depth ex-
planations, and corresponding scores.
For practical applications in industry, we provide two report formats: a
question-and-answer format for strategy analysis and a text report designed to
help business personnel quickly identify errors and facilitate iterative improve-
ment of LLMs. Our framework has achieved significant results in the story text
evaluation task at Alipay, markedly enhancing the efficiency of model iteration.
To summarize, our main contributions in this paper are:
1. We propose a Multi-Agent Evaluation Framework called MATEval1, which
enhances the reliability of scoring by providing accurate diagnostic reports
for text generated by LLMs. This framework not only facilitates model iter-
ation in industrial scenarios but also significantly boosts audit efficiency.
2. We propose a novel method to integrate self-reflection and CoT in our multi-
agent framework. Additionally, we creatively introduce a feedback mech-
MATEval: A Multi-Agent Text Evaluation Framework
3
anism at the end of each discussion round to resolve disagreements and
facilitate the achievement of consensus.
3. We conduct comprehensive experiments on two English and two Chinese
story text datasets, including one constructed based on Alipay’s business
story text dataset. Our experimental results showcase the effectiveness of
our framework and its high correlation with human evaluations.
1
2
Related Work
Traditional NLG Evaluation: For a significant period, open-ended text evalu-
ation primarily depended on human annotations, which incurs substantial human
and financial costs. Subsequent automated NLG evaluations employ computa-
tional models to evaluate the quality of generated texts, such as BLEU [14],
ROUGE [10], and METEOR [1]. Embedding-based metrics refer to the evalua-
tion of generated texts by measuring the semantic similarity between generated
texts and reference texts based on word or sentence embeddings. BERTScore [18]
calculates the similarity between generated text and reference text based on
BERT’s contextual embedding. RUBERBERT [5] is also based on BERT embed-
dings to measure the similarity of texts with and without references through
processes such as pooling and MLP operations.
LLM-based Evaluators: GPTScore [4] utilizes models such as GPT-3 to
evaluate text quality, predicated on the assumption that generative pre-trained
models assign higher probabilities to high-quality generated texts by given in-
structions and context. Recent studies also explore the potential of using Chat-
GPT as an NLG evaluator [15]. G-Eval [12] demonstrates the evaluation of NLG
outputs using prompts in LLMs like ChatGPT through Chain-of-Thought (CoT)
methods.
Communicative Agents: Recently, the concept of using agents for commu-
nication and collaboration to accomplish specific tasks gains widespread applica-
tion. CAMEL [9] introduces a cooperative agent framework called role-playing,
enabling individual agents to collaboratively solve complex tasks autonomously.
ChatEval [3] applies the multi-agent approach to text evaluation, constructing
a multi-agent jury to explore the impact of different communication strategies
in evaluating open-ended questions and traditional NLG tasks.
3
Methodology
In this section, we will provide a detailed exposition of the design of the evalua-
tion framework within MATEval 1, the utilization of various strategies, and the
functional specifications of different roles.
1 We have made the datasets and results used in our experiments publicly available at
https://github.com/kse-ElEvEn/MATEval. Due to the user privacy of Alipay, we
cannot make the “Ant” dataset public.
4
Y. Li et al.
Story: We moved quite a bit
when I was a kid. This made it
difficult to make enemies. I
didn't mind because I had my
favorite stuffed toy. It was a
twenty-six inch short plush
monster I'd had since I was
four. Out of nowhere, a purple
unicorn introduced itself to me
when I was seven and we
moved my parents made me
leave it behind.
Multi-agent Text Evaluation Framework
First stage discussion
Self-reflection
Self-reflection
Feedback
Second stage discussion
Self-reflection
Feedback
Self-reflection
Summary
Q: Does the story contain ‘Repetition’?
A: No
Q: Does the story contain ‘Logical Inconsistency’?
A: Yes, “difficult to make enemies.” should be “difficult to make friends.”
Q: Does the story contain ‘Discontinuity’?
A: Yes, Completely unrelated sentences appear in the story. ” Out of
nowhere, a purple unicorn introduced itself to me “
Q: Does the story contain ‘Inappropriate Lexical Choice’?
A: Yes, “twenty-six inch” should not be used with “short”.
Q: Does the story contain ‘Factual Error’?
A: No
Score: -3 points
The story does not contain any element of ‘Repetition’. However, it does
include ‘Logical Inconsistency’, such as the phrase “difficult to make
enemies” which should actually be “difficult to make friends.” Additionally,
the story suffers from ‘Discontinuity’, exemplified by an abruptly
introduced sentence like “Out of nowhere, a purple unicorn introduced
itself to me.” Regarding ‘Inappropriate Lexical Choice’, using “twenty-six
inch” to describe something as “short” is inappropriate. As for ‘Factual
Error’, the story does not contain any. Overall, this story scores -3 points.
Self-reflection + CoT Strategy
QA result
Summary
…
Fig. 1: The overall process diagram of the MATEval Framework. The input to
the framework is a text with quality questions, which after going through a
multi-agent discussion that combines self-reflection and CoT strategies, outputs
a detailed evaluation report.
3.1
Design of the Framework
Our framework primarily consists of agents with different roles combined with
discussion strategies. The roles of agents we utilize include Evaluator Agent,
Feedback Agent, and Summarizer Agent, who collaborate to complete text eval-
uation tasks. The Evaluator Agent is the main entity in the evaluation task,
the Feedback Agent plays a crucial role in improving discussion quality and pro-
moting consensus, and the Summarizer Agent is indispensable for consolidating
discussion information, summarizing, and forming evaluation reports. In our
framework, we employ a discussion strategy that integrates self-reflection and
CoT.
The framework accepts text as input, which may contain various quality is-
sues. The output is a detailed evaluation report outlining error type, location,
explanation, and score. We present the results in two formats: one is a Q&A for-
mat conducive to evaluating correlation, allowing easy extraction of correlation
scores for similarity calculations. The other is a report format that is conducive
to iteration by relevant business personnel. This enables business personnel to
quickly identify text issues and refine models using the analysis reports, enhanc-
ing efficiency. This is shown in the right part of Figure 1.
3.2
Application of Different Roles
In this section, we introduce several key roles within our framework and their
respective functions.
Evaluator Agent: The core element in the framework is the evaluator, for
which we use GPT-4 to conduct multi-round evaluations and responses that are
guided through carefully designed prompts. The evaluator stores and processes
statements from other agents, using this as a reference for dialogue history.
Each agent not only receives responses from others but also provides their own
statements, with the entire process requiring minimal human intervention.
MATEval: A Multi-Agent Text Evaluation Framework
5
Prompt: Story:"…". Please evaluate the above text from the aspects of
"Repetition," "Inappropriate Lexical Choice," "Logical Inconsistency,"
"Discontinuity," and "Factual Error." Let's think it step by step.
Let's first discuss the issues of "Logical Inconsistency" and "Discontinuity." I
noticed in the text that... Therefore, the score for this story on these two
aspects is...
Prompt: Please carefully reconsider your previous statements and confirm
whether your assessment is accurate.
After careful consideration, I realize that I indeed made an error in judgment
regarding the "Logical Inconsistency" issue. We should reconsider the contextual
information for this problem...
Prompt: Regarding the above issues, "Logical Inconsistency" and "Discontinuity,"
Mike has the following perspective: ..., Do you have any differing views?
After careful consideration, I realize that I indeed made an error in judgment
regarding the "Logical Inconsistency" issue. We should reconsider the contextual
information for this problem...
Prompt: Please carefully reconsider your previous statements and confirm
whether your assessment is accurate.
After careful consideration, I stand by my views on these two issues. If Mike
has any objections, please let me know so we can continue the discussion.
First stage discussion
Prompt: Please assess the aforementioned discussion process. Does it contain a
significant amount of repetitive information? Are there unresolved disagreements
within the discussion?
The previous round of discussion was efficient and did not contain redundant
information. However, there was a disagreement on the issue of "Discontinuity."
Please aim to reach a consensus on this matter in the next round of discussion.
Prompt: Here is the feedback from Lily on the previous round of discussion: ..., Please
take her suggestions into account and proceed with the next round of discussion.
Regarding the issue of "Discontinuity," I agree with John's perspective, and we have
reached a consensus. Next, let's discuss the "Factual Error" issue. I believe...
Prompt: Please carefully reconsider your previous statements and confirm whether your
assessment is accurate.
……
Prompt: : To summarize all the above discussions, form a text evaluation report in the
following format: ...
Second stage discussion
Question: Does the above story contain ‘Repetition’?
Answer: Yes
Score for individual questions: -1, …
Question: Does the above story contain ‘Logical Inconsistency’?
Answer: Yes
Score for individual questions: -1, …
Question: Does the above story contain ‘Discontinuity’?
Answer: Yes
Score for individual questions: -1, …
Question: Does the above story contain ‘Inappropriate Lexical Choice’?
Answer: No
Score for individual questions: 0.
Question: Does the above story contain ‘Factual Error’?
Answer: Yes
Score for individual questions: -1,…
Summary
……
Evaluator
Evaluator
Feedback
Summarizer
The story includes 'Repetition', as evidenced by the phrase…
Fig. 2: The diagram includes prompts and dialogue that incorporates a process
of discussion with self-reflection, CoT, feedback mechanisms and final summary.
Feedback Agent: The feedback agent evaluates the content and quality of
each discussion round. It focuses on identifying inefficient dialogues and disagree-
ments. If issues are detected, it suggests improvements for subsequent rounds to
enhance efficiency and consensus through prompts.
Summarizer Agent: After all discussions are concluded, the summarizer
compiles the entire process and outcomes. It provides a Q&A format evaluation
report, detailing the identification, analysis, and scoring of various issues. Addi-
tionally, we provide a comprehensive text-based format evaluation report that
includes detailed problem descriptions and is easy to read to help improve model
performance in industrial production.
3.3
Feedback Mechanism
The feedback mechanism is a well-designed component in our framework. At
the end of each discussion round, we use a prompt to guide a feedback agent
to summarize and evaluate the discussion. Its role is to steer subsequent dis-
cussions towards less repetition, enhance the efficiency of the discussion, and
importantly, guide the participants towards reaching a consensus. All of these
are achieved by conveying the feedback provider’s remarks to the agents involved
in the discussion.
3.4
Combined Self-reflection and CoT
Self-reflection Strategy: After each agent’s statements, they engage in a pro-
cess of self-reflection. Guided by the prompt, agents adjust their statements by
integrating the viewpoints of other agents. The final statements of each agent
6
Y. Li et al.
are stored and used as historical information for subsequent discussions. In a
new round of discussion, the statements from the previous round are stored as
historical information.
CoT Strategy: We guide agents through prompts to autonomously decom-
pose problems and address only one sub-problem in each round of discussion.
Meanwhile, each agent’s statements are stored and used as historical information
for subsequent discussions.
Combined Self-reflection and CoT: As shown in Algorithm 1. Combin-
ing self-reflection and the CoT is an important strategy employed in our frame-
work. Agents autonomously decompose the question according to the prompt,
focusing on one sub-question in each round of multiple discussions: D(Q) =
{Q1, Q2, . . . , Qn}, Pi = I(Qi, H) , where I represents the formation of prelimi-
nary ideas based on the prompt, Q is the evaluation task, Qn is the sub-question
and H is the history information. They then optimize their statements through
self-reflection: Ri = S(Pi, H). Next, update the history: H = H ∪{Ri}. After
each round of discussion, a feedback provider evaluates the discussion to reduce
repetition and disagreement: E = E(H) . Finally, a summarizer compiles all
statements to produce the evaluation report: Rfinal = Summary(H). The overall
prompt and the flow of the discussion are illustrated in Figure 2.
Algorithm 1 Self-Reflection and Chain-of-Thought Discussion with Feedback
Input: Given text, set of sub-questions Q decomposed by LLMs-based agents, number
of agents N
Output: Final evaluation report Rfinal
1: H ←{}, Agents ←{A1, A2, . . . , AN}, SubQIndex ←0
2: while SubQIndex < length(Q) do
3:
SubQCurrent ←Q[SubQIndex]
▷select the current sub-question
4:
SubQIndex ←SubQIndex + 1
▷increment the sub-question index
5:
for each Ai in Agents do
6:
Pi ←Formulate Idea(Qi, H)
▷generate preliminary ideas
7:
Ri ←Self Reflection(Pi, H)
▷reflect on their statements
8:
H ←H ∪{Ri}
▷update historical record with reflected statement
9:
end for
10:
E ←Evaluate(H)
▷evaluator assesses this round’s discussion
11:
Fround ←Feedback(E)
▷provide feedback Fround to all agents
12: end while
13: Rfinal ←Summary(H)
▷summarizer generates error analysis report
14: return Rfinal
4
Experiments
4.1
Implementation Details
In the MATEval framework, we select OpenAI’s GPT-4 as our LLMs due to its
outstanding performance and API accessibility. We set the temperature param-
eter to 0 for result reproducibility. GPT-4’s easy access facilitated effective and
coherent multi-agent interactions in our experiments.
MATEval: A Multi-Agent Text Evaluation Framework
7
4.2
Dataset
We mainly apply our framework to the evaluation of story texts generated by
LLMs in Alipay business scenarios. So in the experiment, we mainly focus on
two open-ended story datasets: ROCStories (ROC) [6] and WritingPrompts
(WP) [6]. Considering GPT-4’s context length limitations and the need for
storing multi-round discussion contexts, we truncate WP stories to the first 200
words, ensuring textual integrity at the truncation point. To test model general-
izability, we conduct similar experiments on two Chinese datasets. These include
the Chinese LOng Text understanding and generation (LOT) [7] dataset, com-
prising human-written stories averaging 106 words, and a dataset of Chinese
fairy tales constructed using prompts from Alipay’s business data with GPT-
3.5 named Ant, mainly involving fables and fairy tales with an average story
length of 125 words. Considering GPT-4’s request rate limits and high usage
costs, we select the first 200 stories from each dataset for multi-agent discussion
experiments.
Using the GPT-4 interface, we introduce five basic types of errors into 200
story texts across different datasets to simulate possible problems in stories.
They are Repetition(REP), Logical Inconsistency (LINC), Discontinu-
ity (DCONT), Inappropriate Lexical Choice (ILC), and Factual Er-
ror (FER). Repetition includes redundant sentences or excessive word use;
Logical Inconsistency encompasses antonym substitution and polarity shifts in
sentences; Discontinuity involves sequencing errors or unrelated content; Inap-
propriate Lexical Choice refers to misused quantifiers or pronouns; and Factual
Error denotes contradictions with established knowledge. We hire five annotators
to assess these datasets, ensuring the data aligns with human preferences. Both
manual and multi-agent scoring follow the same criteria: starting from zero, each
error deducts one point, with scores tallied for each error type and the total for
each text.
4.3
Compared Methods
Referenced Metrics: The BLEU [14] score is used to evaluate lexical sim-
ilarity between candidate and reference texts. ROUGE-L [10] focuses on the
longest common subsequence to assess the fluency and coherence of texts. And
RUBER-BERT [5], an enhancement of the original RUBER model with
BERT’s contextual embeddings, includes both referenced and unreferenced ver-
sions. The referenced version, RUBER-BERTr, measures the similarity be-
tween candidate responses and reference texts using BERT word embeddings.
Unreferenced Metrics: The RUBER’s unreferenced version RUBER-
BERTu [5] predicts relevance between responses and queries using BERT word
embeddings followed by operations such as pooling and MLP. The BERT-based
UNION [8] model distinguishes human-written stories from automatically gen-
erated negative samples and corrects the interference of negative samples. Chat-
Eval [3] evaluates open-ended Q&A quality through multi-agent framework, and
we select its most effective One-by-One approach for comparison.
8
Y. Li et al.
Our Methods: In our experience, we compare various strategies:
– Single-Agent(SA): LLMs directly evaluate stories without multi-agent.
– One-by-One [3](O b O): Agents sequentially evaluate stories in multi-
round discussions, without optimization strategies.
– Self-Reflection (SR): Agents conduct self-reflection, considering their and
others’ previous statements during discussions.
– Chain-of-Thought (CoT): Agents break down the assessment problem
through prompts, solving one sub-problem in each discussion round.
– Self-Reflection + CoT (SR+CoT): By combining CoT and self-reflection
strategies, agents first decompose questions for discussion, then engage in
self-reflection each round.
In all these strategies, we employ feedback mechanisms at the end of each
discussion round, as well as a final summary.
4.4
Experimental Results
Our experiments on the ROC and WP datasets are presented in Table 1, using
MATEval framework strategies to evaluate narrative texts. We calculate Spear-
man (ρ) and Kendall (τ) correlation coefficients to compare models’ evaluations
with human judgments.
Analyzing the experimental results on the ROC and WP datasets, we draw
the following conclusions:
LLMs-based methods show better Spearman (ρ) and Kendall (τ) correla-
tions compared to traditional n-gram and Bert-based methods, proving their
effectiveness in text evaluation.
Multi-agent discussions generally surpass single-agent evaluations in perfor-
mance, suggesting they significantly improve text evaluation quality.
In analyzing the strategies used within the MATEval framework, we found
that employing self-reflection or the CoT independently produces unstable re-
sults across different error types. In some cases, these methods even underper-
formed compared to single-agent evaluations. This might be due to inherent flaws
when applying these strategies separately. For example, self-reflection can lead
to rigid thinking in multiple discussion rounds, where agents often repeat earlier
content without adding new insights. On the other hand, using CoT alone often
results in superficial and divergent perspectives, offering only a basic analysis of
each error type without delving deeper.
The combination of self-reflection and CoT achieved the best overall corre-
lation, particularly excelling over other methods in evaluating Logical Inconsis-
tency, Discontinuity, and Inappropriate Lexical Choice. It significantly improves
the evaluation of Discontinuity compared to the single-agent method, demon-
strating the framework’s high sensitivity to textual coherence. However, its effec-
tiveness was lower for Repetition and Factual Error. Agents often misidentified
emotionally similar sentences as repetitive, despite clear definitions. Further-
more, the framework’s evaluation of Factual Errors was limited by LLMs con-
straints, specifically the absence of external knowledge affecting common sense
MATEval: A Multi-Agent Text Evaluation Framework
9
Table 1: Correlation of evaluation results with human judgment using different
models and different strategies of MATEval on the ROC/WP dataset, where SA
stands for Single-Agent, SR denotes Self-Reflection, and CoT represents Chain-
of-Thought. The symbols ρ/τ respectively indicate the Spearman/Kendall cor-
relation. The highest correlation values are highlighted in bold.
Strategy
REP
LINC
DCONT
ILC
FER
ρ
τ
ρ
τ
ρ
τ
ρ
τ
ρ
τ
ROC
BLEU
0.318 0.260 0.193 0.153 0.156
0.128
0.037
0.031 -0.010 -0.008
ROUGE-L -0.017 -0.014 0.129 0.102 0.202
0.165
0.056
0.045 0.104 0.084
RUBERr
0.036 0.035 0.054 0.049 0.315
0.297
-0.018 -0.017 -0.176 -0.166
RUBERu
-0.111 -0.091 0.038 0.031 0.131
0.107
0.134
0.110 0.180 0.146
UNION
-0.093 -0.076 0.091 0.071 -0.018 -0.015
0.057
0.046 0.072 0.059
SA
0.699 0.694 0.268 0.253 0.318
0.312
0.240
0.236 0.545 0.538
O b O
0.698 0.692 0.170 0.160 0.356
0.349
0.259
0.248 0.484 0.473
SR
0.691 0.680 0.169 0.154 0.354
0.339
0.144
0.138 0.498 0.478
CoT
0.743 0.737 0.189 0.180 0.288
0.282
0.213
0.205 0.502 0.491
SR+CoT
0.735 0.728 0.281 0.264 0.391 0.382 0.263 0.256 0.575 0.561
WP
BLEU
0.087 0.071 0.096 0.073 0.039
0.033
-0.114 -0.091 0.009 0.007
ROUGE-L 0.092 0.074 0.127 0.096 0.083
0.068
-0.046 -0.037 0.049 0.040
RUBERr
0.038 0.036 -0.020 -0.018 -0.081 -0.076
0.035
0.033 0.076 0.071
RUBERu
-0.102 -0.084 0.054 0.041 -0.006 -0.005 -0.006 -0.007 0.111 0.089
UNION
0.048 0.039 0.010 0.008 -0.110 -0.090 -0.038 -0.031 0.052 0.042
SA
0.258 0.246 0.107 0.095 0.111
0.105
0.192 0.1802 0.176 0.171
O b O
0.386 0.380 0.183 0.166 0.081
0.075
0.089
0.082 0.299 0.286
SR
0.491 0.483 0.120 0.107 0.224
0.209
0.057
0.051 0.214 0.208
CoT
0.132 0.129 0.159 0.139 0.203
0.191
0.002
0.001 0.218 0.211
SR+CoT
0.430 0.417 0.215 0.188 0.265 0.248 0.290 0.266 0.299 0.286
error detection, highlighting the need to integrate external knowledge for future
multi-agent framework enhancements.
4.5
Ablation Study
To assess the effectiveness of different modules in MATEval, we conducted ab-
lation experiments on the ROC dataset, involving the removal of the feedback
mechanism, omitting Q&A format explanations, and not using a multi-agent ap-
proach. Table 3 demonstrates that the complete MATEval framework surpasses
its ablated versions, confirming the significance of both feedback mechanisms,
explanations and multi-agent methods. Specifically, this establishes the impor-
tance of our feedback mechanisms in promoting discussion consensus and en-
hancing relevance. It also proves that providing scoring explanations by LLMs
significantly enhances evaluation effectiveness.
10
Y. Li et al.
Table 2: Correlation of evaluation results with human judgment using different
models and different strategies of MATEval on the LOT/Ant dataset. The high-
est correlation values are highlighted in bold.
Strategy
REP
LINC
DCONT
ILC
FER
ρ
τ
ρ
τ
ρ
τ
ρ
τ
ρ
τ
LOT
SA
0.829 0.817 0.120
0.110 0.336 0.324 0.179
0.175
0.284
0.279
O b O
0.770 0.764
0.142
0.131 0.249 0.239 0.069
0.066
0.362 0.349
SR
0.751 0.735
0.054
0.048 0.282 0.267 0.118
0.112
0.296
0.284
CoT
0.636 0.628
0.026
0.024 0.215 0.206 0.051
0.049
0.155
0.151
SR+CoT 0.811 0.798
0.197 0.185 0.354 0.341 0.182 0.175
0.341
0.333
Ant
SA
0.522 0.517
0.281
0.275 0.231 0.231 0.318
0.316
0.495
0.489
O b O
0.545 0.538
0.145
0.141 -0.010 -0.010 0.011
0.018
0.347
0.343
SR
0.563 0.557
0.185
0.175 0.069 0.069 0.187
0.184
0.368
0.360
CoT
0.572 0.562
0.034
0.039 0.024 0.024 0.040
0.042
0.358
0.352
SR+CoT 0.694 0.676 0.403 0.377 0.287 0.282 0.424 0.417 0.552 0.544
Table 3: Correlation of evaluation results with human judgment of ablation ex-
periments on the ROC dataset.
Strategy
REP
LINC
DCONT
ILC
FER
ρ
τ
ρ
τ
ρ
τ
ρ
τ
ρ
τ
MATEval-FB
0.567 0.567 0.039 0.038 0.259 0.255 0.266 0.260 0.477 0.473
MATEval-QA
0.612 0.597 0.071 0.068 0.011 0.011 0.132 0.127 0.283 0.281
MATEval-multi 0.699 0.694 0.268 0.253 0.318 0.312 0.240 0.236 0.545 0.538
MATEval
0.735 0.728 0.281 0.264 0.391 0.382 0.263 0.256 0.575 0.561
4.6
Generalization Experiments
To verify the generalizability of our framework across different languages and
in the industrial field, we experimented with two Chinese datasets: LOT and
a story text dataset Ant, derived from Alipay’s business data. The findings in
Table 2 were similar to those from English datasets. Interestingly, single-agent
evaluations often perform better in Chinese, possibly due to its unique language
and sentence structure. This suggests that optimizing agents for different lan-
guages may require tailored adjustments for optimal performance.
4.7
Case Study
To showcase the MATEval framework’s effectiveness in real-world industrial set-
tings, we illustrated its application using a story text example shown in Table
3. We compared our framework against manual evaluation, single-agent evalua-
tion, and the strategy-less One by One evaluation method. Experimental results
MATEval: A Multi-Agent Text Evaluation Framework
11
Story: Bob and Mike wanted to go to the lake for a fishing trip. Bob and Mike had desired to go for a fishing trip to the lake. They packed up and brought
the camper so everyone could stay the night. After eventually arriving there, it began to rainy so heavily that fishing became impossible! With clear skies at
sunrise, they were free to play chess all day. Hoping for better weather in the morning, they went to sleep early.
Error
Type
Strategy
Error
Location
Reason
REP
Human
Bob and Mike had desired to go for a fishing trip
to the lake.
The action of Bob and Mike wanting to go to the lake for a fishing trip
was reiterated unnecessarily.
SA
Bob and Mike wanted to go to the lake for a
fishing trip.
The first two sentences are repetitive as they convey the same
information.
No_Strategy
Bob and Mike had desired to go for a fishing trip
to the lake.
The issue is that the same information is repeated twice, which is
unnecessary and redundant.
MATEval
Bob and Mike had desired to go for a fishing trip
to the lake.
The first two sentences are repetitive as they convey the same
information.
LINC
Human
With clear skies at sunrise, they were free to play
chess all day.
The stated action contradicts the theme of the story, which is fishing.
The original intention was to fish, not to play chess.
SA
With clear skies at sunrise, they were free to play
chess all day.
This sentence is logically inconsistent with the previous sentence "it
began to rainy so heavily that fishing became impossible!" as it implies
that they played chess all day despite the heavy rain.
No_Strategy
/
/
MATEval
With clear skies at sunrise, they were free to play
chess all day.
If the weather cleared up at sunrise, it would have been logical for
they to resume their original plan of fishing, rather than playing chess.
DCONT
Human
Hoping for better weather in the morning, they
went to sleep early.
This sentence originally after the heavy rain disrupted their fishing plans
and before the clear sky at sunrise. It is not chronologically.
SA
/
/
No_Strategy
/
/
MATEval
Hoping for better weather in the morning, they
went to sleep early.
The transition from them hoping for better weather and going to sleep
early, to them playing chess all day is abrupt.
ILC
Human
They packed up and brought the camper so
everyone could stay the night.
The story originally involved only two characters, Bob and Mike. The use
of 'everyone' is inappropriate as it implies the presence of more
characters.
SA
/
/
No_Strategy
/
/
MATEval
They packed up and brought the camper so
everyone could stay the night.
It is not appropriate to use ‘everyone’ in a situation between two people.
FER
Human
/
/
SA
/
/
No_Strategy
/
/
MATEval
/
/
Fig. 3: The schematic diagram illustrating the comparison of results generated
by the MATEval framework and other methods.
demonstrate that our method is basically consistent with human evaluations,
unlike other methods, which show some gaps, thereby confirming the high cor-
relation between our approach and human evaluation.
5
Conclusion
In this paper, we proposed the MATEval framework, which enhances the evalu-
ation performance of open-ended story text generated by LLMs in the industrial
field. Extensive experiments show that MATEval’s evaluation results on two
classic story datasets are more aligned with human preferences than those of
existing methods. In the Alipay industrial scenario, our framework significantly
improves review efficiency and evaluation accuracy, serving as an effective aid.23
In future work, we can fine-tune LLMs as agents in the industrial field to com-
plete specific domain tasks. When solving a complex domain task, we can enable
these domain-specific agents to collaborate with each other, thereby enhancing
their capability and efficiency in addressing challenges.
2 This work was supported by Ant Group
3 This work was supported by the Natural Science Foundation of China (Grant No.
U21A20488). We thank the Big Data Computing Center of Southeast University for
providing the facility support on the numerical calculations in this paper.
12
Y. Li et al.
References
1. Banerjee, S., Lavie, A.: METEOR: an automatic metric for MT evaluation with
improved correlation with human judgments. In: Proceedings of the Workshop
on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or
Summarization@ACL. pp. 65–72 (2005)
2. Callison-Burch, C.: Fast, cheap, and creative: Evaluating translation quality using
amazon’s mechanical turk. In: Proceedings of the 2009 Conference on Empirical
Methods in Natural Language Processing, EMNLP 2009. pp. 286–295
3. Chan, C.M., et al: Chateval: Towards better llm-based evaluators through multi-
agent debate. arXiv preprint arXiv:2308.07201 (2023)
4. Fu, J., et al: Gptscore: Evaluate as you desire. arXiv preprint arXiv:2302.04166
(2023)
5. Ghazarian, S., et al: Better automatic evaluation of open-domain dialogue systems
with contextualized embeddings. arXiv preprint arXiv:1904.10635 (2019)
6. Guan, J., et al: Openmeva: A benchmark for evaluating open-ended story gener-
ation metrics. In: Proceedings of the 59th Annual Meeting of the Association for
Computational Linguistics and the 11th International Joint Conference on Natural
Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers). pp. 6394–
6407 (2021)
7. Guan, J., et al: LOT: A story-centric benchmark for evaluating chinese long text
understanding and generation 10, 434–451 (2022)
8. Guan, J., Huang, M.: UNION: an unreferenced metric for evaluating open-ended
story generation. In: Proceedings of the 2020 Conference on Empirical Methods in
Natural Language Processing, EMNLP. pp. 9157–9166 (2020)
9. Li, G., et al: Camel: Communicative agents for“ mind” exploration of large scale
language model society. arXiv preprint arXiv:2303.17760 (2023)
10. Lin, C.Y.: ROUGE: A package for automatic evaluation of summaries. In: Text
Summarization Branches Out. pp. 74–81 (Jul 2004)
11. Liu, C., et al: How to evaluate your dialogue system: An empirical study of unsu-
pervised evaluation metrics for dialogue response generation. In: Proceedings of the
2016 Conference on Empirical Methods in Natural Language Processing, EMNLP.
pp. 2122–2132 (2016)
12. Liu, Y., et al: G-eval: NLG evaluation using gpt-4 with better human alignment.
In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language
Processing, EMNLP. pp. 2511–2522 (2023)
13. Madaan, A., et al: Self-refine: Iterative refinement with self-feedback. arXiv
preprint arXiv:2303.17651 (2023)
14. Papineni, K., et al: Bleu: a method for automatic evaluation of machine translation.
In: ACL. pp. 311–318 (2002)
15. Wang, J., et al: Is chatgpt a good nlg evaluator? a preliminary study. arXiv preprint
arXiv:2303.04048 (2023)
16. Wang, P., et al: Large language models are not fair evaluators. arXiv preprint
arXiv:2305.17926 (2023)
17. Wei, J., et al: Chain-of-thought prompting elicits reasoning in large language mod-
els. Advances in Neural Information Processing Systems 35, 24824–24837 (2022)
18. Zhang, T., Kishore, V., Wu, F., Artzi, Y.: Bertscore: Evaluating text generation
with BERT. In: 8th International Conference on Learning Representations, ICLR
(2020)