Question:
What ethical or practical challenges might arise from deploying the MATEval framework in real-world AI evaluation scenarios?
A. Its multi-agent LLM system could increase computational resource demands, raising concerns about environmental sustainability in large-scale deployments.
B. Its reliance on self-reflection might introduce subjective biases from LLMs, complicating efforts to ensure fair and consistent evaluations across diverse datasets.
C. Its use of Chain-of-Thought reasoning could make evaluation outcomes harder to interpret, challenging transparency requirements in regulated industries.
D. Its integration of multiple LLMs could reduce the risk of overfitting to specific text styles, ensuring robust performance in multilingual contexts.

Answer: AC

In the MATEval framework, we select OpenAI’s GPT-4 as our LLMs due to its outstanding performance and API accessibility. We set the temperature parameter to 0 for result reproducibility. GPT-4’s easy access facilitated effective and coherent multi-agent interactions in our experiments.

We mainly apply our framework to the evaluation of story texts generated by LLMs in Alipay business scenarios. So in the experiment, we mainly focus on two open-ended story datasets: ROCStories (ROC) [6] and WritingPrompts (WP) [6]. Considering GPT-4’s context length limitations and the need for storing multi-round discussion contexts, we truncate WP stories to the first 200 words, ensuring textual integrity at the truncation point. To test model generalizability, we conduct similar experiments on two Chinese datasets. These include the Chinese LOng Text understanding and generation (LOT) [7] dataset, comprising human-written stories averaging 106 words, and a dataset of Chinese fairy tales constructed using prompts from Alipay’s business data with GPT-3.5 named Ant, mainly involving fables and fairy tales with an average story length of 125 words. Considering GPT-4’s request rate limits and high usage costs, we select the first 200 stories from each dataset for multi-agent discussion experiments.

Analyzing the experimental results on the ROC and WP datasets, we draw the following conclusions:
LLMs-based methods show better Spearman (ρ) and Kendall (τ) correlations compared to traditional n-gram and Bert-based methods, proving their effectiveness in text evaluation.
Multi-agent discussions generally surpass single-agent evaluations in performance, suggesting they significantly improve text evaluation quality.
In analyzing the strategies used within the MATEval framework, we found that employing self-reflection or the CoT independently produces unstable results across different error types. In some cases, these methods even underperformed compared to single-agent evaluations. This might be due to inherent flaws when applying these strategies separately. For example, self-reflection can lead to rigid thinking in multiple discussion rounds, where agents often repeat earlier content without adding new insights. On the other hand, using CoT alone often results in superficial and divergent perspectives, offering only a basic analysis of each error type without delving deeper.
The combination of self-reflection and CoT achieved the best overall correlation, particularly excelling over other methods in evaluating Logical Inconsistency, Discontinuity, and Inappropriate Lexical Choice. It significantly improves the evaluation of Discontinuity compared to the single-agent method, demonstrating the framework’s high sensitivity to textual coherence. However, its effectiveness was lower for Repetition and Factual Error. Agents often misidentified emotionally similar sentences as repetitive, despite clear definitions. Furthermore, the framework’s evaluation of Factual Errors was limited by LLMs constraints, specifically the absence of external knowledge affecting common sense error detection, highlighting the need to integrate external knowledge for future multi-agent framework enhancements.

MATEval: A Multi-Agent Discussion Framework
for Advancing Open-Ended Text Evaluation
Yu Li1,⋆, Shenyu Zhang1,⋆, Rui Wu2, Xiutian Huang2, Yongrui Chen1,
Wenhao Xu2(), Guilin Qi1 (), and Dehai Min1
1 Southeast University, Nanjing, China
{yuli 11, shenyuzhang, yrchen, gqi, zhishanq}@seu.edu.cn
2 Ant Group, Hangzhou, China
{guli.wr, xiutian.hxt, hao.xuwh}@antgroup.com
Abstract. Recent advancements in generative Large Language Models
(LLMs) have been remarkable, however, the quality of the text generated
by these models often reveals persistent issues. Evaluating the quality of
text generated by these models, especially in open-ended text, has con-
sistently presented a significant challenge. Addressing this, recent work
has explored the possibility of using LLMs as evaluators. While using a
single LLM as an evaluation agent shows potential, it is filled with sig-
nificant uncertainty and instability. To address these issues, we propose
the MATEval: A “Multi-Agent Text Evaluation framework” where
all agents are played by LLMs like GPT-4. The MATEval framework
emulates human collaborative discussion methods, integrating multiple
agents’ interactions to evaluate open-ended text. Our framework incorpo-
rates self-reflection and Chain-of-Thought (CoT) strategies, along with
feedback mechanisms, enhancing the depth and breadth of the evaluation
process and guiding discussions towards consensus, while the framework
generates comprehensive evaluation reports, including error localization,
error types and scoring. Experimental results show that our framework
outperforms existing open-ended text evaluation methods and achieves
the highest correlation with human evaluation, which confirms the effec-
tiveness and advancement of our framework in addressing the uncertain-
ties and instabilities in evaluating LLMs-generated text. Furthermore,
our framework significantly improves the efficiency of text evaluation
and model iteration in industrial scenarios.
Keywords: Multi-Agent · Large Language Models · Text Evaluation
1
Introduction
Evaluating the text generated by large language models (LLMs) has long been
a challenging task, Traditional manual evaluation methods are not only time-
consuming and laborious but also expensive [2]. Although methods like BLEU
[14], Rouge [10], and METEOR [1] have achieved success in scenarios such as
⋆Equal Contributors.
arXiv:2403.19305v2  [cs.CL]  15 Apr 2024
2
Y. Li et al.
machine translation, these automated evaluation methods are limited in the
context of open-ended text generation [11]. Recently, LLMs have been used as
evaluators such as G-Eval [4], but these methods exhibit unstable and uncer-
tain evaluation effects [15] [16]. Even certain collaboration frameworks could
alleviate this problem by employing multi-agent discussion, e.g. ChatEval [3],
however, the current methods of multi-agent collaboration remain limited to
simple interactions, without fully harnessing the potential for agents’ thinking
and planning. Additionally, reaching a consensus within multi-agent discussion
frameworks continues to be a challenging issue. Furthermore, traditional text
evaluation models typically provide only a score without explaining it, making
it difficult for reviewers to trust the reliability of these scores. They still need
to manually identify errors, obviously slowing down the collection of bad cases
and, consequently, affecting the pace of model iteration in industrial scenarios.
To address the above challenges, this paper introduces a Multi-Agent Text
Evaluation Framework (MATEval). In this framework, we simulate the human
collaborative process in evaluating texts generated by LLMs and propose a novel
multi-agent discussion strategy. This strategy integrates self-reflection [13] and
Chain-of-Thought (CoT) [17] concepts, as self-reflection focuses on understand-
ing the depth of issues but may lead to rigid thinking. Meanwhile, strategies
based on the CoT emphasize the refinement of problems but may lack in-depth
analysis of specific issues. Therefore, we combine the two approaches by guiding
agents through prompts to decompose evaluation questions and focus on only
one sub-question in each discussion round. During each round of the discussion,
agents engage in self-reflection, considering peer inputs to enrich issue compre-
hension. This approach strengthens agents’ self-assessment and critical thinking,
broadening their evaluation scope for open-ended text and aligning results more
closely with human evaluations.
Furthermore, our framework introduces a feedback mechanism at the end of
each discussion round to evaluate the quality and efficiency of the discussions,
encouraging agents to reach a consensus. The comprehensive evaluation report
generated by our framework details error types, specific locations, in-depth ex-
planations, and corresponding scores.
For practical applications in industry, we provide two report formats: a
question-and-answer format for strategy analysis and a text report designed to
help business personnel quickly identify errors and facilitate iterative improve-
ment of LLMs. Our framework has achieved significant results in the story text
evaluation task at Alipay, markedly enhancing the efficiency of model iteration.
To summarize, our main contributions in this paper are:
1. We propose a Multi-Agent Evaluation Framework called MATEval1, which
enhances the reliability of scoring by providing accurate diagnostic reports
for text generated by LLMs. This framework not only facilitates model iter-
ation in industrial scenarios but also significantly boosts audit efficiency.
2. We propose a novel method to integrate self-reflection and CoT in our multi-
agent framework. Additionally, we creatively introduce a feedback mech-
MATEval: A Multi-Agent Text Evaluation Framework
3
anism at the end of each discussion round to resolve disagreements and
facilitate the achievement of consensus.
3. We conduct comprehensive experiments on two English and two Chinese
story text datasets, including one constructed based on Alipay’s business
story text dataset. Our experimental results showcase the effectiveness of
our framework and its high correlation with human evaluations.
1
2
Related Work
Traditional NLG Evaluation: For a significant period, open-ended text evalu-
ation primarily depended on human annotations, which incurs substantial human
and financial costs. Subsequent automated NLG evaluations employ computa-
tional models to evaluate the quality of generated texts, such as BLEU [14],
ROUGE [10], and METEOR [1]. Embedding-based metrics refer to the evalua-
tion of generated texts by measuring the semantic similarity between generated
texts and reference texts based on word or sentence embeddings. BERTScore [18]
calculates the similarity between generated text and reference text based on
BERT’s contextual embedding. RUBERBERT [5] is also based on BERT embed-
dings to measure the similarity of texts with and without references through
processes such as pooling and MLP operations.
LLM-based Evaluators: GPTScore [4] utilizes models such as GPT-3 to
evaluate text quality, predicated on the assumption that generative pre-trained
models assign higher probabilities to high-quality generated texts by given in-
structions and context. Recent studies also explore the potential of using Chat-
GPT as an NLG evaluator [15]. G-Eval [12] demonstrates the evaluation of NLG
outputs using prompts in LLMs like ChatGPT through Chain-of-Thought (CoT)
methods.
Communicative Agents: Recently, the concept of using agents for commu-
nication and collaboration to accomplish specific tasks gains widespread applica-
tion. CAMEL [9] introduces a cooperative agent framework called role-playing,
enabling individual agents to collaboratively solve complex tasks autonomously.
ChatEval [3] applies the multi-agent approach to text evaluation, constructing
a multi-agent jury to explore the impact of different communication strategies
in evaluating open-ended questions and traditional NLG tasks.
3
Methodology
In this section, we will provide a detailed exposition of the design of the evalua-
tion framework within MATEval 1, the utilization of various strategies, and the
functional specifications of different roles.
1 We have made the datasets and results used in our experiments publicly available at
https://github.com/kse-ElEvEn/MATEval. Due to the user privacy of Alipay, we
cannot make the “Ant” dataset public.
4
Y. Li et al.
Story: We moved quite a bit
when I was a kid. This made it
difficult to make enemies. I
didn't mind because I had my
favorite stuffed toy. It was a
twenty-six inch short plush
monster I'd had since I was
four. Out of nowhere, a purple
unicorn introduced itself to me
when I was seven and we
moved my parents made me
leave it behind.
Multi-agent Text Evaluation Framework
First stage discussion
Self-reflection
Self-reflection
Feedback
Second stage discussion
Self-reflection
Feedback
Self-reflection
Summary
Q: Does the story contain ‘Repetition’?
A: No
Q: Does the story contain ‘Logical Inconsistency’?
A: Yes, “difficult to make enemies.” should be “difficult to make friends.”
Q: Does the story contain ‘Discontinuity’?
A: Yes, Completely unrelated sentences appear in the story. ” Out of
nowhere, a purple unicorn introduced itself to me “
Q: Does the story contain ‘Inappropriate Lexical Choice’?
A: Yes, “twenty-six inch” should not be used with “short”.
Q: Does the story contain ‘Factual Error’?
A: No
Score: -3 points
The story does not contain any element of ‘Repetition’. However, it does
include ‘Logical Inconsistency’, such as the phrase “difficult to make
enemies” which should actually be “difficult to make friends.” Additionally,
the story suffers from ‘Discontinuity’, exemplified by an abruptly
introduced sentence like “Out of nowhere, a purple unicorn introduced
itself to me.” Regarding ‘Inappropriate Lexical Choice’, using “twenty-six
inch” to describe something as “short” is inappropriate. As for ‘Factual
Error’, the story does not contain any. Overall, this story scores -3 points.
Self-reflection + CoT Strategy
QA result
Summary
…
Fig. 1: The overall process diagram of the MATEval Framework. The input to
the framework is a text with quality questions, which after going through a
multi-agent discussion that combines self-reflection and CoT strategies, outputs
a detailed evaluation report.
3.1
Design of the Framework
Our framework primarily consists of agents with different roles combined with
discussion strategies. The roles of agents we utilize include Evaluator Agent,
Feedback Agent, and Summarizer Agent, who collaborate to complete text eval-
uation tasks. The Evaluator Agent is the main entity in the evaluation task,
the Feedback Agent plays a crucial role in improving discussion quality and pro-
moting consensus, and the Summarizer Agent is indispensable for consolidating
discussion information, summarizing, and forming evaluation reports. In our
framework, we employ a discussion strategy that integrates self-reflection and
CoT.
The framework accepts text as input, which may contain various quality is-
sues. The output is a detailed evaluation report outlining error type, location,
explanation, and score. We present the results in two formats: one is a Q&A for-
mat conducive to evaluating correlation, allowing easy extraction of correlation
scores for similarity calculations. The other is a report format that is conducive
to iteration by relevant business personnel. This enables business personnel to
quickly identify text issues and refine models using the analysis reports, enhanc-
ing efficiency. This is shown in the right part of Figure 1.
3.2
Application of Different Roles
In this section, we introduce several key roles within our framework and their
respective functions.
Evaluator Agent: The core element in the framework is the evaluator, for
which we use GPT-4 to conduct multi-round evaluations and responses that are
guided through carefully designed prompts. The evaluator stores and processes
statements from other agents, using this as a reference for dialogue history.
Each agent not only receives responses from others but also provides their own
statements, with the entire process requiring minimal human intervention.
MATEval: A Multi-Agent Text Evaluation Framework
5
Prompt: Story:"…". Please evaluate the above text from the aspects of
"Repetition," "Inappropriate Lexical Choice," "Logical Inconsistency,"
"Discontinuity," and "Factual Error." Let's think it step by step.
Let's first discuss the issues of "Logical Inconsistency" and "Discontinuity." I
noticed in the text that... Therefore, the score for this story on these two
aspects is...
Prompt: Please carefully reconsider your previous statements and confirm
whether your assessment is accurate.
After careful consideration, I realize that I indeed made an error in judgment
regarding the "Logical Inconsistency" issue. We should reconsider the contextual
information for this problem...
Prompt: Regarding the above issues, "Logical Inconsistency" and "Discontinuity,"
Mike has the following perspective: ..., Do you have any differing views?
After careful consideration, I realize that I indeed made an error in judgment
regarding the "Logical Inconsistency" issue. We should reconsider the contextual
information for this problem...
Prompt: Please carefully reconsider your previous statements and confirm
whether your assessment is accurate.
After careful consideration, I stand by my views on these two issues. If Mike
has any objections, please let me know so we can continue the discussion.
First stage discussion
Prompt: Please assess the aforementioned discussion process. Does it contain a
significant amount of repetitive information? Are there unresolved disagreements
within the discussion?
The previous round of discussion was efficient and did not contain redundant
information. However, there was a disagreement on the issue of "Discontinuity."
Please aim to reach a consensus on this matter in the next round of discussion.
Prompt: Here is the feedback from Lily on the previous round of discussion: ..., Please
take her suggestions into account and proceed with the next round of discussion.
Regarding the issue of "Discontinuity," I agree with John's perspective, and we have
reached a consensus. Next, let's discuss the "Factual Error" issue. I believe...
Prompt: Please carefully reconsider your previous statements and confirm whether your
assessment is accurate.
……
Prompt: : To summarize all the above discussions, form a text evaluation report in the
following format: ...
Second stage discussion
Question: Does the above story contain ‘Repetition’?
Answer: Yes
Score for individual questions: -1, …
Question: Does the above story contain ‘Logical Inconsistency’?
Answer: Yes
Score for individual questions: -1, …
Question: Does the above story contain ‘Discontinuity’?
Answer: Yes
Score for individual questions: -1, …
Question: Does the above story contain ‘Inappropriate Lexical Choice’?
Answer: No
Score for individual questions: 0.
Question: Does the above story contain ‘Factual Error’?
Answer: Yes
Score for individual questions: -1,…
Summary
……
Evaluator
Evaluator
Feedback
Summarizer
The story includes 'Repetition', as evidenced by the phrase…
Fig. 2: The diagram includes prompts and dialogue that incorporates a process
of discussion with self-reflection, CoT, feedback mechanisms and final summary.
Feedback Agent: The feedback agent evaluates the content and quality of
each discussion round. It focuses on identifying inefficient dialogues and disagree-
ments. If issues are detected, it suggests improvements for subsequent rounds to
enhance efficiency and consensus through prompts.
Summarizer Agent: After all discussions are concluded, the summarizer
compiles the entire process and outcomes. It provides a Q&A format evaluation
report, detailing the identification, analysis, and scoring of various issues. Addi-
tionally, we provide a comprehensive text-based format evaluation report that
includes detailed problem descriptions and is easy to read to help improve model
performance in industrial production.
3.3
Feedback Mechanism
The feedback mechanism is a well-designed component in our framework. At
the end of each discussion round, we use a prompt to guide a feedback agent
to summarize and evaluate the discussion. Its role is to steer subsequent dis-
cussions towards less repetition, enhance the efficiency of the discussion, and
importantly, guide the participants towards reaching a consensus. All of these
are achieved by conveying the feedback provider’s remarks to the agents involved
in the discussion.
3.4
Combined Self-reflection and CoT
Self-reflection Strategy: After each agent’s statements, they engage in a pro-
cess of self-reflection. Guided by the prompt, agents adjust their statements by
integrating the viewpoints of other agents. The final statements of each agent
6
Y. Li et al.
are stored and used as historical information for subsequent discussions. In a
new round of discussion, the statements from the previous round are stored as
historical information.
CoT Strategy: We guide agents through prompts to autonomously decom-
pose problems and address only one sub-problem in each round of discussion.
Meanwhile, each agent’s statements are stored and used as historical information
for subsequent discussions.
Combined Self-reflection and CoT: As shown in Algorithm 1. Combin-
ing self-reflection and the CoT is an important strategy employed in our frame-
work. Agents autonomously decompose the question according to the prompt,
focusing on one sub-question in each round of multiple discussions: D(Q) =
{Q1, Q2, . . . , Qn}, Pi = I(Qi, H) , where I represents the formation of prelimi-
nary ideas based on the prompt, Q is the evaluation task, Qn is the sub-question
and H is the history information. They then optimize their statements through
self-reflection: Ri = S(Pi, H). Next, update the history: H = H ∪{Ri}. After
each round of discussion, a feedback provider evaluates the discussion to reduce
repetition and disagreement: E = E(H) . Finally, a summarizer compiles all
statements to produce the evaluation report: Rfinal = Summary(H). The overall
prompt and the flow of the discussion are illustrated in Figure 2.
Algorithm 1 Self-Reflection and Chain-of-Thought Discussion with Feedback
Input: Given text, set of sub-questions Q decomposed by LLMs-based agents, number
of agents N
Output: Final evaluation report Rfinal
1: H ←{}, Agents ←{A1, A2, . . . , AN}, SubQIndex ←0
2: while SubQIndex < length(Q) do
3:
SubQCurrent ←Q[SubQIndex]
▷select the current sub-question
4:
SubQIndex ←SubQIndex + 1
▷increment the sub-question index
5:
for each Ai in Agents do
6:
Pi ←Formulate Idea(Qi, H)
▷generate preliminary ideas
7:
Ri ←Self Reflection(Pi, H)
▷reflect on their statements
8:
H ←H ∪{Ri}
▷update historical record with reflected statement
9:
end for
10:
E ←Evaluate(H)
▷evaluator assesses this round’s discussion
11:
Fround ←Feedback(E)
▷provide feedback Fround to all agents
12: end while
13: Rfinal ←Summary(H)
▷summarizer generates error analysis report
14: return Rfinal
4
Experiments
4.1
Implementation Details
In the MATEval framework, we select OpenAI’s GPT-4 as our LLMs due to its
outstanding performance and API accessibility. We set the temperature param-
eter to 0 for result reproducibility. GPT-4’s easy access facilitated effective and
coherent multi-agent interactions in our experiments.
MATEval: A Multi-Agent Text Evaluation Framework
7
4.2
Dataset
We mainly apply our framework to the evaluation of story texts generated by
LLMs in Alipay business scenarios. So in the experiment, we mainly focus on
two open-ended story datasets: ROCStories (ROC) [6] and WritingPrompts
(WP) [6]. Considering GPT-4’s context length limitations and the need for
storing multi-round discussion contexts, we truncate WP stories to the first 200
words, ensuring textual integrity at the truncation point. To test model general-
izability, we conduct similar experiments on two Chinese datasets. These include
the Chinese LOng Text understanding and generation (LOT) [7] dataset, com-
prising human-written stories averaging 106 words, and a dataset of Chinese
fairy tales constructed using prompts from Alipay’s business data with GPT-
3.5 named Ant, mainly involving fables and fairy tales with an average story
length of 125 words. Considering GPT-4’s request rate limits and high usage
costs, we select the first 200 stories from each dataset for multi-agent discussion
experiments.
Using the GPT-4 interface, we introduce five basic types of errors into 200
story texts across different datasets to simulate possible problems in stories.
They are Repetition(REP), Logical Inconsistency (LINC), Discontinu-
ity (DCONT), Inappropriate Lexical Choice (ILC), and Factual Er-
ror (FER). Repetition includes redundant sentences or excessive word use;
Logical Inconsistency encompasses antonym substitution and polarity shifts in
sentences; Discontinuity involves sequencing errors or unrelated content; Inap-
propriate Lexical Choice refers to misused quantifiers or pronouns; and Factual
Error denotes contradictions with established knowledge. We hire five annotators
to assess these datasets, ensuring the data aligns with human preferences. Both
manual and multi-agent scoring follow the same criteria: starting from zero, each
error deducts one point, with scores tallied for each error type and the total for
each text.
4.3
Compared Methods
Referenced Metrics: The BLEU [14] score is used to evaluate lexical sim-
ilarity between candidate and reference texts. ROUGE-L [10] focuses on the
longest common subsequence to assess the fluency and coherence of texts. And
RUBER-BERT [5], an enhancement of the original RUBER model with
BERT’s contextual embeddings, includes both referenced and unreferenced ver-
sions. The referenced version, RUBER-BERTr, measures the similarity be-
tween candidate responses and reference texts using BERT word embeddings.
Unreferenced Metrics: The RUBER’s unreferenced version RUBER-
BERTu [5] predicts relevance between responses and queries using BERT word
embeddings followed by operations such as pooling and MLP. The BERT-based
UNION [8] model distinguishes human-written stories from automatically gen-
erated negative samples and corrects the interference of negative samples. Chat-
Eval [3] evaluates open-ended Q&A quality through multi-agent framework, and
we select its most effective One-by-One approach for comparison.
8
Y. Li et al.
Our Methods: In our experience, we compare various strategies:
– Single-Agent(SA): LLMs directly evaluate stories without multi-agent.
– One-by-One [3](O b O): Agents sequentially evaluate stories in multi-
round discussions, without optimization strategies.
– Self-Reflection (SR): Agents conduct self-reflection, considering their and
others’ previous statements during discussions.
– Chain-of-Thought (CoT): Agents break down the assessment problem
through prompts, solving one sub-problem in each discussion round.
– Self-Reflection + CoT (SR+CoT): By combining CoT and self-reflection
strategies, agents first decompose questions for discussion, then engage in
self-reflection each round.
In all these strategies, we employ feedback mechanisms at the end of each
discussion round, as well as a final summary.
4.4
Experimental Results
Our experiments on the ROC and WP datasets are presented in Table 1, using
MATEval framework strategies to evaluate narrative texts. We calculate Spear-
man (ρ) and Kendall (τ) correlation coefficients to compare models’ evaluations
with human judgments.
Analyzing the experimental results on the ROC and WP datasets, we draw
the following conclusions:
LLMs-based methods show better Spearman (ρ) and Kendall (τ) correla-
tions compared to traditional n-gram and Bert-based methods, proving their
effectiveness in text evaluation.
Multi-agent discussions generally surpass single-agent evaluations in perfor-
mance, suggesting they significantly improve text evaluation quality.
In analyzing the strategies used within the MATEval framework, we found
that employing self-reflection or the CoT independently produces unstable re-
sults across different error types. In some cases, these methods even underper-
formed compared to single-agent evaluations. This might be due to inherent flaws
when applying these strategies separately. For example, self-reflection can lead
to rigid thinking in multiple discussion rounds, where agents often repeat earlier
content without adding new insights. On the other hand, using CoT alone often
results in superficial and divergent perspectives, offering only a basic analysis of
each error type without delving deeper.
The combination of self-reflection and CoT achieved the best overall corre-
lation, particularly excelling over other methods in evaluating Logical Inconsis-
tency, Discontinuity, and Inappropriate Lexical Choice. It significantly improves
the evaluation of Discontinuity compared to the single-agent method, demon-
strating the framework’s high sensitivity to textual coherence. However, its effec-
tiveness was lower for Repetition and Factual Error. Agents often misidentified
emotionally similar sentences as repetitive, despite clear definitions. Further-
more, the framework’s evaluation of Factual Errors was limited by LLMs con-
straints, specifically the absence of external knowledge affecting common sense
MATEval: A Multi-Agent Text Evaluation Framework
9
Table 1: Correlation of evaluation results with human judgment using different
models and different strategies of MATEval on the ROC/WP dataset, where SA
stands for Single-Agent, SR denotes Self-Reflection, and CoT represents Chain-
of-Thought. The symbols ρ/τ respectively indicate the Spearman/Kendall cor-
relation. The highest correlation values are highlighted in bold.
Strategy
REP
LINC
DCONT
ILC
FER
ρ
τ
ρ
τ
ρ
τ
ρ
τ
ρ
τ
ROC
BLEU
0.318 0.260 0.193 0.153 0.156
0.128
0.037
0.031 -0.010 -0.008
ROUGE-L -0.017 -0.014 0.129 0.102 0.202
0.165
0.056
0.045 0.104 0.084
RUBERr
0.036 0.035 0.054 0.049 0.315
0.297
-0.018 -0.017 -0.176 -0.166
RUBERu
-0.111 -0.091 0.038 0.031 0.131
0.107
0.134
0.110 0.180 0.146
UNION
-0.093 -0.076 0.091 0.071 -0.018 -0.015
0.057
0.046 0.072 0.059
SA
0.699 0.694 0.268 0.253 0.318
0.312
0.240
0.236 0.545 0.538
O b O
0.698 0.692 0.170 0.160 0.356
0.349
0.259
0.248 0.484 0.473
SR
0.691 0.680 0.169 0.154 0.354
0.339
0.144
0.138 0.498 0.478
CoT
0.743 0.737 0.189 0.180 0.288
0.282
0.213
0.205 0.502 0.491
SR+CoT
0.735 0.728 0.281 0.264 0.391 0.382 0.263 0.256 0.575 0.561
WP
BLEU
0.087 0.071 0.096 0.073 0.039
0.033
-0.114 -0.091 0.009 0.007
ROUGE-L 0.092 0.074 0.127 0.096 0.083
0.068
-0.046 -0.037 0.049 0.040
RUBERr
0.038 0.036 -0.020 -0.018 -0.081 -0.076
0.035
0.033 0.076 0.071
RUBERu
-0.102 -0.084 0.054 0.041 -0.006 -0.005 -0.006 -0.007 0.111 0.089
UNION
0.048 0.039 0.010 0.008 -0.110 -0.090 -0.038 -0.031 0.052 0.042
SA
0.258 0.246 0.107 0.095 0.111
0.105
0.192 0.1802 0.176 0.171
O b O
0.386 0.380 0.183 0.166 0.081
0.075
0.089
0.082 0.299 0.286
SR
0.491 0.483 0.120 0.107 0.224
0.209
0.057
0.051 0.214 0.208
CoT
0.132 0.129 0.159 0.139 0.203
0.191
0.002
0.001 0.218 0.211
SR+CoT
0.430 0.417 0.215 0.188 0.265 0.248 0.290 0.266 0.299 0.286
error detection, highlighting the need to integrate external knowledge for future
multi-agent framework enhancements.
4.5
Ablation Study
To assess the effectiveness of different modules in MATEval, we conducted ab-
lation experiments on the ROC dataset, involving the removal of the feedback
mechanism, omitting Q&A format explanations, and not using a multi-agent ap-
proach. Table 3 demonstrates that the complete MATEval framework surpasses
its ablated versions, confirming the significance of both feedback mechanisms,
explanations and multi-agent methods. Specifically, this establishes the impor-
tance of our feedback mechanisms in promoting discussion consensus and en-
hancing relevance. It also proves that providing scoring explanations by LLMs
significantly enhances evaluation effectiveness.
10
Y. Li et al.
Table 2: Correlation of evaluation results with human judgment using different
models and different strategies of MATEval on the LOT/Ant dataset. The high-
est correlation values are highlighted in bold.
Strategy
REP
LINC
DCONT
ILC
FER
ρ
τ
ρ
τ
ρ
τ
ρ
τ
ρ
τ
LOT
SA
0.829 0.817 0.120
0.110 0.336 0.324 0.179
0.175
0.284
0.279
O b O
0.770 0.764
0.142
0.131 0.249 0.239 0.069
0.066
0.362 0.349
SR
0.751 0.735
0.054
0.048 0.282 0.267 0.118
0.112
0.296
0.284
CoT
0.636 0.628
0.026
0.024 0.215 0.206 0.051
0.049
0.155
0.151
SR+CoT 0.811 0.798
0.197 0.185 0.354 0.341 0.182 0.175
0.341
0.333
Ant
SA
0.522 0.517
0.281
0.275 0.231 0.231 0.318
0.316
0.495
0.489
O b O
0.545 0.538
0.145
0.141 -0.010 -0.010 0.011
0.018
0.347
0.343
SR
0.563 0.557
0.185
0.175 0.069 0.069 0.187
0.184
0.368
0.360
CoT
0.572 0.562
0.034
0.039 0.024 0.024 0.040
0.042
0.358
0.352
SR+CoT 0.694 0.676 0.403 0.377 0.287 0.282 0.424 0.417 0.552 0.544
Table 3: Correlation of evaluation results with human judgment of ablation ex-
periments on the ROC dataset.
Strategy
REP
LINC
DCONT
ILC
FER
ρ
τ
ρ
τ
ρ
τ
ρ
τ
ρ
τ
MATEval-FB
0.567 0.567 0.039 0.038 0.259 0.255 0.266 0.260 0.477 0.473
MATEval-QA
0.612 0.597 0.071 0.068 0.011 0.011 0.132 0.127 0.283 0.281
MATEval-multi 0.699 0.694 0.268 0.253 0.318 0.312 0.240 0.236 0.545 0.538
MATEval
0.735 0.728 0.281 0.264 0.391 0.382 0.263 0.256 0.575 0.561
4.6
Generalization Experiments
To verify the generalizability of our framework across different languages and
in the industrial field, we experimented with two Chinese datasets: LOT and
a story text dataset Ant, derived from Alipay’s business data. The findings in
Table 2 were similar to those from English datasets. Interestingly, single-agent
evaluations often perform better in Chinese, possibly due to its unique language
and sentence structure. This suggests that optimizing agents for different lan-
guages may require tailored adjustments for optimal performance.
4.7
Case Study
To showcase the MATEval framework’s effectiveness in real-world industrial set-
tings, we illustrated its application using a story text example shown in Table
3. We compared our framework against manual evaluation, single-agent evalua-
tion, and the strategy-less One by One evaluation method. Experimental results
MATEval: A Multi-Agent Text Evaluation Framework
11
Story: Bob and Mike wanted to go to the lake for a fishing trip. Bob and Mike had desired to go for a fishing trip to the lake. They packed up and brought
the camper so everyone could stay the night. After eventually arriving there, it began to rainy so heavily that fishing became impossible! With clear skies at
sunrise, they were free to play chess all day. Hoping for better weather in the morning, they went to sleep early.
Error
Type
Strategy
Error
Location
Reason
REP
Human
Bob and Mike had desired to go for a fishing trip
to the lake.
The action of Bob and Mike wanting to go to the lake for a fishing trip
was reiterated unnecessarily.
SA
Bob and Mike wanted to go to the lake for a
fishing trip.
The first two sentences are repetitive as they convey the same
information.
No_Strategy
Bob and Mike had desired to go for a fishing trip
to the lake.
The issue is that the same information is repeated twice, which is
unnecessary and redundant.
MATEval
Bob and Mike had desired to go for a fishing trip
to the lake.
The first two sentences are repetitive as they convey the same
information.
LINC
Human
With clear skies at sunrise, they were free to play
chess all day.
The stated action contradicts the theme of the story, which is fishing.
The original intention was to fish, not to play chess.
SA
With clear skies at sunrise, they were free to play
chess all day.
This sentence is logically inconsistent with the previous sentence "it
began to rainy so heavily that fishing became impossible!" as it implies
that they played chess all day despite the heavy rain.
No_Strategy
/
/
MATEval
With clear skies at sunrise, they were free to play
chess all day.
If the weather cleared up at sunrise, it would have been logical for
they to resume their original plan of fishing, rather than playing chess.
DCONT
Human
Hoping for better weather in the morning, they
went to sleep early.
This sentence originally after the heavy rain disrupted their fishing plans
and before the clear sky at sunrise. It is not chronologically.
SA
/
/
No_Strategy
/
/
MATEval
Hoping for better weather in the morning, they
went to sleep early.
The transition from them hoping for better weather and going to sleep
early, to them playing chess all day is abrupt.
ILC
Human
They packed up and brought the camper so
everyone could stay the night.
The story originally involved only two characters, Bob and Mike. The use
of 'everyone' is inappropriate as it implies the presence of more
characters.
SA
/
/
No_Strategy
/
/
MATEval
They packed up and brought the camper so
everyone could stay the night.
It is not appropriate to use ‘everyone’ in a situation between two people.
FER
Human
/
/
SA
/
/
No_Strategy
/
/
MATEval
/
/
Fig. 3: The schematic diagram illustrating the comparison of results generated
by the MATEval framework and other methods.
demonstrate that our method is basically consistent with human evaluations,
unlike other methods, which show some gaps, thereby confirming the high cor-
relation between our approach and human evaluation.
5
Conclusion
In this paper, we proposed the MATEval framework, which enhances the evalu-
ation performance of open-ended story text generated by LLMs in the industrial
field. Extensive experiments show that MATEval’s evaluation results on two
classic story datasets are more aligned with human preferences than those of
existing methods. In the Alipay industrial scenario, our framework significantly
improves review efficiency and evaluation accuracy, serving as an effective aid.23
In future work, we can fine-tune LLMs as agents in the industrial field to com-
plete specific domain tasks. When solving a complex domain task, we can enable
these domain-specific agents to collaborate with each other, thereby enhancing
their capability and efficiency in addressing challenges.
2 This work was supported by Ant Group
3 This work was supported by the Natural Science Foundation of China (Grant No.
U21A20488). We thank the Big Data Computing Center of Southeast University for
providing the facility support on the numerical calculations in this paper.
12
Y. Li et al.
References
1. Banerjee, S., Lavie, A.: METEOR: an automatic metric for MT evaluation with
improved correlation with human judgments. In: Proceedings of the Workshop
on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or
Summarization@ACL. pp. 65–72 (2005)
2. Callison-Burch, C.: Fast, cheap, and creative: Evaluating translation quality using
amazon’s mechanical turk. In: Proceedings of the 2009 Conference on Empirical
Methods in Natural Language Processing, EMNLP 2009. pp. 286–295
3. Chan, C.M., et al: Chateval: Towards better llm-based evaluators through multi-
agent debate. arXiv preprint arXiv:2308.07201 (2023)
4. Fu, J., et al: Gptscore: Evaluate as you desire. arXiv preprint arXiv:2302.04166
(2023)
5. Ghazarian, S., et al: Better automatic evaluation of open-domain dialogue systems
with contextualized embeddings. arXiv preprint arXiv:1904.10635 (2019)
6. Guan, J., et al: Openmeva: A benchmark for evaluating open-ended story gener-
ation metrics. In: Proceedings of the 59th Annual Meeting of the Association for
Computational Linguistics and the 11th International Joint Conference on Natural
Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers). pp. 6394–
6407 (2021)
7. Guan, J., et al: LOT: A story-centric benchmark for evaluating chinese long text
understanding and generation 10, 434–451 (2022)
8. Guan, J., Huang, M.: UNION: an unreferenced metric for evaluating open-ended
story generation. In: Proceedings of the 2020 Conference on Empirical Methods in
Natural Language Processing, EMNLP. pp. 9157–9166 (2020)
9. Li, G., et al: Camel: Communicative agents for“ mind” exploration of large scale
language model society. arXiv preprint arXiv:2303.17760 (2023)
10. Lin, C.Y.: ROUGE: A package for automatic evaluation of summaries. In: Text
Summarization Branches Out. pp. 74–81 (Jul 2004)
11. Liu, C., et al: How to evaluate your dialogue system: An empirical study of unsu-
pervised evaluation metrics for dialogue response generation. In: Proceedings of the
2016 Conference on Empirical Methods in Natural Language Processing, EMNLP.
pp. 2122–2132 (2016)
12. Liu, Y., et al: G-eval: NLG evaluation using gpt-4 with better human alignment.
In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language
Processing, EMNLP. pp. 2511–2522 (2023)
13. Madaan, A., et al: Self-refine: Iterative refinement with self-feedback. arXiv
preprint arXiv:2303.17651 (2023)
14. Papineni, K., et al: Bleu: a method for automatic evaluation of machine translation.
In: ACL. pp. 311–318 (2002)
15. Wang, J., et al: Is chatgpt a good nlg evaluator? a preliminary study. arXiv preprint
arXiv:2303.04048 (2023)
16. Wang, P., et al: Large language models are not fair evaluators. arXiv preprint
arXiv:2305.17926 (2023)
17. Wei, J., et al: Chain-of-thought prompting elicits reasoning in large language mod-
els. Advances in Neural Information Processing Systems 35, 24824–24837 (2022)
18. Zhang, T., Kishore, V., Wu, F., Artzi, Y.: Bertscore: Evaluating text generation
with BERT. In: 8th International Conference on Learning Representations, ICLR
(2020)


Question:
Which of the following is a major limitation of reference-based evaluation methods like ROUGE and BLEU in text generation assessment?
A. They require extensive computational resources, making them impractical for real-time applications, especially when dealing with large-scale generative models.
B. They are primarily designed for short texts and struggle with coherence in longer narratives, often failing to capture the structural complexity of extended outputs.
C. They rely heavily on the quality and availability of reference texts, which may not always align with diverse outputs, leading to potential misalignment in evaluation.
D. They focus too much on syntactic overlap and may fail to capture semantic nuances, especially when multiple valid outputs exist, resulting in an overemphasis on surface-level similarities rather than deeper meaning.

Answer: D

Compared Methods. Our compared methods can be categorized into two groups: a) Reference-based baselines include traditional overlap-based ROUGE [13], BLEU [16], METEOR [2] as well as PLM-based methods BERTScore [21], BARTScore [19], GPTScore [6] and InstructScore [18]. b) Reference-free baselines feature additional methods like TIGERScore [10], Llama-2-chat [17] and GPT-4 [15].

DEE: Dual-stage Explainable Evaluation
Method for Text Generation
Shenyu Zhang1,2,⋆, Yu Li1,2,⋆, Rui Wu3, Xiutian Huang3, Yongrui Chen1,2,
Wenhao Xu3 (B), Guilin Qi1,2 (B)
1 School of Computer Science and Engineering, Southeast University,
Nanjing 211189, China
2 Key Laboratory of New Generation Artificial Intelligence Technology and its
Interdisciplinary Applications (Southeast University), Ministry of Education, China
{shenyuzhang, yuli 11, yrchen, gqi}@seu.edu.cn
3 Ant Group, Hangzhou, China
{guli.wr, xiutian.hxt, hao.xuwh}@antgroup.com
Abstract. Automatic methods for evaluating machine-generated texts
hold significant importance due to the expanding applications of gener-
ative systems. Conventional methods tend to grapple with a lack of ex-
plainability, issuing a solitary numerical score to signify the assessment
outcome. Recent advancements have sought to mitigate this limitation
by incorporating large language models (LLMs) to offer more detailed
error analyses, yet their applicability remains constrained, particularly in
industrial contexts where comprehensive error coverage and swift detec-
tion are paramount. To alleviate these challenges, we introduce DEE, a
Dual-stage Explainable Evaluation method for estimating the quality of
text generation. Built upon Llama 2, DEE follows a dual-stage principle
guided by stage-specific instructions to perform efficient identification of
errors in generated texts in the initial stage and subsequently delves into
providing comprehensive diagnostic reports in the second stage. DEE is
fine-tuned on our elaborately assembled dataset AntEval, which en-
compasses 15K examples from 4 real-world applications of Alipay that
employ generative systems. The dataset concerns newly emerged issues
like hallucination and toxicity, thereby broadening the scope of DEE’s
evaluation criteria. Experimental results affirm that DEE’s superiority
over existing evaluation methods, achieving significant improvements in
both human correlation as well as efficiency.
Keywords: Text generation evaluation · Large language models · Ex-
plainable metrics.
1
Introduction
Recent advancements in LLMs, like LLaMA [17] and OpenAI’s GPT series [15],
have led to widespread use in various applications, especially in industrial scenar-
ios. One significant challenge is ensuring the quality of the content these models
⋆Equal Contributors.
arXiv:2403.11509v1  [cs.CL]  18 Mar 2024
2
S. Zhang et al.
Supervised Finetuning
Write a social media post on mountain
climbing.
Loving this beautiful weather! Just had
to get outside for a workout.  I felt like a
dog
, running freely and unrestrained.
Machine Output
User Input
ANTEVAL
Reliability
Toxicity
Basic
stereotype
offensive
hatred
...
...
...
DEE
Error 1
Type: offensive language
Location: felt like a dog
Explanation: In some cultures, comparing people to dogs is offensive.
Severity: -3
Error 2
Type: misalignment
Location: running
Explanation: The user asks to write a post on mountain climbing, but
the machine writes on running.
Severity: -5
Final Score: -8
Stage 2
Detailed Error Diagnosis for
Offline Scenarios
Containing errors？ Yes
Error category: Toxicity or Bias, Reliability
Stage 1
Fast Error Detection for
Online Scenarios
Fig. 1. DEE is fine-tuned on AntEval, applying dual-stage strategy to perform fast
error detection in Stage I and provide diagnostic report in Stage II.
generate. For instance, Alipay employs generative systems for the automatic gen-
eration of social media posts, but this raises issues like potential toxicity [4] or
incoherence [5] in the content. Given the impracticality of human evaluation for
such services with millions of users, there is a growing need for reliable automatic
evaluation methods. These methods are crucial for maintaining content quality,
thereby enhancing user experience on platforms that utilizes generative models.
Existing methods [6,10,13,18,19,21] for text generation primarily focus on ba-
sic aspects but inadequately address recent emerging challenges. These methods,
though providing quantitative scores, lack the ability to offer detailed feedback
and face latency problems in real-time applications. Their weaknesses include:
Limited Evaluation Dimensions. Predominantly, existing methodologies con-
centrate on conventional aspects such as fluency and consistency. However, as
LLMs evolve, more sophisticated and fluent outputs are generated, bringing forth
novel challenges like hallucinations [9], biases and toxicity [4]. These issues are
not adequately addressed by existing evaluation frameworks.
Deficient Explainability. Existing methods like ROUGE [13] and BERTScore
[21] primarily provide quantitative scores without explanatory feedback. This
lack of detailed analysis on the types and causes of errors in generated texts
hinders the improvement of generative systems in the context offline development
and reduces the reliability and interpretability of these evaluation methods.
Lack of Efficiency. In online applications, rapid evaluation is crucial to iden-
tify and prevent poor text generation outcomes in real-time. LLM-based meth-
ods, such as InstructScore [18] and TIGERScore [10], provide detailed diagnostic
reports but suffer from significant latency issues due to the inherent inefficiency
of LLM inference. This limits their effectiveness in time-sensitive environments.
In this paper, we introduce DEE, a Dual-stage Explainable Evaluation method
for text generation in industrial scenarios. DEE leverages Llama 2 and operates in
two stages, as shown in Fig. 1. Initially, it quickly identifies and classifies errors
in generated text into principal categories, allowing for rapid inference suitable
for real-time applications. The second stage, powered by our assembled An-
DEE: Dual-stage Explainable Evaluation Method for Text Generation
3
tEval dataset, conducts an in-depth analysis of each error, providing detailed
explanations. AntEval encompasses 15K examples from 4 real-world applica-
tions of Alipay based on generative systems. By including the newly emerged
issues mentioned above, AntEval enables DEE to perform comprehensive eval-
uations. Our experimental results on 4 tasks demonstrate that DEE represents a
substantial leap forward in automatically evaluating text generation, promising
heightened correlation with human ratings and operational efficiency. In sum-
mary, the contributions of this paper include:
– We present an innovative dual-stage evaluation method for text generation
in industrial scenarios. By decomposing the evaluation process, DEE ensures
capability of the LLM-based method to conduct efficient error detection in
real-time online applications as well as provide explainable error analysis.
– We introduce a dataset derived from real-world industrial applications con-
taining recently emerged problems of generative systems. It facilitates the
development of a text generation evaluation method that encompasses multi-
aspect evaluation dimensions and comprehensive error coverage.
– Experimental results on the real-world dataset elaborate the superiority of
our method compared to existing competitors, achieving state-of-the-art per-
formance in industrial scenarios.
2
Preliminaries
2.1
Text Generation Evaluation
Given a source input text sequence X = ⟨x1, x2, . . . , xj⟩and the corresponding
output Y′ = ⟨y′
1, y′
2, . . . , y′
k⟩produced by generative systems, the goal of a text
generation evaluation method is to produce a score S = Fθ(X, Y′) as the quality
estimation, where Fθ denotes the evaluation method. Noted that some methods
[2,13,16,18,21] depend on a reference output Y for scoring, called reference-based
methods. And our method only relies on Y′, which is reference-free.
2.2
Problem Formulation
Our goal is to learn an explainable method to provide error analysis along with
quality scores. Taking X and Y′ as the input, we aim to train Fθ : (X, Y′) −→
{S, R}. R = {(T , L, E)}n
i=1 denotes the analysis report for S, containing error
type T , location L and explanation E for each of the n errors in Y′.
3
The AntEval Dataset
We introduce AntEval, a dataset featuring real-world user interactions and re-
sponses generated by LLMs from Alipay’s live applications. AntEval is built on
three key principles: a) Error Comprehensiveness: It encompasses not only
traditional evaluation dimensions like fluency and coherence but also addresses
4
S. Zhang et al.
basic
toxicity
&
bias
reliability
not fluent
incoherent
inconsistent
nonsensical
stereotype
offensive language
hatred
sexual discrimination
region discrimination
racial discrimination
misalignment
extrinsic hallucination
intrinsic hallucination
non-factual
misinformation
SMPG
DG
TP
SG
Reliability
Bias & Toxicity
Basic
Error count per example
0.5
1.0
1.5
2.0
Fig. 2. Left: The distribution of error categories in AntEval. The inner circle depicts
the principle categories and the outer circle depicts the corresponding sub-error cat-
egories. Right: Error distribution in each task, calculated by Nerror / Nexample. Here
we count the number of sub-errors, which may occur more than once in one example.
newly emerged issues such as bias, toxicity, and hallucination. b) Task Diver-
sity: AntEval contains examples from four distinct NLG applications of Alipay,
each with its specific purpose, and involves outputs from various LLM-based sys-
tems including LLaMA [17], ChatGLM [20], and Qwen [1]. c) Explainability:
For each example in AntEval, we utilize OpenAI GPT-4 [15] by prompting it to
detect and analyze the errors as well as give corresponding explanations.
3.1
Comprehensive Evaluation Dimensions
As LLMs evolve, their generated content becomes more natural. However, recent
research [9,14] points out that they often produce undesired or socially disruptive
content, which is a critical issue especially in user-facing industrial applications.
To address this, we introduce AntEval, designed to evaluate errors in LLM-
generation related to social norms and human alignment. We define a set CM
containing 3 principle errors categories: a) Reliability: Errors where the text
is inaccurate, hallucinatory, inappropriate, or misunderstands user intent. b)
Bias and Toxicity: Instances where the text contains stereotypes, offends, or
is hateful towards certain user groups. c) Basic Errors: Problems with flu-
ency, coherence, consistency, or nonsensical content due to language misuse or
repetition. Detailed categorization and distribution are depicted in Fig. 2 Left.
3.2
Diverse Task Sources
15K examples are collected from 4 real-world applications of Alipay to assemble
AntEval: a) Social Media Post Generation (SMPG) — 30%: SMPG
involves crafting engaging content for the Alipay social platform based on spe-
cific input scenarios or themes; b) Dialogue Generation (DG) — 30%: DG
focuses on generating contextually relevant responses in a conversation, ensuring
DEE: Dual-stage Explainable Evaluation Method for Text Generation
5
coherence and engagement.; c) Text Paraphrase (TP) — 10%: TP involves
rewriting input text to enhance richness and variety while preserving the origi-
nal semantic information; d) Story Generation (SG) — 30%: SG is the task
of creating coherent narratives based on specified prompts or settings, focusing
on plot development and long text generation. All raw examples are in the for-
mat of user input X — machine output Y′ pairs. Specifically, a raw example
e = {X, Y′}. The error distribution in each task is shown in Fig. 2 Right.
3.3
Explainable Error Diagnosis
In the construction of the AntEval dataset, a significant emphasis is placed on
the explainability aspect of error diagnosis. In this paper, OpenAI’s GPT-4 is
utilized to produce diagnostic reports. We wrap the raw examples by our care-
fully designed prompting template T to formulate the input of GPT-4: T (X, Y′).
In T , we provide GPT-4 with the detailed definitions of the pre-defined eval-
uation dimensions described in Section 3.1. GPT-4 is required to determine
the principle error categories existing in Y′ and provide a further report on the
errors. Hence that we require GPT-4 to produce a severity score lies in [-5, -1]
for each error in the report and get the final score by adding the severity scores.
3.4
Human Evaluation for Test Set Creation
We randomly sample 1,000 examples to formulate the test set of AntEval. To
get the real human preference score as the gold standard, we organize human
experts to rate the generated texts. The experts are required to score from 0 to
5 across 3 dimensions. The annotation interface is shown in Fig. 3. To compare
with reference-based methods, we use GPT-4 to generate reference texts for each
instance (X), producing 3 outputs per instance. GPT-4’s outputs are assessed
together with the real system outputs. The highest-scoring outputs (usually
above 4.5) are chosen as the golden references. Thus, the examples in AntEval’s
test set are enriched with both human preference scores and reference texts.
Read the user input and the corresponding machine outputs 1-4.
Please rate the following machine-generated text from the following dimensions:
•
Reliability.
•
Bias or Toxicity.
•
Basic Dimensions.
Rate the machine-generated text from the above perspectives, with scores
ranging from 1 to 5, where a higher score represents higher quality of the text in
that dimension.
Annotation Guide
Given the following user input:
The system outputs are:
Examples to Evaluate
Please write a social media post on Mountain Climbing
Output 1: Loving this beautiful ...
Output 2: Just conquered the summit ...
Output 3: Sunrise from Everest Base ...
Output 4: Embracing the challenge ...
1 real-world system output + 3 GPT-4 outputs
Please rate the machine-generated text from the following dimensions:
•
Reliability. The generated text is accurate, conforms to facts without containing hallucinations,
matches the user input, and is the result that the user hopes to obtain.
•
Bias or Toxicity. The generated text does not contain stereotypes, conforms to social norms, and
does not show disrespect, offense, or hatred towards certain user groups.
•
Basic Dimensions. The generated text does not contain misuse of vocabulary, repetitive
expressions, etc. The text is fluent, coherent, consistent, and meaningful.
Rating Interface
1
2
3
5
4
1
2
3
5
4
1
2
3
5
4
1
2
3
5
4
1
2
3
5
4
Please rate the machine-generated text from the following dimensions:
•
Reliability. The generated text is accurate, conforms to facts without containing hallucinations,
matches the user input, and is the result that the user hopes to obtain.
•
Bias or Toxicity. The generated text does not contain stereotypes, conforms to social norms, and
does not show disrespect, offense, or hatred towards certain user groups.
•
Basic Dimensions. The generated text does not contain misuse of vocabulary, repetitive
expressions, etc. The text is fluent, coherent, consistent, and meaningful.
Rating Interface
1
2
3
5
4
1
2
3
5
4
1
2
3
5
4
•
Reliability:
5
•
Bias or Toxicity:
2
•
Basic Dimensions:
4
•
Overall:
3.67
Annotation for Output 1
•
Reliability:
5
•
Bias or Toxicity:
5
•
Basic Dimensions:
4
•
Overall:
4.67
Annotation for Output 4
···
Read the user input and the corresponding machine outputs 1-4.
Please rate the following machine-generated text from the following dimensions:
•
Reliability.
•
Bias or Toxicity.
•
Basic Dimensions.
Rate the machine-generated text from the above perspectives, with scores
ranging from 1 to 5, where a higher score represents higher quality of the text in
that dimension.
Annotation Guide
Given the following user input:
The system outputs are:
Examples to Evaluate
Please write a social media post on Mountain Climbing
Output 1: Loving this beautiful ...
Output 2: Just conquered the summit ...
Output 3: Sunrise from Everest Base ...
Output 4: Embracing the challenge ...
1 real-world system output + 3 GPT-4 outputs
Please rate the machine-generated text from the following dimensions:
•
Reliability. The generated text is accurate, conforms to facts without containing hallucinations,
matches the user input, and is the result that the user hopes to obtain.
•
Bias or Toxicity. The generated text does not contain stereotypes, conforms to social norms, and
does not show disrespect, offense, or hatred towards certain user groups.
•
Basic Dimensions. The generated text does not contain misuse of vocabulary, repetitive
expressions, etc. The text is fluent, coherent, consistent, and meaningful.
Rating Interface
1
2
3
5
4
1
2
3
5
4
1
2
3
5
4
•
Reliability:
5
•
Bias or Toxicity:
2
•
Basic Dimensions:
4
•
Overall:
3.67
Annotation for Output 1
•
Reliability:
5
•
Bias or Toxicity:
5
•
Basic Dimensions:
4
•
Overall:
4.67
Annotation for Output 4
···
Fig. 3. The annotation interface for human experts to score generated texts.
6
S. Zhang et al.
4
The DEE Method
DEE is adept at identifying a diverse array of errors, including, but not limited
to hallucination, linguistic errors, and issues related to biases and toxicity. This
is achieved through the integration of the AntEval dataset, which encompasses
examples from real-world applications, reflecting a multitude of error types that
are encountered in practical scenarios. This dataset ensures that DEE is not only
trained to detect common errors but is also sensitive to complex and nuanced
issues that are increasingly prevalent in advanced generative models.
Further, DEE employs a dual-stage evaluation strategy to perform evaluation
for generated texts. In Stage I, it detects and categorizes errors in the generated
text. Stage II involves producing a detailed error analysis report, offering ex-
plainable insights crucial for continuous improvement of generative systems.
4.1
Instruction-guided Dual-stage Evaluation
DEE implements an instruction-guided dual-stage evaluation, leveraging a singu-
lar backbone Pre-trained Language Model (PLM) denoted as Fθ where θ are the
tunable parameters of F. This process is directed by stage-specific instructions,
namely Ii and Iii, uniquely tailored to each respective stage. The operational
mechanics of DEE are delineated as follows:
– Stage I: In Stage I of the DEE methodology, the primary focus is on the swift
detection of errors in machine-generated text. This initial phase is critically
designed to be both rapid and accurate, recognizing the essential need for
prompt evaluations in time-sensitive industrial applications.
DEE utilizes Ii to ascertain the principle error categories present within
the evaluated text, if any. This is formally represented as:
ˆC = Fθ(Ii, X, Y′)
(1)
Here, ˆC ⊆CM, which are predefined principle errors outlined in Section
3.1. This categorization is typically achieved using less than 10 tokens, a
feature that significantly contributes to the system’s rapid inference capabil-
ities. By efficiently narrowing down the error types in Stage I, DEE sets the
groundwork for a more detailed analysis in the subsequent phase, ensuring
that the overall evaluation process remains both time-efficient and thorough.
– Stage II: In Stage II of the DEE framework, the system transitions from
rapid error detection to providing an explainable error analysis along with
the quality score. In this stage, DEE delves into a detailed examination of the
errors identified in Stage I. DEE generates an exhaustive diagnostic report,
leveraging the identified error categories ˆC. The quality score and report are
formulated as follows:
{ ˆS, ˆR} = { ˆS, {(T , L, E)}n
i=1} = Fθ(Iii, ˆC, X, Y′)
(2)
DEE: Dual-stage Explainable Evaluation Method for Text Generation
7
The report illuminates the underlying causes of each error, contextualizing
them within the specific content and structure of the text. This level of detail
allows for a nuanced understanding of why certain errors occur, offering in-
depth insights.
This dual-stage approach, governed by bespoke instructions at each phase,
enables DEE to conduct efficient yet nuanced evaluation of machine-generated
texts, aligning with the rigorous standards expected in industrial applications.
4.2
Training Strategies
DEE leverages Llama-2-7B, an open-source PLM as the backbone. Incorporating
the AntEval dataset, DEE adopts a simultaneous training approach, wherein
examples from both stages are intermixed during the training process. The train-
ing objectives for instances from each stage are specified as follows:
Li(C, Ii, X, Y′) = −logP(C|Ii, X, Y′; θ)
Lii(R, Iii, C, X, Y′) = −logP(R|Iii, C, X, Y′; θ)
(3)
5
Experiments
5.1
Experimental Setup
Dataset Our experiments involve AntEval, a crafted dataset containing ex-
amples from 4 real-world applications of Alipay, as is described in Section 3. Our
method as well as all baselines are evaluated on the test set of AntEval.
Evaluation Metrics. We evaluate DEE from two aspects: a) Correlation with
Human Judgement: We use Kendall’s Tau τ and Pearson’s correlation coeffi-
cient ρ, as per AntEval’s human preference scores, to assess the correlation
between automated evaluation methods and human judgments. Kendall’s Tau,
suitable for ordinal data, analyzes the association between ranked variables.
Pearson’s correlation evaluates the linear relationship between continuous vari-
ables. Applying both methods offers comprehensive insight into the alignment of
automatic methods with human standards. b) Qualitative Human Evaluation:
The validity of the error analysis performed by DEE was appraised through ex-
pert review. These professionals were tasked with evaluating the extent of error
coverage and Veridicality Rate in DEE’s outputs, as detailed in Section 5.4.
Compared Methods. Our compared methods can be categorized into two
groups: a) Reference-based baselines include traditional overlap-based ROUGE
[13], BLEU [16], METEOR [2] as well as PLM-based methods BERTScore [21],
BARTScore [19], GPTScore [6] and InstructScore [18]. b) Reference-free base-
lines feature additional methods like TIGERScore [10], Llama-2-chat [17] and
8
S. Zhang et al.
Table 1. Experimental results for comparison with baselines on AntEval. Kendall’s
Tau τ (%) and Pearson’s correlation coefficient ρ (%) are reported.
Category
Method
SMPG
DG
TP
SG
Avg.
τ
ρ
τ
ρ
τ
ρ
τ
ρ
τ
ρ
w/
Reference
ROUGE [13]
22.0 17.7 12.7
4.0
20.6 29.1 −0.1 −6.3 15.1 10.0
BLEU [16]
23.6 18.4
5.8 −12.8 5.4
7.4 −2.3 −8.7 11.8
2.4
METEOR
30.4 29.4 13.2
4.1
14.8 22.6
8.5
7.2
19.7 16.6
BERTScore [21]
36.9 35.4 17.6
10.9
22.6 30.5
3.8
0.3
23.7 20.8
BARTScore [19]
28.6 25.8 11.0
2.8
12.4 21.0 27.3 33.0 21.1 18.7
BARTScore-para [19]
30.2 29.7 14.1
8.7
15.2 23.6 23.0 27.7 22.3 21.7
GPTScore [6]
20.9 30.6
7.9
13.5 24.6 30.9 13.8 25.6 15.6 23.9
InstructScore [18]
31.8 33.0
9.1
23.8
16.7 18.3 25.4 25.5 21.7 27.5
w/o
Reference
BARTScore* [19]
26.0 35.3 24.7
29.2
4.8
18.4 −5.3 −3.9 20.1 27.0
BARTScore-para* [19] 22.7 28.5 15.0
19.1
6.4
7.9 −2.4 −3.5 14.7 18.4
GPTScore* [6]
32.8 36.5 −0.6 −0.4 17.1 23.2
9.5
10.2 16.4 18.5
TIGERScore [10]
29.9 45.0 39.3
48.4
18.2 18.3 11.3 18.4 28.8 37.1
Llama-2-chat [17]
20.1 37.6 28.6
39.4
9.1
18.2
8.8
2.2
19.7 27.3
GPT-4 [15]
46.5 51.3 42.5
50.1
21.2 27.6 48.5 50.1 42.1 48.1
DEE (Ours)
51.8 56.6 48.2 56.4 18.8 20.4 52.5 57.1 47.2 53.7
GPT-4 [15]. For GPTScore, we utilize FLAN-T5-large [3] as the backbone.
BARTScore offers two versions of checkpoints, one is based on BART-base [12]
and one fine-tuned using the ParaBank2 [8] dataset. We present results for both
versions in our analysis. For Llama-2-chat and GPT-4, we directly prompt them
to generate scores within a range of 0 to 5, serving as the evaluation results.
Noted that BARTScore and GPTScore can also function as reference-free meth-
ods by employing a source — hypothesis evaluation format [6,19].
Implementation Details. Our method runs on single NVIDIA RTX 3090
GPU. LoRA [7] is employed for training under the following hyper-parameter
settings: the training batch size is set to 16 and the maximum input length is
limited to 2048. The model is trained for 3 epochs with a learning rate of 1e-4.
5.2
Main Results
In our comparative analysis, summarized in Table 1, we evaluate DEE’s aver-
age performance across various tasks. DEE generally outperforms other methods
in Pearson’s correlation and Kendall’s Tau. An exception is noted in the Text
Paraphrase task, where DEE scores slightly lower. This may due to the minimal
semantic differences between user inputs and system outputs in this task and the
prevalence of Basic errors, areas where conventional methods are effective. No-
tably, DEE significantly excels in other tasks, particularly in Story Generation,
affirming its strength in processing long texts.
DEE: Dual-stage Explainable Evaluation Method for Text Generation
9
Compared to reference-based methods, DEE substantially performs better in
the majority of cases, which demonstrate its advanced understanding of textual
nuances. As for recent LLM-based methods [10,18], DEE emerges as the superior
choice, thanks to its comprehensive consideration of contemporary challenges like
hallucination and toxicity. Additionally, we employ GPT-4 as a powerful base-
line. The findings reveal that DEE maintains its competitive edge even against
such large-scale language models, highlighting its effectiveness.
5.3
Efficiency in Error Detection
In this section, we assess the proficiency of DEE in detecting errors, particu-
larly focusing on its applicability in real-world, online settings. The methods
are tasked with determining if generated texts contains errors. To simulate this
scenario, we assign a binary score (0 or -1) to represent accurate/erroneous ex-
amples.
We benchmark DEE against other PLM-based methods. For methods based
on the LLaMA architecture [10, 18], we utilize VLLM [11] to enhance infer-
ence speed. The comparative results are depicted in Figure 4. In this setup,
BERTScore emerges as the quickest in terms of inference speed due to its smallest
model scale. However, two LLM-based methods, InstructScore and TIGERScore,
exhibit inference times exceeding 200ms per example, which is impractical for
online applications. Through its dual-stage approach, DEE achieves an inference
speed on par with methods based on smaller language models like T5-base and
BART-base by employing its Stage I only. Importantly, DEE maintains superior
correlation with human judgment even when Stage II is not involved.
SMPG DG
TP
SG
Avg.
101
102
103
Time per Example (ms)
SMPG DG
TP
SG
Avg.
0
20
40
60
ρ (%)
BERTScore
BARTScore
BARTScore*
BARTScore-para
BARTScore-para*
GPTScore
GPTScore*
InstructScore
TIGERScore
DEE
SMPG DG
TP
SG
Avg.
0
20
40
60
τ (%)
Fig. 4. Experiments for comparison with PLM-based methods. Inference time per ex-
ample (ms), Kendall’s Tau τ (%) and Pearson’s correlation coefficient ρ(%) are re-
ported.
5.4
Qualitative Human Evaluation
In Stage II, DEE produces error analysis in human-readable reports, augment-
ing quantitative scores with explanatory insights. For assessing the quality of
the analysis, we engage human experts, whose routine involves reviewing and
10
S. Zhang et al.
evaluating textual materials, to appraise the reports based on following criteria,
using the standards outlined in AntEval’s test set as benchmarks:
Error Coverage (EC). Experts assess if the reports include all identified
errors. EC measures how well reports identify all errors in generated texts. For
K such reports, EC =
1
K
PK
i=1 1(Ei
g ⊆Ei
p), where Ei
p represents the set of
predicted sub-errors and Ei
g represents the gold standards. A higher EC value
indicates a higher recall of errors.
Veridicality Rate (VR). In a complementary manner, experts also evalu-
ate if the reported errors genuinely exist. VR assesses the accuracy of identified
errors in reports, focusing on actual errors and excluding false ones. VR is cal-
culated as VR =
1
K
PK
i=1 1(Ei
p \ Ei
g = ∅). A higher VR indicates a greater
precision in error identification and less hallucination.
SMPG DG
TP
SG
Avg.
88
90
92
EC (%)
Stage I
SMPG DG
TP
SG
Avg.
82
84
86
EC (%)
Stage II
SMPG DG
TP
SG
Avg.
90
95
VR (%)
Stage I
SMPG DG
TP
SG
Avg.
86
88
90
92
VR (%)
Stage II
SMPG
DG
TP
SG
Avg. Reliability
Toxicity
Basic
0.8
0.9
1
EC’ (%)
Fig. 5. Left Four: EC (%) and VR (%) across different tasks. Right: EC’ (%) across
different error categories.
Based on the above results, we analysis from the following aspects:
EC & VR across Tasks. Evaluation results are illustrated in Fig. 5 Left.
Noted that we also include evaluation on Stage I, the only difference lies in
Eg, which is the subset of principle errors predefined in Section 3.1. In both
stages, DEE attains an impressive EC of (89.4%/83.2%), suggesting that it suc-
cessfully identifies the majority of errors. Concurrently, the VR for DEE is at
(93.3%/89.1%), indicating a high accuracy in error detection with minimal in-
stances of hallucinations. A notable observation is that DEE achieves the highest
EC in Dialogue Generation and, conversely, the lowest in Story Generation. This
trend may be attributed to the greater number of errors per example in Story
Generation whereas Dialogue Generation examples exhibit a lower error count,
suggesting that DEE may encounter limitations when facing long-tail examples.
EC across Error Categories. In our further analysis of DEE’s perfor-
mance across various error categories. We calculate EC′i = N i
ptrue/N i
g for each
Ci ∈CM, here N i
ptrue denotes the number of errors correctly predicted and N i
g
denotes the the total number of errors in that category. Results are illustrated
in Fig. 5 Right. DEE generally exhibits robust performance across different
evaluation dimensions, while it demonstrates slightly lower coverage in the Re-
liability category. This indicates that accurately evaluating errors related to
human alignment and textual hallucinations remains a challenging task.
DEE: Dual-stage Explainable Evaluation Method for Text Generation
11
5.5
Ablation Study
We compare the performance of our proposed DEE in the following settings:
– w/o Stage I. We exclude Stage I of DEE by altering the training examples
derived from the AntEval dataset. Specifically, the model is instructed to
conduct error analysis without first making predictions about the principal
errors.
– w/o Stage II. We exclude Stage II examples in the training stage. By doing
so, the model is limited to conducting inferences pertaining to Stage I only.
This restriction inhibits the model’s ability to generate explanations.
Table 2 presents the τ, ρ, EC and VR of different settings. Our proposed DEE,
employing the complete dual-stage strategy, exhibit the best performance. The
results indicate that each stage of the model mutually enhances the other. Omit-
ting either stage leads to a notable decline in performance, with a 2% to 6%
reduction in correlation with human judgment. Intriguingly, we observe that
decomposing the evaluation process into two distinct stages not only expands
the scope of error detection but also substantially eliminates hallucinations, as
evidenced by the increases in EC and VR. In conclusion, the two stages of the
DEE function synergistically, each complementing and reinforcing the other.
Table 2. Experimental results for comparison with baselines on AntEval. Correlation
with human and qualitative evaluation are reported.
Stage
Method
Avg.
τ
ρ
EC
VR
First Stage
w/o Stage II
48.8
51.7
86.3
91.2
DEE
51.5
53.2
89.4
93.3
Second Stage
w/o Stage I
41.9
49.4
78.5
84.3
DEE
47.2
53.7
83.2
89.1
6
Conclusion
In this paper, we propose DEE, a novel method for text generation evaluation
in industrial settings. It utilizes a dual-stage process incorporating Llama 2 and
the AntEval dataset. While effective in rapid error detection and in-depth
analysis, a limitation of DEE is its potential difficulty in handling texts with
an excessive number of errors, which could challenge its error categorization and
analysis efficiency. Despite this, DEE represents a significant advancement in text
evaluation, combining efficiency with comprehensive, explainable assessments.
This work was supported by Ant Group Research Fund.
12
S. Zhang et al.
References
1. Bai, J., Bai, S., Chu, Y., Cui, Z., et al.: Qwen technical report. arXiv (2023)
2. Banerjee, S., Lavie, A.: METEOR: an automatic metric for MT evaluation with
improved correlation with human judgments. In: Goldstein, J., Lavie, A., Lin, C.,
Voss, C.R. (eds.) Proceedings of the Workshop on Intrinsic and Extrinsic Eval-
uation Measures for Machine Translation and/or Summarization@ACL 2005. pp.
65–72 (2005)
3. Chung, H.W., Hou, L., Longpre, S., Zoph, B., et al.: Scaling instruction-finetuned
language models. arXiv (2022)
4. Deshpande, A., Murahari, V., Rajpurohit, T., Kalyan, A., Narasimhan, K.: Toxi-
city in chatgpt: Analyzing persona-assigned language models. arXiv (2023)
5. Dziri, N., Kamalloo, E., Mathewson, K.W., Za¨ıane, O.R.: Evaluating coherence in
dialogue systems using entailment. In: Burstein, J., Doran, C., Solorio, T. (eds.)
NAACL-HLT. pp. 3806–3812 (2019)
6. Fu, J., Ng, S., Jiang, Z., Liu, P.: Gptscore: Evaluate as you desire. arXiv (2023)
7. Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., et al.: Lora: Low-rank adaptation of
large language models. In: ICLR (2022)
8. Hu, J.E., Singh, A., Holzenberger, N., Post, M., Durme, B.V.: Large-scale, diverse,
paraphrastic bitexts via sampling and clustering. In: Bansal, M., Villavicencio, A.
(eds.) CoNLL. pp. 44–54 (2019)
9. Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., et al.: Survey of hallucination in natural
language generation. ACM Computing Surveys 55(12), 1–38 (2023)
10. Jiang, D., Li, Y., Zhang, G., Huang, W., Lin, B.Y., Chen, W.: Tigerscore: Towards
building explainable metric for all text generation tasks. arXiv (2023)
11. Kwon, W., Li, Z., Zhuang, S., Sheng, Y., et al.: Efficient memory management
for large language model serving with pagedattention. In: Flinn, J., Seltzer, M.I.,
Druschel, P., Kaufmann, A., Mace, J. (eds.) SOSP. pp. 611–626 (2023)
12. Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., et al.: BART: denoising sequence-
to-sequence pre-training for natural language generation, translation, and compre-
hension. In: Jurafsky, D., Chai, J., Schluter, N., Tetreault, J.R. (eds.) ACL. pp.
7871–7880 (2020)
13. Lin, C.Y.: ROUGE: A package for automatic evaluation of summaries. In: Text
Summarization Branches Out. pp. 74–81 (Jul 2004)
14. Liu, Y., Yao, Y., Ton, J.F., Zhang, X., et al.: Trustworthy llms: a survey and
guideline for evaluating large language models’ alignment. arXiv (2023)
15. OpenAI: GPT-4 technical report. arXiv (2023)
16. Papineni, K., Roukos, S., Ward, T., Zhu, W.: Bleu: a method for automatic eval-
uation of machine translation. In: ACL. pp. 311–318 (2002)
17. Touvron, H., Lavril, T., Izacard, G., Martinet, X., et al.: Llama: Open and efficient
foundation language models. arXiv (2023)
18. Xu, W., Wang, D., Pan, L., Song, Z., Freitag, M., Wang, W., Li, L.: IN-
STRUCTSCORE: towards explainable text generation evaluation with automatic
feedback. In: Bouamor, H., Pino, J., Bali, K. (eds.) EMNLP. pp. 5967–5994 (2023)
19. Yuan, W., Neubig, G., Liu, P.: Bartscore: Evaluating generated text as text gener-
ation. In: Ranzato, M., Beygelzimer, A., Dauphin, Y.N., Liang, P., Vaughan, J.W.
(eds.) NeurIPS. pp. 27263–27277 (2021)
20. Zeng, A., Liu, X., Du, Z., Wang, Z., et al.: Glm-130b: An open bilingual pre-trained
model. In: ICLR (2022)
21. Zhang, T., Kishore, V., Wu, F., Weinberger, K.Q., Artzi, Y.: Bertscore: Evaluating
text generation with BERT. In: ICLR (2020)