Question:
The study compares two synthetic data configurations for training humor classifiers: [Synthetic unfun; Original satire] and [Human unfun; Synthetic satire], and applies MISTRAL fine-tuning with 4-bit QLoRA and a classification head, AdamW with a constant LR, and class-weighted loss for the English–Hindi task (39% non-humorous, 61% humorous). Which statements align with the methodological details and their implications?
A. Using class-weighted loss is appropriate for the English–Hindi dataset to counteract minority-class underrepresentation (non-humor 39%), reducing bias toward predicting humor.
B. QLoRA fine-tuning with an added classification head enables parameter-efficient adaptation of MISTRAL while preserving base model weights, which is advantageous given large-batch sweeps (up to 256) in hyperparameter tuning.
C. Evaluating synthetic data by training binary detectors to separate original satire from unfunned outputs directly tests generation quality via distributional alignment.
D. The near-zero overlap of synthetically generated samples with the original Unfun data (e.g., GPT-4: 7/200 in Unfun, 0/200 in Satire) guarantees absence of train–test leakage and thus unbiased estimates of generalization.

Answer: ABC

First, following recent work on synthetic data (Li et al., 2023; Veselovsky et al., 2023) we evaluate the data quality of outputs from LLMs by testing whether binary humor classifiers trained on the synthetic outputs can differentiate between actual humorous and unfunned headlines from the original Unfun dataset. We compare training on data from human players and actual satirical headlines to two configurations of synthetic data:
[Synthetic unfun; Original satire]
[Human unfun; Synthetic satire]
These two configurations enable comparing the “unfunning” and joke writing capabilities of LLMs. Additionally, we consider the alternative of using actual unrelated news headlines as non-humorous examples. Using data from each approach, we fine-tune ROBERTA and MISTRAL-7B for humor classification. Our test set comprises a subset of headline pairs from the Unfun corpus that were highly rated in the original game. Additional evaluation details are provided in Appendix A.4.

For the Unfun task, we fine-tune MISTRAL (Jiang et al., 2023)6 and ROBERTA (Liu et al., 2019)7 models. For Hindi-English, we consider HING-ROBERTA (Nayak and Joshi, 2022)8. All models are trained with the AdamW optimizer (Loshchilov and Hutter, 2019) and a constant learning rate. Due to the class imbalance in the available English-Hindi dataset (39% non-humorous, 61% humorous), we weight the loss by the inverse proportion of class frequency.

We fine-tune our MISTRAL classifier with 4-bit quantized LoRA (Dettmers et al., 2023) and the addition of a classification head. For all classifiers, we first perform hyperparameter tuning on the original human authored datasets.

For the Unfun dataset we consider:
• Learning Rates ∈{5e −5, 2.5e −5, 1.25e −5, 6.25e −6, 3.125e −6, 1.5625e −6}
• Batch Size ∈[32] (Due to resource constraints)
For the English-Hindi Dataset dataset we consider:
• Learning Rates ∈{5e −5, 2.5e −5, 1.25e −5, 6.25e −6, 3.125e −6, 1.5625e −6}
• Batch Size ∈{256, 128, 64, 32, 16, 8}
After selecting the highest performing configuration, we run each experiment with 5 seeds ([1234, 2345, 3456, 4567, 5678]). We include the most performant hyperparameters in Table 7. All model trains use a single NVIDIA A100 GPU. We estimate the total compute budge to be 200 hours.

Getting Serious about Humor:
Crafting Humor Datasets with Unfunny Large Language Models
Zachary Horvitz1,*, Jingru Chen1,*, Rahul Aditya1, Harshvardhan Srivastava1,
Robert West2, Zhou Yu1, Kathleen McKeown1
1Columbia University, 2EPFL
{zfh2000, jc5898, ra3261, hs3447, zy2461}@columbia.edu
robert.west@epfl.ch, kathy@cs.columbia.edu
Abstract
Humor is a fundamental facet of human cog-
nition and interaction. Yet, despite recent ad-
vances in natural language processing, humor
detection remains a challenging task that is
complicated by the scarcity of datasets that
pair humorous texts with similar non-humorous
counterparts. We investigate whether large lan-
guage models (LLMs) can generate synthetic
data for humor detection via editing texts. We
benchmark LLMs on an existing human dataset
and show that current LLMs display an im-
pressive ability to “unfun” jokes, as judged by
humans and as measured on the downstream
task of humor detection. We extend our ap-
proach to a code-mixed English-Hindi humor
dataset where we find that GPT-4’s synthetic
data is highly rated by bilingual annotators and
provides challenging adversarial examples for
humor classifiers.
1
Introduction
Despite their success on natural language tasks,
large language models (LLMs) struggle to reliably
detect and explain humor (Baranov et al., 2023;
Góes et al.; Hessel et al., 2023), and generate novel
jokes (Jentzsch and Kersting, 2023). Notably, hu-
mans also struggle to write jokes; even at satiri-
cal newspapers like The Onion, less than 3% of
proposed headlines are printed (West and Horvitz,
2019; Glass, 2008). In contrast, humans are able
to consistently edit jokes to unfun them, an in-
sight which motivated West and Horvitz (2019)
to host a game where internet users competed to
edit satirical headlines to make them serious. The
resulting dataset, the Unfun Corpus (West and
Horvitz, 2019), has been a valuable tool for advanc-
ing computational humor research. The dataset
has been used to study properties of both humor
and transformer architectures (West and Horvitz,
2019; Peyrard et al., 2021) and even to generate
*Equal contribution.
Figure 1: Outputs from GPT-4. We leverage language
models to edit away (or "unfun") humor in existing
human-written jokes, resulting in aligned datasets that
pair humorous texts with non-humorous counterparts.
novel satire (Horvitz et al., 2020). Additionally,
recent work has found that despite the relatively
small size of the original dataset, humor detection
models trained on Unfun data generalize remark-
ably well to other datasets, while models trained on
other humor datasets perform poorly at classifying
Unfun-edited data (Baranov et al., 2023).
While useful contributions, Unfun and other
aligned humor datasets (Hossain et al., 2019, 2020)
are limited in both size and scope, due to their re-
liance on human annotation. We investigate the
alternative of using LLMs to create datasets of
aligned humorous and non-humorous texts.1 Pre-
vious work (Jentzsch and Kersting, 2023; Li et al.,
2023; Veselovsky et al., 2023) has found that LLMs
are limited in their ability to create synthetic humor.
We take a new approach, exploiting the asymmet-
rical difficulty (Josifoski et al., 2023) of synthetic
humor generation. Rather than only testing whether
LLMs can generate humor, we explore their ability
to edit away humor in existing jokes. Validating
1Our
code
and
datasets
will
be
made
avail-
able
at
https://github.com/zacharyhorvitz/
Getting-Serious-With-LLMs.
1
arXiv:2403.00794v2  [cs.CL]  21 Jun 2024
and harnessing this capability could provide large
paired datasets and support future work on improv-
ing humor detection and even generation.
Our contributions include benchmarking against
human-curated data in the Unfun corpus, where we
find that LLMs like GPT-4 and GPT-3.5 (OpenAI,
2023, 2022) can (1) outperform humans at remov-
ing humor from texts and that (2) this ability can be
harnessed to generate high quality synthetic data
for training humor classifiers. While these models
can also be prompted to modify unfunny headlines
to craft satire, we find that this ability is more in-
consistent and lags behind satirical writers. Finally,
we consider a code-mixed English-Hindi humor
dataset to evaluate whether GPT-4’s “unfunning”
ability generalizes to other domains and languages.
We find that the resulting synthetic unfunny dataset
is rated highly by bilingual annotators and poses
challenging adversarial data for models trained on
the original corpus.
2
Getting Serious with Language Models
We first revisit the Unfun task and resulting dataset,
but with language models as players.
2.1
Unfun Dataset
In the original Unfun game (West and Horvitz,
2019), players were tasked with editing existing
satirical headlines from The Onion,2 to transform
the original satire into corresponding serious head-
lines. For example (removing “Delicious”):
“Scientists Discover Delicious New Species"
Players were rewarded for preserving token-level
similarity with the original satire and for crafting
convincingly serious headlines that other players
rated as real. The resulting dataset includes approx-
imately 11K unfunned headlines, with a subset
rated by players. We leverage Unfun pairs, of satir-
ical headlines and their unfunned counterparts, to
benchmark the performance of LLMs at editing
humorous texts against humans. We include ad-
ditional details on data preparation in Appendix
A.1.1.
2.2
Unfun Generation
We consider a few-shot setting (Brown et al., 2020),
and provide LLMs with a short task description,
along with a set of input-output exemplar pairs: (hu-
morous text, serious text). Following Veselovsky
2https://www.theonion.com/
et al. (2023), we encourage diversity in our syn-
thetic data by sampling these exemplars from a
subset of the existing pairs rated as high-quality
by the original human players.
For the unfun-
ning task, we consider four popular LLMs: GPT-4
(OpenAI, 2023) and GPT-3.5-TURBO, along with
MISTRAL-7B-INSTRUCT and MISTRAL-7B (Jiang
et al., 2023).
We also consider a lightweight alternative ap-
proach, ROBERTA-SWAP, that replaces low proba-
bility tokens using predictions from a ROBERTA
masked language model (Liu et al., 2019). This
approach is motivated by the Incongruity Theory of
Humor (Hutcheson, 1750; Morreall, 2023), which
associates humor with surprise, and previous work
that has found humorous headlines to have higher
perplexities (Peyrard et al., 2021). ROBERTA-SWAP
edits satirical headlines by iteratively performing
token swaps at k positions. At each selected posi-
tion, the original token is replaced with the highest
probability token predicted by the model at that
masked time-step. The k swap positions are se-
lected using the ratio between the probability of the
original token and the probability assigned to the
language model’s prediction. Additional details on
unfun generation are included in Appendix A.2.1.
3
Unfun Evaluation
3.1
Experimental Setup
The existing Unfun data enables comparison of hu-
man and LLM players, via both automatic and
human evaluations. We first evaluate the quality
of synthetically generated data through automated
evaluation on the downstream task of Unfun detec-
tion, and then follow this with a human evaluation.
3.1.1
Automatic Evaluations
First, following recent work on synthetic data (Li
et al., 2023; Veselovsky et al., 2023) we evaluate
the data quality of outputs from LLMs by testing
whether binary humor classifiers trained on the
synthetic outputs can differentiate between actual
humorous and unfunned headlines from the orig-
inal Unfun dataset. We compare training on data
from human players and actual satirical headlines
to two configurations of synthetic data:
[Synthetic unfun; Original satire]
[Human unfun; Synthetic satire]
These two configurations enable comparing the
“unfunning” and joke writing capabilities of LLMs.
Additionally, we consider the alternative of using
2
Data Characteristics
Holdout Accuracy
Direction
Source
Diversity (TTR)
Edit Dist
MISTRAL
ROBERTA
Unfun
ROBERTA-SWAP
0.262
2.7
69.9 (0.9)
62.7 (0.7)
MISTRAL
0.257
2.1
70.7 (0.7)
61.7 (0.3)
MISTRAL INSTRUCT
0.255
2.4
70.9 (0.7)
64.7 (0.5)
GPT-3.5
0.259
4.5
72.9 (0.2)
65.9 (0.4)
GPT-4
0.252
3.8
76.5 (0.2)
69.9 (0.5)
News Headlines
0.306
-
66.3 (0.2)
64.1 (0.2)
Unfun Players
0.271
2.9
80.3 (0.5)
72.7 (0.4)
Humor
MISTRAL
0.244
2.8
66.3 (0.7)
56.3 (0.4)
MISTRAL INSTRUCT
0.221
4.5
65.2 (0.8)
58.8 (0.4)
GPT-3.5
0.24
4.6
69.9 (0.5)
58.7 (0.4)
GPT-4
0.246
5.5
69.5 (0.7)
59.7 (0.6)
The Onion
0.262
-
-
-
Table 1: Automatic evaluations of synthetic Unfun data. We consider the two directions of editing away (Unfun) and
editing in humor (Humor). We report median accuracies (and standard error) on a balanced holdout set (n = 750)
over 5 seeds when fine-tuning MISTRAL (Jiang et al., 2023) and ROBERTA (Liu et al., 2019) humor classifiers.
Direction
Source
Rated Real
Slightly Funny / Funny
Grammatical
Coherence
Unfun
ROBERTA-SWAP
30%
15% / 5%
93%
86%
MISTRAL INSTRUCT
21%
50% / 14%
100%
96%
GPT-3.5
51%
23% / 3%
100%
98%
GPT-4
49%
21% / 3%
100%
99%
News Headlines
81%
2% / 0%
99%
93%
Human Players
33%
21% / 7%
94%
92%
Humor
MISTRAL INSTRUCT
21%
34% / 9%
99%
93%
GPT-3.5
11%
54% / 8%
100%
94%
GPT-4
10%
45% / 10%
100%
98%
The Onion
4%
68% / 24%
99%
97%
Table 2: Human evaluations of synthetic Unfun data. We consider n = 100 samples per approach. We collect three
annotations per example and assign labels by majority agreement.
actual unrelated news headlines as non-humorous
examples. Using data from each approach, we
fine-tune ROBERTA and MISTRAL-7B for humor
classification. Our test set comprises a subset of
headline pairs from the Unfun corpus that were
highly rated in the original game. Additional evalu-
ation details are provided in Appendix A.4.
3.1.2
Human evaluations
To perform our human evaluations, we recruited 10
university students as annotators, all of whom were
American and native English speakers. Annotators
were tasked with rating headlines as real/satire/nei-
ther. In the case of the “satire” label, we also
task the annotators with rating funniness ([0 =
not funny, 1 = slightly humorous, 2 = funny]). If
the annotator selects “neither”, we ask them to rate
the headline’s grammaticality ({0, 1}) and coher-
ence ({0, 1}). We gather three annotations for each
sample and assign labels based on majority vote.
We include additional information on our human
evaluations and annotation scheme in Appendix
A.3 and C.1
3.2
Results
Automatic Evaluations Table 1 contains the au-
tomatic evaluations on the Unfun corpus.
No-
tably, when validated on human data, humor clas-
sifiers trained on GPT-4’s synthetic unfun data
are very performant, incurring the smallest accu-
racy drop relative to human-edited training data
3
Source
Edit Dist Humor Coherence
Non-Humor
-
16.8%
92.8%
GPT-4 Unfuns
6.6
16.0%
93.6%
+ GPT-4 Filter
6.9
3.6%
89.3%
Humor
-
48.0%
93.6%
Table 3: Human evaluations and edit distance of original
and synthetic English-Hindi Tweet data (Khandelwal
et al., 2018). n = 125 per approach.
(∆Mistral = −3.8% and ∆RoBERTa = −2.8%). In
contrast, classifiers trained with real news head-
lines as unfunny data perform poorly, highlight-
ing the importance of aligned data for this task.
However, we find that not all aligned data is cre-
ated equal, and that classifiers perform significantly
worse when trained on synthetic humor data rela-
tive to human-edited data (∆< −10%). Even data
from our ROBERTA-SWAP unfun baseline dramati-
cally outperforms, or is on par with, all synthetic
humor approaches. The edit distances demonstrate
that each approach retains a large portion of the
original humorous text. However, GPT-4 and GPT-
3.5 tend to modify headlines more than human
players (3.8 and 4.5 vs 2.9).
Human Evaluations Table 2 displays the results
from our human evaluations. All approaches for
generating synthetic humor significantly underper-
form Onion headlines on funniness and realness rat-
ings (p < 0.05). Notably, we do not observe a sig-
nificant improvement between GPT-3.5 and GPT-
4. In contrast, synthetic unfuns from both GPT-
3.5 and GPT-4 were significantly more likely than
human unfuns to be rated as real news headlines.
They were also rated as similarly unfunny and more
grammatical and coherent. Surprisingly, our simple
ROBERTA-SWAP approach also performed compa-
rably with Unfun players on funniness and real
headline metrics, but underperformed on coher-
ence. Together, these results indicate that current
LM-based methods underperform satirical writers
on humor generation, but can outperform human
crowd-workers at editing away humor in satire to
craft aligned datasets.
4
Extending Unfun to Other Languages
Recent work has found that GPT-4 exhibits strong
multilingual capabilities (Møller et al., 2023; Jiao
et al., 2023; Ahuja et al., 2023). Motivated by these
findings, we investigate whether its ability to edit
away humor generalizes to other languages and
forms of joke.
4.1
Experimental Setup
We consider an existing corpus of code-mixed
English-Hindi tweets, previously annotated as hu-
morous or non-humorous (Khandelwal et al., 2018).
Here, we prompt GPT-4 to unfun humorous tweets.
To remove low quality results, we secondarily fil-
ter outputs that GPT-4 still classifies as humorous.
We provide additional details on dataset prepara-
tion in Appendix A.1.2 and English-Hindi unfun
generation in A.2.
We perform a human evaluation with bilingual
annotators who rated these unfunned outputs from
GPT-4 alongside samples from the original dataset.
We also run an automatic evaluation, testing the
performance of humor classifiers trained with dif-
ferent proportions of synthetic non-humorous data.
We evaluate on holdout synthetic data rated by
the annotators as coherent and successfully non-
humorous. For the humor classifier, we fine-tune an
XLM-ROBERTA model (Conneau et al., 2020) pre-
viously fine-tuned on English-Hindi Twitter data
(Nayak and Joshi, 2022).
4.2
Results
Tables 3 and 4 contain the human evaluations and
automatic results for English-Hindi data.
GPT-
4 edited texts were rated comparably to non-
humorous human tweets despite being derived from
humorous tweets, which were rated as humorous
by our annotators (48%) of the time. Filtering with
GPT-4 yielded a smaller sample (56/125) that was
rated as much less humorous (3.6%). These re-
sults demonstrate that GPT-4 is able to reliably
unfun English-Hindi tweets, but with more edits
than American satirical headlines (6.6 vs 3.8). Ad-
ditionally, unfunned data can provide a challenging
adversarial dataset. In Table 4 we evaluate the per-
formance of humor classifiers on human-vetted un-
funned data. When trained on the original dataset,
the classifier fails to generalize to the unfunned
samples and performs poorly (23% accuracy). In-
corporating synthetic training data improves this
metric at a cost to accuracy on humorous exam-
ples in the original dataset. Together, these results
provide evidence that the humor classifier relies on
superficial features to identify humorous text, and
that, even with fine-tuning, the model struggles to
recognize synthetic unfunny data.
4
Original Dataset
Source
Unfuns
Balanced Accuracy
Humor
Non-Humor
Original
22.6 (3.7)
67.9 (0.9)
80.3 (3.5)
56.9 (5.1)
(25%) Synth Unfuns
34.0 (8.4)
67.7 (1.7)
78.4 (3.3)
55.4 (5.9)
(50%) Synth Unfuns
57.7 (6.0)
62.1 (0.6)
68.4 (5.7)
55.9 (4.7)
Table 4: Automatic evaluations with English-Hindi synthetic data. We report median accuracies (and standard error)
on a holdout set from the original dataset (n = 591) and the human-vetted unfuns (n = 97). We also report median
class-level accuracies for the original dataset.
5
Discussion
Our results indicate that current LLMs struggle to
generate humor, but can outperform crowd-workers
at editing away (or unfunning) humor. We hypothe-
size that maximum likelihood training, combined
with autoregressive sampling techniques, does not
endow models with the creative spark required for
joke writing, and instead lends itself to making high
probability, reasonable substitutions to replace in-
congruous twists. Our evaluations on code-mixed
English Hindi Twitter data indicate that, for GPT-
4, this ability can impressively generalize to other
languages and settings to create novel Unfun-like
datasets. We are excited for future work that har-
nesses this capability and resulting data to improve
humor detection and generation systems, and also
to demystify fundamental properties of humor.
6
Limitations
We consider two settings, English satirical head-
lines and code-mixed English-Hindi tweets. Hu-
mor practices and references vary by culture (Alden
et al., 1993; Jiang et al., 2019), and we leave in-
vestigating cultural impacts on LLMs and humor
to future work. In both of our evaluations, the
subjectivity of humor presents a challenge for our
evaluations (Warren et al., 2021). We see evidence
of this in Table 3, where only 48% of tweets previ-
ously annotated as humorous were also rated as hu-
morous by our annotators, and where 16% of non-
humorous tweets were rated as humorous. This
likely reflects differences in background knowledge
and context between annotators. Additionally, we
note that human Unfun players were incentivized
to perform minimal edits, which may have affected
their human evaluation metrics and lowered edit
distances. On average, however, GPT-4 performs
less than one additional word edit, and several ap-
proaches, including ROBERTA-SWAP, were perfor-
mant with lower edit distances than human players.
Another concern is data contamination (Sainz et al.,
2023), and that a portion of the text from the Unfun
corpus could have been trained on and memorized
by the LLMs we evaluated. We investigate this
concern in Appendix A.6. We note that our results
on English-Hindi data show that GPT-4’s abilities
generalize to a dataset where these pairs do not
already exist on the internet.
7
Ethical Statement
Humor brings joy to people and plays a critical role
in building and maintaining social relationships
(Basso, 1979). However, its importance presents
a double-edged sword; offensive and hurtful hu-
mor can cause real harms, and reinforce prejudice
(Benatar, 1999). As a result, with their widespread
adoption, it will be paramount for AI systems to
be more capable of identifying and appropriately
navigating jokes. We believe that our work on
benchmarking LLM humor abilities and building
challenging detection datasets is an important step
in this direction. However, one possible concern
is that malicious actors could leverage our unfun-
ning approach to circumvent existing safeguards.
In our experimentation, we found numerous set-
tings where GPT-4 refused to generate jokes for
offensive topics, but had no trouble editing texts
to remove humor and offensiveness. This could
enable building large parallel datasets of (offensive-
text, non-offensive counterparts) that could then be
used to train models for offensive joke generation.
Acknowledgements
We would like to thank Eric Horvitz for guidance
that helped shape the direction of this work. We
are also grateful to Nicholas Deas, Debasmita Bhat-
tacharya, and Maximillian Chen for their feedback.
Additionally, we would like to extend our grati-
tude to Amith Ananthram, Samir Gadre, Fei-Tzin
Lee, Matthew Toles, Elsbeth Turcan, Melanie Sub-
5
biah, Emily Allaway, Tymon Nieduzak, Rattandeep
Singh, Prabhpreet Singh Sodhi, and Apoorva Joshi
for support on human evaluations.
References
Kabir Ahuja, Rishav Hada, Millicent Ochieng, Prachi
Jain, Harshita Diddee, Krithika Ramesh, Samuel C.
Maina, Tanuja Ganu, Sameer Segal, Maxamed
Axmed, Kalika Bali, and Sunayana Sitaram. 2023.
Mega:
Multilingual evaluation of generative ai.
ArXiv, abs/2303.12528.
Dana L. Alden, Wayne D. Hoyer, and Chol Lee. 1993.
Identifying global and culture-specific dimensions
of humor in advertising: A multinational analysis.
Journal of Marketing, 57:64 – 75.
Alexander Baranov, Vladimir Kniazhevsky, and Pavel
Braslavski. 2023. You told me that joke twice: A sys-
tematic investigation of transferability and robustness
of humor detection models. In Proceedings of the
2023 Conference on Empirical Methods in Natural
Language Processing, pages 13701–13715, Singa-
pore. Association for Computational Linguistics.
K.H. Basso. 1979. Portraits of ’the Whiteman’: Linguis-
tic Play and Cultural Symbols among the Western
Apache. Cambridge University Press.
David Benatar. 1999. Prejudice in jest: When racial
and gender humor harms. Public Affairs Quarterly,
13(2):191–203.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie
Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind
Neelakantan, Pranav Shyam, Girish Sastry, Amanda
Askell, Sandhini Agarwal, Ariel Herbert-Voss,
Gretchen Krueger, Tom Henighan, Rewon Child,
Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu,
Clemens Winter, Christopher Hesse, Mark Chen, Eric
Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess,
Jack Clark, Christopher Berner, Sam McCandlish,
Alec Radford, Ilya Sutskever, and Dario Amodei.
2020. Language models are few-shot learners.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal,
Vishrav Chaudhary, Guillaume Wenzek, Francisco
Guzmán, Edouard Grave, Myle Ott, Luke Zettle-
moyer, and Veselin Stoyanov. 2020. Unsupervised
cross-lingual representation learning at scale.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and
Luke Zettlemoyer. 2023. Qlora: Efficient finetuning
of quantized llms.
Ira Glass. 2008. Tough room.
Fabrício Góes, Piotr Sawicki, Marek Grze´s, Daniel
Brown, and Marco Volpe. Is gpt-4 good enough to
evaluate jokes?
Jack Hessel, Ana Marasovic, Jena D. Hwang, Lillian
Lee, Jeff Da, Rowan Zellers, Robert Mankoff, and
Yejin Choi. 2023.
Do androids laugh at electric
sheep? humor “understanding” benchmarks from
the new yorker caption contest. In Proceedings of the
61st Annual Meeting of the Association for Compu-
tational Linguistics (Volume 1: Long Papers), pages
688–714, Toronto, Canada. Association for Compu-
tational Linguistics.
Zachary Horvitz, Nam Do, and Michael L. Littman.
2020. Context-driven satirical news generation. In
Proceedings of the Second Workshop on Figurative
Language Processing, pages 40–50, Online. Associa-
tion for Computational Linguistics.
Nabil Hossain, John Krumm, and Michael Gamon. 2019.
“president vows to cut <taxes> hair”: Dataset and
analysis of creative text editing for humorous head-
lines.
In Proceedings of the 2019 Conference of
the North American Chapter of the Association for
Computational Linguistics: Human Language Tech-
nologies, Volume 1 (Long and Short Papers), pages
133–142, Minneapolis, Minnesota. Association for
Computational Linguistics.
Nabil Hossain, John Krumm, Tanvir Sajed, and Henry
Kautz. 2020. Stimulating creativity with funlines: A
case study of humor generation in headlines.
F. Hutcheson. 1750. Reflections Upon Laughter: And
Remarks Upon the Fable of the Bees. Garland Pub-
lishing.
Sophie Jentzsch and Kristian Kersting. 2023. Chatgpt
is fun, but it is not funny! humor is still challenging
large language models.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Men-
sch, Chris Bamford, Devendra Singh Chaplot, Diego
de las Casas, Florian Bressand, Gianna Lengyel, Guil-
laume Lample, Lucile Saulnier, Lélio Renard Lavaud,
Marie-Anne Lachaux, Pierre Stock, Teven Le Scao,
Thibaut Lavril, Thomas Wang, Timothée Lacroix,
and William El Sayed. 2023. Mistral 7b.
Tonglin Jiang, Hao Li, and Yubo Hou. 2019. Cultural
differences in humor perception, usage, and implica-
tions. Frontiers in Psychology, 10.
Wenxiang Jiao, Wenxuan Wang, Jen tse Huang, Xing
Wang, and Zhaopeng Tu. 2023. Is chatgpt a good
translator? yes with gpt-4 as the engine.
Martin Josifoski, Marija Sakota, Maxime Peyrard, and
Robert West. 2023. Exploiting asymmetry for syn-
thetic training data generation: SynthIE and the case
of information extraction. In Proceedings of the 2023
Conference on Empirical Methods in Natural Lan-
guage Processing, pages 1555–1574, Singapore. As-
sociation for Computational Linguistics.
Ankush Khandelwal, Sahil Swami, Syed S. Akhtar,
and Manish Shrivastava. 2018.
Humor detection
in english-hindi code-mixed social media content :
Corpus and baseline system.
6
Zhuoyan Li, Hangxiao Zhu, Zhuoran Lu, and Ming Yin.
2023. Synthetic data generation with large language
models for text classification: Potential and limita-
tions. ArXiv, abs/2310.07849.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man-
dar Joshi, Danqi Chen, Omer Levy, Mike Lewis,
Luke Zettlemoyer, and Veselin Stoyanov. 2019.
Roberta: A robustly optimized bert pretraining ap-
proach.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled
weight decay regularization.
Anders Giovanni Møller, Jacob Aarup Dalsgaard, Ari-
anna Pera, and Luca Maria Aiello. 2023. Is a prompt
and a few samples all you need? using gpt-4 for data
augmentation in low-resource classification tasks.
ArXiv, abs/2304.13861.
John Morreall. 2023. Philosophy of Humor. In Ed-
ward N. Zalta and Uri Nodelman, editors, The Stan-
ford Encyclopedia of Philosophy, Summer 2023 edi-
tion. Metaphysics Research Lab, Stanford University.
Ravindra Nayak and Raviraj Joshi. 2022.
L3Cube-
HingCorpus and HingBERT: A code mixed Hindi-
English dataset and BERT language models.
In
Proceedings of the WILDRE-6 Workshop within the
13th Language Resources and Evaluation Confer-
ence, pages 7–12, Marseille, France. European Lan-
guage Resources Association.
OpenAI. 2022. Chatgpt: Optimizing language models
for dialogue.
OpenAI. 2023. Gpt-4 technical report.
Maxime Peyrard, Beatriz Borges, Kristina Gligori´c, and
Robert West. 2021. Laughing heads: Can transform-
ers detect what makes a sentence funny?
Oscar Sainz, Jon Campos, Iker García-Ferrero, Julen
Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre.
2023. NLP evaluation in trouble: On the need to mea-
sure LLM data contamination for each benchmark.
In Findings of the Association for Computational
Linguistics: EMNLP 2023, pages 10776–10787, Sin-
gapore. Association for Computational Linguistics.
Veniamin Veselovsky, Manoel Horta Ribeiro, Akhil
Arora, Martin Josifoski, Ashton Anderson, and
Robert West. 2023. Generating faithful synthetic data
with large language models: A case study in compu-
tational social science. ArXiv, abs/2305.15041.
Caleb Warren, Adam Barsky, and A. Peter McGraw.
2021.
What makes things funny? an integrative
review of the antecedents of laughter and amusement.
Personality and Social Psychology Review, 25(1):41–
65. PMID: 33342368.
Robert West and Eric Horvitz. 2019.
Reverse-
engineering satire, or “paper on computational humor
accepted despite making serious advances”. In Pro-
ceedings of the 33rd AAAI Conference on Artificial
Intelligence.
A
Appendix
A.1
Data Preparation
A.1.1
Unfun Corpus
We use the February 2, 2023 Unfun (West and
Horvitz, 2019) database backup,3 and consider all
valid unfunned headlines (i.e. not None). This re-
sults in 11831 pairs. A subset of these have ratings
from other players. We use these to curate a high
quality evaluation subset of pairs where:
• There is at least one annotation.
• The satirical headline has a funniness rating
≥0.8.
• The unfunned headline has a funniness rating
≤0.2.
The resulting 867 pairs were split among prompt
examples (10%), dev (30%), and test (60%) shards.
For our training set, we consider the remaining
headlines, again ensuring that there is no overlap
with other shards. The resulting dataset has many
instances where there are multiple unfunned coun-
terparts for each satirical headline. As an additional
step, we randomly filter our training, dev, and test
shards so that there is only one unfunned headline
per satirical headline. This results in a training set
of 3882 unfuns, a dev set of 186 unfuns, and a test
set of 375 unfuns, in each case, these are included
alongside their corresponding satirical headlines.
For an additional training data baseline, we also
retrieve an equal number of real news headlines
included in the Unfun database.
A.1.2
Code-Mixed English-Hindi Humor
We use the version of the English-Hindi Humor
dataset by Khandelwal et al. (2018) hosted on
GitHub.4 We use the provided labels for the avail-
able data. Notably, a portion of annotated samples
appear to be unavailable. We divide the available
dataset (n = 2951) into training, dev, and test
shards (60%, 20%, 20%). Additionally, we filter
tweets containing links.
A.2
Data Generation Details
We include our full prompts in Appendix B. For
decoding hyperparameters, we use top-p = 0.85
and τ = 1.0 for all LLMs.
3https://github.com/epfl-dlab/unfun
4https://github.com/Ankh2295/
humor-detection-corpus
7
A.2.1
Unfun Data Generation
To generate synthetic Unfun for each LLM ap-
proach, we prompt each model with 8 randomly
sampled in-context pairs from examples from our
high quality subset that was set aside for prompt-
ing. For our ROBERTA-SWAP baseline, we replace
tokens in the original satirical headline using a
ROBERTA-BASE5 model. To select each replace-
ment, we iterate over and individually mask each
token in the headline, and then predict the masked
token:
ˆxi = arg max
x
P(x | x̸=i, θRoBERTa)
The position with the largest ratio between the pre-
dicted token and the original token probabilities is
selected as the swap position:
swap position = arg max
i
[P(ˆxi | x̸=i, θRoBERTa)
P(xi | x̸=i, θRoBERTa)]
We then replace xi with ˆxi, and repeat this proce-
dure k times. We set k = 3 in our experiments.
A.2.2
Hindi-English Data Generation
Unlike for Unfun, we do not have existing pairs
of (un-humorous, humorous) English Hindi tweets.
To remedy this, we first generated 50 examples in
a zero-shot setting on our training set, and then
selected nine high quality results to serve as our
prompt. We additionally prompt GPT-4 with hu-
morous and non-humourous texts to classify the
resulting unfunned tweets as humorous or non-
humorous. We filter unfunned tweets if they are
still classified as humorous.
A.3
Human Evaluations
We recruited 10 university students as annotators
for the Unfun task. All annotators were Ameri-
can and native English speakers. For the English-
Hindi dataset, we worked with three bilingual
(Hindi and English) speakers. For both evaluations,
we gathered three unique annotations per exam-
ple, and assigned labels based on majority votes.
Our Unfun evaluation assumes that any headline
labeled as satirical or as real headline is grammati-
cal and coherent. In contrast, we do not consider
the grammatical label for English-Hindi data, due
to the varied syntactic styles of tweets.
In Table 2, headlines are only rated "Real" if a
majority of annotators rated the headline as "Real"
5https://huggingface.co/FacebookAI/
roberta-base
(not "Satire" or "Neither"). Headlines are rated
"Slightly Funny" if a majority of annotators as-
signed the headline funniness ≥1, and "Funny"
with funniness = 2. Our full instructions for both
human evaluations are included in Appendix C.1.
Tables 5 and 6 display inter-annotator agreement
statistics.
Human Label
Krippendorff
Real
0.507
Funny
0.333
Very Funny
0.214
Grammar
0.271
Coherence
0.214
Table 5: Krippendorff’s α results on Unfun dataset.
Human Label
Krippendorff
Coherence
0.206
Humorous
0.377
Table 6: Krippendorff’s α results on English-Hindi
dataset.
A.4
Automatic Evaluations
On the Unfun dataset, for each synthetic Unfun ap-
proach, we generate data using the corresponding
original 3882 training examples as inputs. We then
evaluate classifiers trained on each dataset on the
filtered high quality holdout data. To generate hu-
mor, we provide the unfunned example as input. To
edit away humor, we provide the original satirical
headline. We also provide in-context pairs drawn
from the high quality prompt examples (See A.1.1).
For our Real News baseline, we randomly select
3882 real news headlines to serve as non-humorous
examples.
On the English-Hindi dataset, we compare
training on the original dataset to training on data
where (25%) and (50%) of non-humorous exam-
ples have been replaced by GPT-4 Filtered un-
funned data. We evaluate classifiers on a holdout
set from original dataset (n = 591), and also set of
Unfuns (n = 97), derived from humorous exam-
ples in our holdout set and rated by our annotators
as both coherent and non-humorous. All results for
both datasets are computed over 5 seeds.
8
A.5
Humor Classifier Training
For the Unfun task, we fine-tune MISTRAL (Jiang
et al., 2023)6 and ROBERTA (Liu et al., 2019)7
models. For Hindi-English, we consider HING-
ROBERTA (Nayak and Joshi, 2022)8. All models
are trained with the AdamW optimizer (Loshchilov
and Hutter, 2019) and a constant learning rate. Due
to the class imbalance in the available English-
Hindi dataset (39% non-humorous, 61% humor-
ous), we weight the loss by the inverse proportion
of class frequency.
We fine-tune our MISTRAL classifier with 4-bit
quantized LoRA (Dettmers et al., 2023) and the
addition of a classification head. For all classifiers,
we first perform hyperparameter tuning on the orig-
inal human authored datasets.
For the Unfun dataset we consider:
• Learning Rates ∈{5e −5, 2.5e −5, 1.25e −
5, 6.25e −6, 3.125e −6, 1.5625e −6}
• Batch Size ∈[32] (Due to resource con-
straints)
For the English-Hindi Dataset dataset we con-
sider:
• Learning Rates ∈{5e −5, 2.5e −5, 1.25e −
5, 6.25e −6, 3.125e −6, 1.5625e −6}
• Batch Size ∈{256, 128, 64, 32, 16, 8}
After selecting the highest performing config-
uration, we run each experiment with 5 seeds
([1234, 2345, 3456, 4567, 5678]). We include the
most performant hyperparameters in Table 7. All
model trains use a single NVIDIA A100 GPU. We
estimate the total compute budge to be 200 hours.
A.6
Considering Memorization
We investigate whether data contamination and
memorization is affecting our results by testing
how often synthetic unfuns or humor appear in
the original Unfun corpus. We find that only a
small fraction of outputs appear to match human-
unfunned text or satire headlines. We include re-
sults in Table 8. Of these, the majority represent
simple edits, indicating that the models may have
rediscovered trivial unfuns. For example:
“Egypt plunges into state of Middle East crisis"
6https://huggingface.co/mistralai/
Mistral-7B-v0.1
7https://huggingface.co/FacebookAI/
roberta-base
8https://huggingface.co/l3cube-pune/
hing-roberta
B
Prompts
B.1
Unfun Task Prompts
B.1.1
Humor Generation
Chat Models
"You are a helpful assistant that edits real-
istic headlines to make them humorous."
{"role":
"user", "content":
<Unfunned
Headline>},
{"role":
"assistant", "content":
<Satire
Headline>}
Completion Models
"The following realistic headlines can be
edited to be humorous:"
"<Unfunned Headline> -> <Satire Head-
line>"
B.1.2
Unfun Generation
Chat Models
"You are a helpful assistant that edits hu-
morous headlines to make them realistic."
{"role": "user", "content": <Satire Head-
line>},
{"role": "assistant", "content": <Unfunned
Headline>},
...
Completion Models
"The following humorous headlines can be
edited to be realistic:"
"<Satire Headline> -> <Unfunned Head-
line>"
B.2
English-Hindi Task Prompts
B.2.1
Unfun Generation
Chat Models
"Kya ye diye hue tweet ka humor wala part
hata kar use normal bana sakti ho? Aur
jitna ho sake utna punctuation use same
rakhne ki koshish karna" [Can you remove
the humorous part of the given tweets and
make them normal? And try to keep the
punctuation as much the same as possible.].
9
Model
Learning Rate
Batch Size
MISTRAL (QLoRA)
6.25e-06
32
ROBERTA
1.25e-05
32
HING-ROBERTA
1.5625e-06
8
Table 7: The training configurations for our automatic evaluations, after hyperparameter tuning.
Model
Unfun
Satire
GPT-3.5
3/200
0/200
GPT-4
7/200
0/200
MISTRAL
2/200
1/200
MISTRAL INSTRUCT
2/200
0/200
ROBERTA-SWAP
0/200
-
Table 8: The number of overlapping samples between
human-curated headlines and synthetic headlines in our
test examples (n = 200).
{"role": "user", "content": <Context Funny
Tweet>},
{"role": "assistant", "content": <Context Un-
funned Tweet>}
B.2.2
Unfun Filtering
Chat Models
"You are a pattern-following assistant used
to rigorously determine whether a Hindi
tweet is intended to be humorous. Given a
Hindi tweet, respond only with either of Yes
or No. Yes if it is humoruous and No if it is
not humorous"
{"role":
"user",
"content":
<Context
Tweet>},
{"role": "assistant", "content": <Context
Yes/No Label>}
C
Human Evaluation Instructions
C.1
Unfun Task Instructions
Each annotator has been assigned a series
of text samples to review. First, you are
asked to evaluate whether the text sounds
like a
• r) real news headline (like from a non-
humorous news website)
• OR s) satirical news headline (like
from a humorous newspaper like TheO-
nion.)
• OR n) neither (text that would not ap-
pear in either setting, because it is un-
grammatical, or incoherent.
If you rate a headline as n (neither), you will
be further prompted to rate it as a grammat-
ical [no=0,yes=1 (for a news headline) and
coherent [no=0,yes=1].
If you rate a headline as s (satire), you will
be prompted to subjectively rate the quality
of humor:
• 0 - not funny
• 1 - slightly humorous / there is some
identifiable joke
• 2 - funny
Content Warning: Several headlines may
contain references to upsetting content.
EXAMPLES: Satirical Headlines
• nhl not quite sure why it has a presea-
son
• america’s sweetheart dumps u.s. for
some douchebag
• apple: new iphone good
• cat general says war on string may be
unwinnable
• fire chief grants fireman 3-day exten-
sion on difficult fire
News Headlines
• the word ’doofuses’ may cost ex-yahoo
ceo bartz $10 million
• 2 meteorites hit connecticut
10
• world outraged by north korea’s latest
nuke test
• poverty rate hits 17-year high
• philippines: 5 foreign terror suspects
in south
C.2
English-Hindi Task Instructions
The following task instructions specify additional
information based on the original instructions pro-
vided to annotators in (Khandelwal et al., 2018).
Each annotator has been assigned a series
of text samples to review. First, you are
asked to evaluate whether the text is h) hu-
morous n) non-humorous
Secondarily, you will be asked to rate
whether a text is coherent [no=0,yes=1] A
tweet should be marked as coherent, even if
you don’t have all the required background
knowledge, as long as you can reasonably
understand its meaning.
Additional info:
• Any tweets stating any facts, news or
reality should be classified as non-
humorous.
• Tweets which consisted of any humor-
ous anecdotes, fantasy, irony, jokes, in-
sults should be annotated as humorous
• Tweets stating any facts, dialogues or
speech which did not contain amuse-
ment should be put in non-humorous
class.
• Tweets containing normal jokes and
funny quotes should be placed in the
humorous category.
• Some tweets consist of poems or lines
of a song but modified. If such tweets
contain satire or any humoristic fea-
tures, then they could be categorized
as humorous otherwise not.
Content Warning: Several tweets may con-
tain references to upsetting/offensive con-
tent.
EXAMPLES (We give the English Transla-
tions of each in brackets but they were not
presented to the annotators):
Humorous Tweets
• Jhonka hawa ka aaj bhi chhup ke
hilaata hoga na #Samir #HawaKa-
Jhonka
#BeingSalmanKhan
[Does
the breeze still sway secretly to-
day? #Samir #HawaKaJhonka #Be-
ingSalmanKhan)
• Working on a Sunday, chand rupye ka-
maane ke liye insaan apni khushiyon
ka bhi sauda kar leta hai. [Working
on a Sunday, to earn a few rupees, a
person sometimes even sacrifices their
happiness.]
• DJ wale babu bhosdike ab to gaana
baja de iska.. bol bol ke kaan se khoon
nikaal diya hai isne [DJ wale babu,
play the song now.. he has made our
ears bleed by talking so much.]
• Is Arvind Kejriwal new Che Guavara
? RT @ashutosh83B Is Rahul Gandhi
new Arvind Kejariwal ? [Is Arvind
Kejriwal the new Che Guevara? RT
@ashutosh83B Is Rahul Gandhi the
new Arvind Kejriwal?]
• Sukh bhare din beete re bhaiya,
Babadook aayo re [Brother, may the
days filled with joy pass by.
The
Babadook has arrived.]
Non-Humorous Tweets
• Apne support wale MLAs ko farm-
house main band kar lenge. Parade
karayenge. Takhta palat karenge. Ak-
hand chutiyap. [We will lock up our
supporting MLAs in the farmhouse. Pa-
rade them. Flip the throne. Absolute
nonsense.]
• Hrithik Roshan is using Vodafone.
[Hrithik Roshan is using Vodafone.]
• PLEASE STOP MAKING JOKES
ON SALMAN KHAN. BHAI BOLA
NAHI CHALA RAHA THA GAADI
TO NAHI CHALA RAHA THA.
#BHAIROXX [Please stop making
jokes on Salman Khan. Bhai was not
driving the car if he said he was not
driving the car. #BHAIROXX]
11
• Bhaari sankhya mein vote karein,
aapke TL par wph hi nazar aayega
[Vote in large numbers, wph will only
appear in your TL.]
D
Reference Examples
Tables 9, 10, and 11 include reference samples for
English synthetic unfun outputs, English satire out-
puts, and English-Hindi unfun outputs respectively.
12
Original
Satire
tom
petty
to
play
some new stuff he’s
been working on at
super bowl
jaguars offensive line
not
sure
they
can
open big enough hole
for
maurice
jones
drew
obama takes surprise
caller during weekly
radio address
Human
tom
petty
to
play
some new stuff he’s
been working on at
coachella
jaguars offensive line
not sure they can open
stable positioning hole
for maurice jones drew
obama takes caller dur-
ing weekly radio ad-
dress
GPT-
3.5
tom petty to perform
classic hits at super
bowl
jaguars offensive line
not sure they can create
sufficient gap for mau-
rice jones drew
obama takes surprise
caller during live radio
interview
GPT-4
tom petty to perform
new material at super
bowl
jaguars offensive line
unsure if they can open
big enough hole for
maurice jones drew
obama
takes
unex-
pected caller during
weekly radio address
MISTRAL tom petty to play some
new stuff he’s been
working on at super-
bowl
jaguars offensive line
not sure they can open
big enough hole for joe
flacco
obama takes surprise
caller during weekly ra-
dio address
MISTRAL
IN-
STRUCT
tom petty to play some
new songs he’s been
working on at super
bowl halftime show
jaguars offensive line
not sure they can clear
big enough opening for
maurice jones drew
obama takes surprise
caller during weekly in-
terview
ROBERTA-
SWAP
he wants to play some
new stuff he’s been
working on at the bowl
jaguars offensive line
not sure they can dig
big enough hole for
maurice jakes
panama takes in mi-
grants during weekly
radio address
Table 9: Unfun model output reference samples.
13
Original
Unfun
obama,
congress
must strike a deal on
budget by march 1
espn sports segment
reveals
science
be-
hind
tony
romo’s
precision pass
jimmie johnson dis-
qualified after pre-
race
inspection
re-
veals car to be violat-
ing rules
Human
obama, congress must
reach deal on budget
by march 1, and then
april 1, and then april
20, and then april 28,
and then may 1
espn sports segment re-
veals science behind
tony romo’s shittiness
jimmie johnson dis-
qualified after pre-race
inspection reveals car
to be airplane
GPT-
3.5
obama, congress must
strike a deal on budget
by march 1 or face de-
tention
espn sports segment
reveals magic behind
tony romo’s precision
pass
jimmie johnson dis-
qualified after pre-race
inspection reveals car
to be a Transformer in
disguise
GPT-4
obama, congress must
strike a deal on budget
by march 1 or risk be-
ing grounded
espn sports segment re-
veals magic spell be-
hind tony romo’s pre-
cision pass
jimmie johnson dis-
qualified after pre-race
inspection reveals car
to be a spaceship
MISTRAL obama, congress must
strike a deal on budget
by march 1
espn sports segment re-
veals science behind
tony romo’s precision
penis
jimmie johnson dis-
qualified after pre-race
inspection reveals car
to be black
MISTRAL
IN-
STRUCT
obama, congress must
strike a deal on bud-
get by march 1, or else
there will be no more
kittens
espn sports segment re-
veals science behind
kate middleton’s royal
wave
jimmie johnson dis-
qualified after pre-race
inspection reveals car
to be violating laws
and joking about it
Table 10: Satire model output reference samples.
Original Humorous Tweet
GPT-4 English-Hindi Unfuns
Ab ki baar.. #MaaBetaFarar..
Ab ki baar.. yeh log farar hain..
Husbands should be like Vim bar, gale
kam aur chale zyada.
Patidev ko samarpit aur lambe samay
tak saath dena chahiye.
O naadan parindey ghar aaja. Parinda:
naadan tera baap.
O naadan parindey ghar aaja. Parinda:
Mujhe ghar aane do.
Neend aaja nahi to kal se tujhe KRK
bulaunga
Neend aaja nahi to kal se tujhe alag
naam se bulaunga
Bhai ab itne velle bhi nahi hai ki #Ind-
VsBan test match dekhenge
Bhai ab itne samay nahi hai ki #IndVs-
Ban test match dekhenge
Asli toofan andar hai, jail ke andar.
#SalmanVerdict
Asli samasya jail ke andar hai. #Salman-
Verdict
Vodafone use karne se acha to ek ka-
bootar pal lo.
Vodafone use karne se acha to kisi aur
network provider ka use karo.
Table 11: GPT-4 English-Hindi unfunned reference samples. See Table 12 for English translations.
14
Original Humorous Tweet
GPT-4 English-Hindi Unfuns
This time.. #MotherSonGone..
This time.. these people are gone..
Husbands should be like Vim bar, less
talk and more work.
Husbands should be dedicated and sup-
port for a long time.
Oh naive bird, come home. Bird: Your
dad is naive.
Oh naive bird, come home. Bird: Let
me come home.
If sleep doesn’t come, from tomorrow I
will call you KRK.
If sleep doesn’t come, from tomorrow I
will call you by a different name.
Bro, we’re not that free to watch the
#IndVsBan test match.
Bro, we don’t have that much time to
watch the #IndVsBan test match.
The real storm is inside, inside the jail.
#SalmanVerdict
The real problem is inside the jail.
#SalmanVerdict
It’s better to raise a pigeon than to use
Vodafone.
It’s better to use another network
provider than Vodafone.
Table 12: Translation of GPT-4 English-Hindi unfunned reference samples.
15


Question:
Based on the experimental results in closed information extraction, which of the following observations about the impact of constrained decoder model size on SketchGCD’s performance are supported?
A. Larger constrained decoders (e.g., LLaMA-2-70
B. consistently achieve higher F1-scores than smaller decoders due to their superior ability to handle complex syntactic constraints.
B. Using a constrained decoder with fewer parameters (e.g., LLaMA-2-7
B. can yield competitive or better results than larger decoders when the refinement task is relatively simple.
C. The performance gap between constrained decoders of different sizes diminishes when the initial sketch from the blackbox LLM contains significant semantic errors requiring correction.
D. The choice of constrained decoder size is inversely correlated with the structural complexity of the constraints, with smaller decoders performing better for highly nested grammars.

Answer: BC

Impact of constrained decoder. We investigate the impact of the constrained decoder on the performance of SketchGCD. As shown in Table 2, given GPT-4 as the sketcher, the choice of the constrained decoder can affect the performance of SketchGCD. Contrary to our expectations, larger constrained decoder models do not always lead to better performance. Our intuition is that step 2 of SketchGCD (constrained decoding) is relatively simple, and the additional capacity of larger constrained decoders does not necessarily provide an advantage.

Wiki-NRE
SynthIE-text
Prec
Recall
F1
Prec
Recall
F1
GPT-4
42.4
44.9
43.6
46.1
44.4
45.2
+ LLaMA-2-7B
38.7
57.1
46.1
58.9
47.3
52.4
+ LLaMA-2-13B
42.9
52.8
47.3
53.6
51.4
52.5
+ LLaMA-2-70B
35.2
54.0
42.6
58.1
53.1
55.5
Table 2: Impact of constrained decoder model (used in step 2 of SketchGCD) on closed information extraction. GPT-4 is used as the sketcher in all cases.

Sketch-Guided Constrained Decoding for
Boosting Blackbox Large Language Models without Logit Access
Saibo Geng, Berkay Döner, Chris Wendler, Martin Josifoski, Robert West
EPFL
{saibo.geng, berkay.doner, chris.wendler, martin.josifoski, robert.west}@epfl.ch
Abstract
Constrained decoding, a technique for enforc-
ing constraints on language model outputs, of-
fers a way to control text generation without
retraining or architectural modifications. Its
application is, however, typically restricted to
models that give users access to next-token dis-
tributions (usually via softmax logits), which
poses a limitation with blackbox large language
models (LLMs). This paper introduces sketch-
guided constrained decoding (SketchGCD), a
novel approach to constrained decoding for
blackbox LLMs, which operates without access
to the logits of the blackbox LLM. SketchGCD
utilizes a locally hosted auxiliary model to re-
fine the output of an unconstrained blackbox
LLM, effectively treating this initial output as
a “sketch” for further elaboration. This ap-
proach is complementary to traditional logit-
based techniques and enables the application
of constrained decoding in settings where full
model transparency is unavailable. We demon-
strate the efficacy of SketchGCD through ex-
periments in closed information extraction and
constituency parsing, showing how it enhances
the utility and flexibility of blackbox LLMs for
complex NLP tasks.1
1
Introduction
Large language models (LLMs) have seen a re-
markable expansion in scope, being used for di-
verse tasks including tool interaction, SQL trans-
lation, robotic navigation and item recommenda-
tions, where adherence to specific constraints is
paramount (Bubeck et al., 2023; Schick et al., 2023;
Poesia et al., 2022; Shah et al., 2022; Zhang et al.,
2023; Hua et al., 2023). Despite their versatility,
LLMs often struggle with constraint adherence in
few-shot scenarios, leading to outputs that violate
task-specific requirements (Chen and Wan, 2023;
Agrawal et al., 2023; Huang et al., 2023).
1Code and data available at https://github.com/
epfl-dlab/SketchGCD
Figure 1: Overview of sketch-guided constrained de-
coding (SketchGCD). In the initial sketching phase,
a blackbox LLM generates a preliminary “sketch” an-
swer without applying any constraints. Then, in the
constrained decoding phase, an auxiliary model, the
constrained decoder, refines the sketch. The refined,
final output respects the specified constraints by con-
struction.
Constrained decoding offers a solution, restrict-
ing model outputs to respect predefined constraints
without necessitating model retraining or architec-
tural modifications (Poesia et al., 2022; Shin et al.,
2021; Beurer-Kellner et al., 2023; Scholak et al.,
2021; Geng et al., 2023). However, existing con-
strained decoding methods require access to the
model’s logits during inference, which is not al-
ways feasible in practice (cf. Appendix A). Since
the most powerful LLMs tend to be commercial
and blackbox (Lee et al., 2023), this has restricted
the application of constrained decoding methods.
arXiv:2401.09967v4  [cs.CL]  22 Jul 2024
Contributions.
To overcome this restriction,
we present sketch-guided constrained decoding
(SketchGCD), which bypasses the need for direct
logit access. SketchGCD uses a locally hosted
(lightweight) open-source LLM to refine the out-
puts of a (heavyweight) blackbox LLM to satisfy
the specified constraints. We validate our method
on closed information extraction, where the con-
straints require generating triples grounded in a
knowledge base, and constituency parsing, where
the constraints require generating tree-structured
outputs. Our experiments show that SketchGCD
significantly boosts the performance of LLMs and
beats previous approaches by a wide margin.
2
Method
SketchGCD splits the constrained decoding task
into two distinct phases: sketching and constrained
decoding.
During sketching, a sketcher—a powerful black-
box LLM denoted as Psk—is employed. It inter-
prets an instruction I alongside a set of demonstra-
tion pairs D = {(xi,yi)}n
i=1, producing a prelimi-
nary draft y∗via unconstrained decoding:
y∗≈argmax
y∈S
Psk(y|I,D,x),
(1)
where S is the set of all possible sequences.
Constrained decoding is done by a constrained
decoder, a smaller-scale, locally hosted LLM Pcg.
Given an instruction Icg, a set of input–sketch–
output demonstrations Dcg = {(xi,yi,zi)}n
i=1, the
original input x, and the sketch y∗, it refines y∗into
z∗≈argmax
z∈S∩C
Pcg(z|Icg,Dcg,x,y∗),
(2)
subject to constraints C. (Optionally, x and xi may
be omitted, with loss of information.)
The sketcher’s output y∗is typically of high qual-
ity, encapsulating the necessary information for the
constrained decoder to produce the final sequence
z∗that adheres to the constraints C. Given the
quality of y∗, the constrained decoder can be imple-
mented using a much smaller model, as its primary
task is to rewrite the sketch y∗with the help of con-
strained decoding, thus facilitating deployment on
standard consumer-grade hardware.
On the contrary, classical, direct few-shot
prompting with constrained decoding would usu-
ally require a larger constrained generator Pcg to be
run locally, in order to find
w∗≈argmax
w∈S∩C
Pcg(w|I,D,x).
(3)
Another basic alternative, unconstrained few-
shot prompting (Brown et al., 2020), yields y∗as
the end product.
SketchGCD builds on the expectation that the
constrained refined output z∗should be at least as
good as both y∗(as z∗respects the constraints) and
w∗(as Psk is a more powerful LLM than Pcg).
3
Experiments
In our experimental setup, we evaluate the effi-
cacy of SketchGCD by comparing it against two es-
tablished baselines: (1) few-shot-prompted uncon-
strained decoding with powerful blackbox LLMs
(Eq. 1) and (2) few-shot-prompted constrained
decoding with open-source LLMs (Eq. 3). The
SketchGCD method remains flexible and is agnos-
tic to the exact implementation of constrained de-
coding. Here we adopt the grammar constrained
decoding framework of Geng et al. (2023), but any
other constraining method can be plugged in.
In our evaluation, we distinguish between se-
quences that are valid (i.e., that satisfy the con-
straints) and those that are correct (i.e., those that
are equal to the intended output for the given input).
A valid output is a prerequisite for being correct,
but it is not the sole criterion for correctness.
3.1
Closed information extraction
Task description. The goal of closed information
triplet extraction (IE) is to extract a comprehensive
set of facts from natural-language text. Formally,
given a knowledge base represented by a knowl-
edge graph (KG) containing a catalog of entities E
and a catalog of relations R, the goal is to extract
the complete set yset ⊂E ×R×E of fact triplets ex-
pressed in a given input text x. It is crucial that the
entities and relations in these triplets be accurately
grounded in the KG’s catalog. An example of this
process can be seen in Fig. 1. The instructions I
and Icg for the sketcher and constrained decoder,
respectively, are listed in Appendix D.1.
Constraints.
We apply the constraints in Ap-
pendix D.2, which restrict entities (1.5 million) and
relations (857) to the Wikidata KG, and enforce the
structural constraint that outputs must be formatted
as sequences of entity–relation–entity triplets.
Datasets and evaluation metrics. We use the
Wiki-NRE (Trisedya et al., 2019) and SynthIE-
text (Josifoski et al., 2023) datasets (details in Ap-
Wiki-NRE
SynthIE-text
Precision
Recall
F1
Precision
Recall
F1
Without logit access
GPT-4
42.4± 2.7
44.9±3.0
43.6± 2.6
46.1± 2.3
44.4± 2.2
45.2± 2.2
+ SketchGCD 7B 38.7± 3.2 (↓3.7)
57.1± 2.9 (↑12.2) 46.1± 2.8 (↑2.5)
58.8± 7.9 (↑12.7) 47.3± 2.3 (↑2.9)
52.4± 2.2 (↑7.2)
GPT-3.5-Turbo
27.4± 2.0
27.4± 2.5
27.4± 2.5
24.6± 2.0
23.1± 1.9
23.8± 1.9
+ SketchGCD 7B 31.3± 3.3 (↑3.9)
46.4± 2.8 (↑18.7) 37.4± 2.8 (↑10.0) 49.5± 2.8 (↑24.9) 41.4± 2.1 (↑18.3) 45.1± 2.1 (↑21.3)
Claude
34.1± 3.1
28.2± 2.8
30.8± 2.7
27.0± 2.0
26.7± 2.0
26.8± 2.0
+ SketchGCD 7B 30.4± 2.5 (↓3.7)
40.6± 2.8 (↑12.4) 34.8± 2.9 (↑4.0)
51.4± 2.5 (↑24.4) 36.3± 2.2 (↑9.6)
42.5± 2.2 (↑15.7)
Claude-instant
24.5± 2.9
18.0± 2.2
20.8± 2.4
13.0± 1.7
15.2± 1.6
14.0± 1.6
+ SketchGCD 7B 44.9± 3.3 (↑20.4) 31.1± 2.7 (↑13.1) 36.7± 2.5 (↑15.9) 44.9± 2.6 (↑31.9) 31.1± 2.1 (↑15.9) 36.7± 2.1 (↑22.7)
With logit access
LLaMA-2-7B
18.3± 2.4
14.0± 1.8
15.9± 1.2
12.0± 1.5
8.6± 1.1
10.0± 1.3
+ SketchGCD 7B 23.6± 2.7 (↑5.3)
34.2± 2.9 (↑20.2) 28.0± 2.4 (↑12.1) 33.3± 2.5 (↑21.3) 21.0± 2.0 (↑12.4) 25.7± 2.1 (↑15.7)
+ CD
33.6± 2.7
32.9± 2.9
32.8± 2.5
34.0± 2.3
25.9± 2.0
29.4± 2.0
LLaMA-2-13B
22.6± 2.3
23.6± 2.4
23.1± 2.3
15.7± 1.6
12.7± 1.2
14.0± 1.5
+ SketchGCD 7B 28.8± 2.6 (↑6.2)
44.2± 3.0 (↑20.6) 34.9± 2.5 (↑11.8) 36.1± 2.0 (↑20.4) 25.1± 1.8 (↑12.4) 29.6± 1.8 (↑15.6)
+ CD
35.5± 2.6
39.1± 3.0
37.2± 2.5
39.7± 2.0
32.5± 1.8
35.7± 1.8
LLaMA-2-70B
26.1± 2.7
24.5± 2.3
25.7± 2.4
32.6± 2.0
26.9± 1.8
29.4± 1.8
+ SketchGCD 7B 26.9± 2.7 (↑0.8)
41.0± 2.6 (↑16.5) 32.5± 2.1 (↑6.8)
52.0± 2.0 (↑19.4) 37.6± 1.8 (↑10.7) 43.6± 2.0 (↑14.2)
+ CD
39.9± 2.6
46.5± 2.6
42.3± 2.1
62.7± 2.0
50.3± 2.0
55.8± 2.0
Table 1: Results for closed information extraction, in terms of triplet-based precision, recall, and F1-score (micro-
averaged, with bootstrapped 95% confidence intervals) on the Wiki-NRE and SynthIE-text datasets. The results
compare the effectiveness of SketchGCD (blue rows) against two baselines: (1) few-shot-prompted unconstrained
decoding with powerful blackbox LLMs (“without logit access”, white rows, Eq. 1) and (2) few-shot-prompted
constrained decoding (“CD”) with open-source LLMs (“with logit access”, Eq. 3). Four demonstrations are used in
few-shot prompting. LLaMA-7B serves as the constrained generator Pcg for SketchGCD.
pendix D.3). Performance is measured using micro
precision, recall, and F1-score.
Results.
We make the following observations
based on Table 1: (1) The best blackbox LLMs
(e.g., GPT-4) demonstrate strong performance even
without constrained decoding, outperforming small
open-source LLMs (LLaMA-2 7B/13B/33B) with
constrained decoding. (2) Even without requir-
ing access to logits, SketchGCD still manages to
enhance the performance of LLMs significantly
across all models of any size. (3) In case where
logit access is available, constrained decoding is
more effective than SketchGCD, as shown by the
second half of the table. Given these observations,
we conjecture that, if logits were accessible for
blackbox LLMs, a further improvement in perfor-
mance could be achieved with constrained decod-
ing. However, without logit access, SketchGCD
provides an effective alternative.
Impact of constrained decoder. We investigate
the impact of the constrained decoder on the perfor-
mance of SketchGCD. As shown in Table 2, given
GPT-4 as the sketcher, the choice of the constrained
decoder can affect the performance of SketchGCD.
Contrary to our expectations, larger constrained
Wiki-NRE
SynthIE-text
Prec
Recall
F1
Prec
Recall
F1
GPT-4
42.4
44.9
43.6
46.1
44.4
45.2
+ LLaMA-2-7B
38.7
57.1
46.1
58.9
47.3
52.4
+ LLaMA-2-13B
42.9
52.8
47.3
53.6
51.4
52.5
+ LLaMA-2-70B
35.2
54.0
42.6
58.1
53.1
55.5
Table 2: Impact of constrained decoder model (used
in step 2 of SketchGCD) on closed information extrac-
tion. GPT-4 is used as the sketcher in all cases.
decoder models do not always lead to better perfor-
mance. Our intuition is that step 2 of SketchGCD
(constrained decoding) is relatively simple, and the
additional capacity of larger constrained decoders
does not necessarily provide an advantage.
Impact of beam size. Our experiments show that
using beam search is critical for the performance
of both SketchGCD and classical constrained de-
coding. As shown in Table 3, employing beam
search (even with a minimal beam size of 2) signifi-
cantly improves performance over greedy decoding.
Larger beam sizes further enhance performance, al-
lowing the model to explore a larger search space,
but with a diminishing returns.
The following example illustrates the importance
of beam search. Suppose we are doing closed in-
Wiki-NRE
LLaMA-2-7B + CD
LLaMA-2-13B + CD
Prec
Recall
F1
Prec
Recall
F1
1 beam
29.9
22.6
25.8
32.7
32.3
32.5
2 beams
33.6
32.1
32.8
35.9
39.6
37.7
4 beams
33.7
32.9
33.3
36.0
38.5
37.2
8 beams
36.6
30.8
33.4
39.6
36.0
37.7
Table 3: Impact of beam size in beam search on closed
information extraction during classical constrained de-
coding. “1 beam” is equivalent to greedy decoding.
formation extraction on the sentence “Mona Lisa
is housed in the Musée du Louvre in Paris.” Our
entity catalog contains among other, the entities
Louvre Museum and Musée d’Orsay. During un-
constrained decoding, the model might generate
the following output with highest probability: “[s]
Mona Lisa [r] located in [o] Musée du Louvre”.
This output is invalid as the entity Musée du Louvre
is not in the entity catalog and should be rendered
as Louvre Museum instead.
With constrained decoding, the non-bold part
of the output remains unaltered, as it satisfies the
constraints. However, the bold suffix “du Louvre”
is rejected by constrained decoding because Musée
du Louvre is not in the entity catalog. The model
will be forced to sample from the allowed entity
catalog only, which can lead to “Musée d’Orsay”
as the output. In this example, greedy constrained
decoding was able to produce a valid yet incor-
rect output. On the contrary, had we used beam
search, the model would have been able to consider
both Musée du Louvre and Louvre Museum simul-
taneously, and would have been able to select the
correct entity, Louvre Museum, for the output.
3.2
Constituency parsing
Task description. Constituency parsing involves
breaking down a sentence into its syntactic com-
ponents to form a parse tree that represents the
sentence’s structure. For instance, the sentence “I
saw a fox” corresponds to the parse tree [S [NP
[PRP I]] [VP [VBD saw] [NP [DT a] [NN fox]]]].
For a visual representation of this tree, see Ap-
pendix E Fig. 5. The instructions I and Icg are
listed in Appendix E.1.
Constraints. We apply the context-free grammar
constraints in Appendix E.2 to ensure that brackets
are balanced, and labels are consistent.
Dataset and evaluation metrics. Our evaluation
uses the Penn Treebank test split. The parsing error
rate of LLMs, regardless of size, is generally high,
so we use only the shortest 25% of the samples
for evaluation (up to 128 tokens according to the
LLaMA tokenizer). We assess performance using
bracketing recall and precision, as well as tag ac-
curacy, as measured by the EVALB tool (Sekine
and Collins, 2008). Since these metrics are only
applicable to valid parse trees, and since models
typically generate valid trees only for simpler in-
puts, one needs to be careful while interpreting the
results, as weaker model may have better scores
because they only generate a small fraction of valid
parse trees (simpler ones) (Deutsch et al., 2019).
Results. The results in Table 4 show that even
advanced LLMs like GPT-4 struggle to generate
valid parse trees, especially for longer sentences.
The following observations can be made: (1) Both
SketchGCD and classical constrained decoding sig-
nificantly help the model generate more structurally
valid parse trees. (2) The other metrics mostly re-
main unchanged or slightly drop, as a larger validity
rate means more difficult examples are included in
the evaluation. (3) The most common errors in the
unconstrained setting are imbalanced brackets, in-
valid tags, and missing words, as shown in Table 5.
(4) With SketchGCD, the error rate for imbalanced
brackets and invalid tags is significantly reduced,
while the error rate for missing words increases
significantly.
Note that constrained decoding with a more so-
phisticated grammar, as described in Appendix E.2,
can achieve 100% valid trees and 100% valid tags
(see Table 9). However, as implementing such a
grammar is non-trivial, we use a simpler context-
free grammar here (see Appendix E.2) to mimic
the real-world scenario where a simpler might be
preferred over a perfect grammar.
4
Related work
Constrained decoding. Deutsch et al. (2019) intro-
duced a general constrained decoding framework
for text generation based on automata. Scholak
et al. (2021); Poesia et al. (2022); Geng et al. (2023)
implemented incremental parsing for domain-
specific tasks such as SQL generation. Beurer-
Kellner et al. (2023); Poesia et al. (2023) have pro-
posed iterative approaches to constrained decoding
using blackbox LLM APIs, albeit with potential
limitations such as excessive API calls (thus in-
creasing monetary cost), as detailed in Appendix C.
Method
Bracket prec*
Bracket recall*
Bracket F1*
Tag accuracy*
Tag validity
Tree validity
Without logit access
GPT-4
76.6± 5.0
67.7± 4.5
71.9± 4.0
95.4± 0.9
93.6± 4.0
86.0± 4.0
+ SketchGCD
75.8± 2.4 (↓0.8)
67.8± 2.4 (↓0.1)
71.5± 2.4 (↓0.4)
95.3± 0.8 (↓0.1)
100± 0.0 (↑6.4)
92.5± 4.0 (↑6.5)
GPT-3.5-Turbo
68.2± 0.7
55.5± 1.1
61.2± 0.6
93.1± 0.5
91.7± 2.4
76.9± 5.2
+ SketchGCD
68.7± 3.2 (↑0.5)
56.6± 2.8 (↑1.1)
62.1± 2.0 (↑0.9)
92.6± 1.3 (↓0.5)
100± 0.0 (↑8.3)
81.5± 4.0 (↑4.6)
Claude 2.1
73.1± 3.3
63.1± 2.6
67.7± 2.5
94.5± 1.1
95.1± 2.5
62.6± 5.2
+ SketchGCD
71.6± 3.0 (↓1.5)
62.9± 2.5 (↓0.2)
66.9± 2.6 (↓0.8)
93.4± 1.3 (↓1.1)
100± 0.0 (↑4.9)
68.7± 5.5 (↑6.1)
Claude-instant 1.2
71.3± 2.4
59.1± 1.4
64.7± 1.9
89.6± 1.6
91.7± 2.3
56.6± 5.2
+ SketchGCD
66.6± 3.3 (↓4.7)
57.4± 3.1 (↓1.7)
61.6± 3.3 (↓3.1)
87.9± 2.5 (↓1.7)
100± 0.0 (↑8.3)
67.8± 3.7 (↑11.2)
With logit access
llama-2-7B
23.1± 4
10.4± 3
14.3± 4
14.9± 3
93.2± 3
32.1± 5
+ CD
28.5± 6 (↑5.4)
16.5± 3 (↑6.1)
20.9± 5 (↑6.6)
13.8± 2 (↓1.1)
100± 0 (↑6.8)
35.1± 5 (↑3.0)
llama-2-13B
33.4± 7
22.4± 4
26.8± 5
29.3± 4
95.5± 2
38.5± 6
+ CD
33.3± 6 (↓0.1)
21.8± 5 (↓0.6)
26.3± 5 (↓0.5)
34.0± 4 (↑4.7)
100± 0 (↑4.5)
43.4± 5 (↑4.9)
llama-2-70B
45.5± 6
37.7± 5
41.2± 5
55.5± 5
75.8± 5
40.4± 6
+ CD
39.8± 6 (↓5.7)
35.6± 4 (↓2.1)
37.6± 4 (↓3.6)
53.8± 4 (↓1.7)
100± 0 ((↑24.2))
47.6± 5 (↑7.2)
Table 4: Results for constituency parsing, in terms of bracketing precision, recall, F1-score, tag accuracy, tag
validity, and parse tree validity (with bootstrapped 95% confidence intervals), on Penn Treebank test split. Only
subset of samples whose ground-truth parse trees are shorter than 128 tokens (per LLaMA tokenizer) are considered
(shortest 25% of the full dataset). Disclaimer: a weak method can have high precision by predicting very few valid
parse trees (simple ones), and a strong method can have low precision by predicting more valid parse trees including
complex ones (Deutsch et al., 2019). Four demonstrations are used in few-shot prompting. LLaMA-7B serves as
the constrained generator Pcg for SGCD. (* Considering only sentences with valid parse trees.)
Error type
Method
InvalidTag Extra Imbal Missing
GPT-4
6.4%
0.4% 10.2%
2.3%
+ SketchGCD
0.0%
0.0% 2.6%
6.0%
GPT-3.5-Turbo
8.3%
2.6% 9.4%
2.3%
+ SketchGCD
0.0%
1.5% 1.9%
16.2%
Claude 2.1
4.9%
3.8% 3.4%
30.2%
+ SketchGCD
0.0%
3.0% 3.8%
29.8%
Table 5: Error analysis for constituency parsing on
the Penn Treebank dataset. InvalidTag refers to model
generating invalid tags, Extra to model adding extra
words absent from input, Imbal to model generating
imbalanced brackets, and Missing to model dropping
words from input.
Collaborative generation. Vernikos et al. (2023)
and Welleck et al. (2023) explored training smaller
language models to refine the outputs from larger
models for enhanced quality.
The skeleton-of-
thought method (Ning et al., 2023) generates an
initial output skeleton and then concurrently de-
velops each segment. Grammar prompting (Wang
et al., 2023) creates a meta-grammar to guide the
output of LLMs in producing valid results.
5
Conclusion
So far, constrained decoding has been limited to
open-source models that provide access to their log-
its during generation. Overcoming this limitation,
we propose sketch-guided constrained decoding
(SketchGCD), a simple method for constrained de-
coding with blackbox LLMs that does not require
access to next-token logits during generation. By
using separate sketching and refinement phases,
SketchGCD allows to benefit from the power of
blackbox LLMs while still enforcing constraints.
Our work is complementary to existing methods for
constrained decoding and can be used in conjunc-
tion with them. Despite its simplicity, SketchGCD
achieves strong performance on tasks exhibiting
strong structural constraints, outperforming uncon-
strained generation by a large margin.
6
Limitations
The limitations of our method include the follow-
ing. First, SketchGCD adds an overhead as it re-
quires a constrained decoder to refine the sketches
after the sketching phase. Second, as LLMs keep
getting better, the benefits of SketchGCD might di-
minish on some tasks as the unconstrained model’s
performance improves. Third, just as classical con-
strained decoding, SketchGCD can only enforce
constraints at the structure level or the syntactic
level, but not at the semantic level. The model
can still generate semantically incorrect outputs.
However, in many real-world applications, we have
observed semantic errors to be less common than
structural errors.
Acknowledgments
We thank Grant Slatton for insightful discus-
sions during the ideation phase.
West’s lab is
partly supported by grants from Swiss National
Science Foundation (200021_185043, TMSGI2_-
211379), Swiss Data Science Center (P22_08),
H2020 (952215), Microsoft Swiss Joint Research
Center, and Google, and by generous gifts from
Meta, Google, and Microsoft.
References
Lakshya A. Agrawal, Aditya Kanade, Navin Goyal,
Shuvendu K. Lahiri, and Sriram K. Rajamani. 2023.
Guiding Language Models of Code with Global Con-
text using Monitors.
Luca Beurer-Kellner, Marc Fischer, and Martin Vechev.
2023. Prompting is programming: A query language
for large language models. Proceedings of the ACM
on Programming Languages, 7(PLDI):1946–1969.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie
Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind
Neelakantan, Pranav Shyam, Girish Sastry, Amanda
Askell, Sandhini Agarwal, Ariel Herbert-Voss,
Gretchen Krueger, Tom Henighan, Rewon Child,
Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu,
Clemens Winter, Christopher Hesse, Mark Chen, Eric
Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess,
Jack Clark, Christopher Berner, Sam McCandlish,
Alec Radford, Ilya Sutskever, and Dario Amodei.
2020. Language models are few-shot learners.
Sébastien Bubeck, Varun Chandrasekaran, Ronen El-
dan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Pe-
ter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg,
Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro,
and Yi Zhang. 2023. Sparks of artificial general in-
telligence: Early experiments with gpt-4.
Xiang Chen and Xiaojun Wan. 2023. A comprehensive
evaluation of constrained text generation for large
language models.
Sehyun
Choi,
Tianqing
Fang,
Zhaowei
Wang,
and Yangqiu Song. 2023.
KCTS: Knowledge-
Constrained Tree Search Decoding with Token-Level
Hallucination Detection. In Proceedings of the 2023
Conference on Empirical Methods in Natural Lan-
guage Processing, pages 14035–14053, Singapore.
Association for Computational Linguistics.
Daniel Deutsch, Shyam Upadhyay, and Dan Roth. 2019.
A general-purpose algorithm for constrained sequen-
tial inference. In Proceedings of the 23rd Confer-
ence on Computational Natural Language Learning
(CoNLL), pages 482–492, Hong Kong, China. Asso-
ciation for Computational Linguistics.
Saibo Geng, Martin Josifoski, Maxime Peyrard, and
Robert West. 2023. Grammar-constrained decoding
for structured NLP tasks without finetuning. In Pro-
ceedings of the 2023 Conference on Empirical Meth-
ods in Natural Language Processing, pages 10932–
10952, Singapore. Association for Computational
Linguistics.
Dan Hendrycks, Collin Burns, Steven Basart, Andy
Zou, Mantas Mazeika, Dawn Song, and Jacob Stein-
hardt. 2021. Measuring massive multitask language
understanding. Proceedings of the International Con-
ference on Learning Representations (ICLR).
Chris Hokamp and Qun Liu. 2017.
Lexically con-
strained decoding for sequence generation using grid
beam search. In Proceedings of the 55th Annual
Meeting of the Association for Computational Lin-
guistics (Volume 1: Long Papers), pages 1535–1546,
Vancouver, Canada. Association for Computational
Linguistics.
Wenyue Hua, Shuyuan Xu, Yingqiang Ge, and
Yongfeng Zhang. 2023. How to index item ids for rec-
ommendation foundation models. In Proceedings of
the Annual International ACM SIGIR Conference on
Research and Development in Information Retrieval
in the Asia Pacific Region, SIGIR-AP ’23. ACM.
Wenlong Huang, Fei Xia, Dhruv Shah, Danny Driess,
Andy Zeng, Yao Lu, Pete Florence, Igor Mordatch,
Sergey Levine, Karol Hausman, and Brian Ichter.
2023. Grounded Decoding: Guiding Text Generation
with Grounded Models for Embodied Agents.
Pere-Lluís Huguet Cabot and Roberto Navigli. 2021.
REBEL: Relation extraction by end-to-end language
generation. In Findings of the Association for Com-
putational Linguistics: EMNLP 2021, pages 2370–
2381, Punta Cana, Dominican Republic. Association
for Computational Linguistics.
Martin Josifoski, Nicola De Cao, Maxime Peyrard,
Fabio Petroni, and Robert West. 2022. GenIE: Gen-
erative information extraction. In Proceedings of
the 2022 Conference of the North American Chap-
ter of the Association for Computational Linguistics:
Human Language Technologies, pages 4626–4643,
Seattle, United States. Association for Computational
Linguistics.
Martin Josifoski, Marija Sakota, Maxime Peyrard, and
Robert West. 2023. Exploiting asymmetry for syn-
thetic training data generation: SynthIE and the case
of information extraction. In Proceedings of the 2023
Conference on Empirical Methods in Natural Lan-
guage Processing, pages 1555–1574, Singapore. As-
sociation for Computational Linguistics.
Tony Lee, Michihiro Yasunaga, Chenlin Meng, Yi-
fan Mai, Joon Sung Park, Agrim Gupta, Yunzhi
Zhang, Deepak Narayanan, Hannah Benita Teufel,
Marco Bellagente, Minguk Kang, Taesung Park, Jure
Leskovec, Jun-Yan Zhu, Li Fei-Fei, Jiajun Wu, Ste-
fano Ermon, and Percy Liang. 2023. Holistic evalua-
tion of text-to-image models.
Mitchell P. Marcus, Beatrice Santorini, and Mary Ann
Marcinkiewicz. 1993. Building a large annotated cor-
pus of English: The Penn Treebank. Computational
Linguistics, 19(2):313–330.
Xuefei Ning, Zinan Lin, Zixuan Zhou, Zifu Wang,
Huazhong Yang, and Yu Wang. 2023. Skeleton-of-
Thought: Large Language Models Can Do Parallel
Decoding. ArXiv:2307.15337 [cs].
Gabriel Poesia, Kanishk Gandhi, Eric Zelikman, and
Noah D. Goodman. 2023. Certified deductive reason-
ing with language models.
Gabriel Poesia, Alex Polozov, Vu Le, Ashish Tiwari,
Gustavo Soares, Christopher Meek, and Sumit Gul-
wani. 2022. Synchromesh: Reliable code generation
from pre-trained language models. In The Tenth In-
ternational Conference on Learning Representations,
ICLR 2022, Virtual Event, April 25-29, 2022. Open-
Review.net.
Aarne Ranta. 2019. Grammatical framework: an inter-
lingual grammar formalism. In Proceedings of the
14th International Conference on Finite-State Meth-
ods and Natural Language Processing, pages 1–2,
Dresden, Germany. Association for Computational
Linguistics.
Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta
Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola
Cancedda, and Thomas Scialom. 2023. Toolformer:
Language models can teach themselves to use tools.
Torsten Scholak, Nathan Schucher, and Dzmitry Bah-
danau. 2021. PICARD: Parsing incrementally for
constrained auto-regressive decoding from language
models. In Proceedings of the 2021 Conference on
Empirical Methods in Natural Language Processing,
pages 9895–9901, Online and Punta Cana, Domini-
can Republic. Association for Computational Lin-
guistics.
Satoshi Sekine and Michael Collins. 2008.
Evalb:
Bracket scoring program.
Dhruv Shah, Blazej Osinski, Brian Ichter, and Sergey
Levine. 2022. Lm-nav: Robotic navigation with large
pre-trained models of language, vision, and action.
Richard Shin, Christopher Lin, Sam Thomson, Charles
Chen, Subhro Roy, Emmanouil Antonios Platanios,
Adam Pauls, Dan Klein, Jason Eisner, and Benjamin
Van Durme. 2021. Constrained language models
yield few-shot semantic parsers. In Proceedings of
the 2021 Conference on Empirical Methods in Natu-
ral Language Processing, pages 7699–7715, Online
and Punta Cana, Dominican Republic. Association
for Computational Linguistics.
Bayu Distiawan Trisedya, Gerhard Weikum, Jianzhong
Qi, and Rui Zhang. 2019. Neural Relation Extraction
for Knowledge Base Enrichment. In Proceedings
of the 57th Annual Meeting of the Association for
Computational Linguistics, pages 229–240, Florence,
Italy. Association for Computational Linguistics.
Giorgos Vernikos, Arthur Bražinskas, Jakub Adamek,
Jonathan Mallinson, Aliaksei Severyn, and Eric
Malmi. 2023. Small language models improve giants
by rewriting their outputs.
Bailin Wang, Zi Wang, Xuezhi Wang, Yuan Cao, Rif A.
Saurous, and Yoon Kim. 2023. Grammar prompting
for domain-specific language generation with large
language models.
Sean Welleck, Ximing Lu, Peter West, Faeze Brah-
man, Tianxiao Shen, Daniel Khashabi, and Yejin
Choi. 2023. Generating sequences by learning to
self-correct. In The Eleventh International Confer-
ence on Learning Representations, ICLR 2023, Ki-
gali, Rwanda, May 1-5, 2023. OpenReview.net.
Kexun Zhang, Hongqiao Chen, Lei Li, and William
Wang. 2023. Syntax error-free and generalizable tool
use for llms via finite-state decoding.
A
Blackbox LLM logit access
Model
Logit bias
Token probs
MMLU
GPT-4-0614
Yes
Top 5
86.4
GPT-3.5-Turbo-0614
Yes
Top 5
70.0
Claude-2.1
No
No
78.5
Claude-instant
No
No
73.4
PaLM-2-text-bison
Yes
Top 5
78.3
Table 6: The blackbox LLMs we use in our experiments
and the access they provide to the logit distribution.
MMLU is the mainstream metric for LLM benchmark-
ing.
Logit bias indicates whether the model’s API
allows user to pass in a logit bias vector to steer
the decoding process, i.e., write access to the logit
distribution. Token probs indicates whether the
model’s API allows user to access the model’s next
token probability distribution, i.e., read access to
the logit distribution. MMLU (Hendrycks et al.,
2021) is the mainstream metric for LLM bench-
marking.
B
Grammar constrained decoding
Grammar-constrained decoding takes a formal
grammar G as input and ensures that the output
string w is a valid sentence in the formal language
L(G) defined by the grammar G. This process is
achieved through the integration of two key compo-
nents: a grammar completion engine (Poesia et al.,
2022) and a sampling method, e.g. greedy search,
nucleus sampling, etc. The grammar completion
engine is used to ensure the grammaticality of the
output string, while the LLM is used to ensure the
plausibility of the output string.
We use Grammatical Framework’s runtime pow-
ered completion engine (Ranta, 2019) with con-
strained beam search as the sampling method.
C
Logit bias-based iterative decoding
Most blackbox LLM APIs do not provide complete
access to the model’s next token probability distri-
bution at each decoding step. Nonetheless, many
allow users to input a logit bias parameter to in-
fluence the decoding process, i.e., granting users
write access but not read access to the model’s log-
its at each decoding step. This parameter accepts
a vector of logits that is added to the logits of the
next token probability distribution at each decod-
ing step. By using the logit bias parameter, users
can direct the decoding process, effectively mask-
ing the logits of invalid tokens. This approach is
particularly effective for static constraints, such as
lexical constraints (Hokamp and Liu, 2017), where
the constraints remain constant throughout the de-
coding.
However, the logit bias parameter is a static array
and does not change during the decoding process.
This makes it challenging to apply dynamic con-
straints, which change as decoding progresses, such
as constraints involving membership in formal lan-
guages (Deutsch et al., 2019; Poesia et al., 2022;
Geng et al., 2023).
A straightforward but costly solution for dy-
namic constraints is to iteratively invoke the black-
box LLMs API with updated logit bias vectors at
each decoding step (Beurer-Kellner et al., 2023;
Poesia et al., 2023; Agrawal et al., 2023; Choi et al.,
2023). However, this approach is prohibitively ex-
pensive. Each API call generates only a single
token, and the cost is calculated based on both the
input and output tokens2. The expense of itera-
tively calling the blackbox LLMs API with new
context and prefix at each step scales quadratically,
being O(n2) where n is the length of the output
sequence. Although methods like those proposed
by Beurer-Kellner et al. (2023) and Poesia et al.
(2023) use speculation to reduce the number of
API calls, the costs can remain high, especially
when the constraints are complex.
D
Task 1. closed information extraction
In this section, we provide more details about the
closed information extraction task.
D.1
Task instruction
We provide the instruction for the IE task in Fig-
ure 2. The few-shot demonstrations are rather long
and thus we do not include them here. The full
prompt is available in our code repository.
D.2
Grammar
The grammar is defined as follows, where V rep-
resents the set of variables, Σ the set of terminal
symbols, and P the set of production rules:
V = {S,T,A,B,C,E,R},Σ = {tokens}
P = {S →[ST|ϵ],T →[ABC[e]]
A →[[s] E],E →(entity1|entity2|...),
B →[[r] R],R →(rel1|rel2|...)
C →[[o] E],ϵ →</s>}
2See https://openai.com/pricing for details.
Extract the subject-relation-object triples
in fully-expanded format from texts below.
The subjects and objects are entities in
Wikidata, and the relations are Wikidata
properties. Here are a few examples.
(a) Instruction for sketcher
In this task, you will be provided with texts
along with draft annotations that represent
extracted information triples in the form
of subject-relation-object. Your role is to
refine these triples to ensure completeness
and accuracy. Here are a few examples.
(b) Instruction for constrained decoder
Figure 2: Instructions for parsing tasks.
The outputs are structured as a sequence of
triplets, where each triplet is separated by a special
marker [e]. Every triplet consists of a subject, a
relation, and an object. These elements are each
preceded by a special marker: [s] for the subject,
[r] for the relation, and [o] for the object, re-
spectively. The subject and object are pre-defined
Wikidata entities, and the relation is a pre-defined
Wikidata property. This grammar is classified as
context-free, more specifically, as a regular gram-
mar.
D.3
IE datasets
The original SynthIE-text and Wiki-NRE datasets
comprise 50,000 and 30,000 samples, respectively.
To minimize the evaluation cost on Large Language
Models (LLMs), we use a smaller subset consisting
of 1,000 samples from each dataset.
As noted by Josifoski et al. (2023), the Wiki-
NRE dataset displays a significant skew in its re-
lations distribution: the top 10 relations constitute
92% of the triplets, with the top 3 alone accounting
for 69%. To ensure our test set accurately reflects
the overall dataset, we have downscaled it to 1,000
samples to balance the distribution of relations, as
shown in Fig. 3
The SynthIE-text dataset, synthesized by re-
verse prompting Text-Davinci-003 with triplets
from Wikidata, stands out due to its substantial
size, diverse content, and high-quality human rat-
ings, as highlighted in (Josifoski et al., 2023).
This contrasts with prior datasets such as REBEL
(Huguet Cabot and Navigli, 2021), whose annota-
tion quality is low (Josifoski et al., 2022). However,
a potential minor bias may exist towards GPT-4
and GPT-3.5-Turbo, as SynthIE-text was generated
from a model in their family, Text-Davinci-003.
Despite this, we maintain that this does not com-
promise the validity of our method, given that our
primary focus is on the comparative performance
with and without the application of SketchGCD.
(a) Original relation distribution in WikiNRE test set
(b) Stratified relation distribution in WikiNRE test set
Figure 3: Relation distribution in WikiNRE before and
after stratification.
D.4
Discussion of GPT-4 on cIE
An intriguing finding is that SketchGCD’s perfor-
mance on the SynthIE-text dataset using GPT-4
(F1=45.6) is marginally lower than that achieved
with few-shot prompting alone, without con-
strained decoding (F1=45.8). Our analysis suggests
that the constrained decoder occasionally strug-
gles to adhere to the sketcher’s outline, resulting
in fewer triplets than expected in the output. This
observation is consistent with Wang et al. (2023)’s
findings, where constrained decoding was noted to
reduce the diversity of the generated samples.
More critically, since SynthIE-text is syntheti-
cally generated by reverse prompting Text-Davinci-
Ratio of invalid triplets
Wiki-NRE
SynthIE-text
Entity
Rel
Triplet
Entity
Rel
Triplet
GPT-4
19.4
21.9
45.2
7.6
28.1
37.3
GPT-3.5-Turbo
23.4
50.2
65.8
13.8
52.5
63.3
Claude
17.0
41.1
55.5
17.4
52.8
64.6
Claude-ins
19.6
48.4
62.6
13.8
43.3
52.7
SketchGCD
0
0
0
0
0
0
Table 7: Triplets grounding analysis. We report the
percentage of generated entities, relations, and triplets
that are not present in the knowledge catalogue in few-
shot unconstrained setting. The grounding precision for
constrained methods is 100% by construction, and thus
0% invalid triplets.
003 with triplets from Wikidata, its text doesn’t
exhibit the naturalness characteristic of the Wiki-
NRE dataset. For instance, sentences in SynthIE-
text often resemble direct copies with slight alter-
ations from the original entity and relation names.
This tendency facilitates the LLMs’ task of ground-
ing entities and relations in the Knowledge Graph
(KG), thereby diminishing the necessity for con-
strained decoding.
However, in real-world scenarios, text is typi-
cally more intricate, and grounding entities and
relations in the KG is not as straightforward. De-
spite this, the overall performance enhancement
provided by SketchGCD across various models re-
mains noteworthy, averaging gains of up to 10.7%
and 8.1% on Wiki-NRE and SynthIE-text, respec-
tively.
D.5
Grounding analysis
In this study, we delve into the grounding efficacy
of GPT-4’s output. A triplet is deemed grounded
when both its subject and object entities, as well
as the relation, are present in the KG. Furthermore,
for a grounded triplet to be considered correct, it
must also be part of the target triplet set.
Given that being grounded is essential but not
solely adequate for being correct, it is crucial to
assess how well GPT-4’s output aligns with the
KG. According to the data presented in Table 7,
we observe that a significant portion of the output
triplets from GPT-4 are not grounded in the KG,
amounting to 45% and 37% on the Wiki-NRE and
SynthIE-text datasets, respectively. This finding
sheds light on the importance of constrained decod-
ing, as it ensures that the output is grounded in the
KG, thereby increasing the likelihood of validity.
E
Task 2. constituency parsing
In this section, we provide more details about the
constituency parsing task.
E.1
Task instruction
We provide the instruction for the CP task in Fig-
ure 4. The few-shot demonstrations are rather long
and thus we do not include them here. The full
prompt is available in our code repository.
Perform constituency parsing on the
provided sentences in accordance
with the Penn TreeBank annotation
guidelines. Here are a few examples.
(a) Instruction for sketcher
In this task, you will be provided with a
draft annotations that represent the parse
tree of a sentence in Penn TreeBank format.
Your task is to rewrite the parse tree and
fix error if any. Here are a few examples.
(b) Instruction for constrained decoder
Figure 4: Instructions for parsing tasks.
E.2
Constraints and grammar
(a)
The
correct
con-
stituency parse tree
(b) A grammatical but in-
correct parse tree
Figure 5: Parse trees for the sentence “I saw a fox”.
Here we describe the grammar used to constrain
the generative constituency parsing task.
Linearization. A constituency parse tree is inher-
ently a recursive structure. To effectively represent
this tree as a sequence of tokens for generation by
a Large Language Model (LLM), a linearization
is required. Two common strategies for this lin-
earization are pre-order traversal and post-order
traversal.
We have chosen to adopt the pre-order traversal
strategy. This approach is also the default method
used in the PYEVALB tool (Sekine and Collins,
2008) and in the construction of the Penn Treebank
(Marcus et al., 1993). As an illustration, the parse
tree in Fig. 5a is linearized in the following format:
[S [NP [PRP I]] [VP [VBD saw] [NP [DT a] [NN
fox]]]].
The linearised parse tree needs to satisfy the
following structural constraints:
• Completeness: Every word in the sentence
needs to be included in the parse tree.
• Balanced brackets: At any point in the lin-
earized parse tree, the right bracket ] should
close a previously unclosed left bracket [
and every left bracket [ should be eventually
closed by a right bracket ].
• Label consistency: The label of terminal and
non-terminal nodes needs to be consistent
with the Penn Treebank format.
Simple Context-Free Grammar. The tree struc-
ture of the parse tree is usually captured by a
context-free grammar as shown in Table 8.
root ::= tree;
tree ::= node;
node ::= clause | phrase | word;
clause ::= spaced_open_parenthesis, space,
clause_tag, function_tag*,
index?, node*,
spaced_close_parenthesis;
phrase ::= spaced_open_parenthesis, space,
phrase_tag, function_tag*,
index?, node*,
spaced_close_parenthesis;
word ::= spaced_open_parenthesis, space,
word_tag, space, actual_word,
spaced_close_parenthesis;
clause_tag ::= "S" | ...
| "SQ";
phrase_tag ::= "ADJP" | ...| "WHADVP";
word_tag ::= "CC" |...|"WRB";
function_tag ::= "-ADV" |... | "-TTL";
actual_word ::= "xxx";
index ::= "-", [1-9], {0-9};
spaced_open_parenthesis ::= space, "(";
spaced_close_parenthesis ::= space, ")";
space ::= " ";
Table 8: Lite Context-Free Grammar for constituency
parsing.
Sophisticated Regular Grammar. However, the
context-free grammar is not sufficient to capture
the completeness constraint, motivating the use of
a more restrictive grammar. Geng et al. (2023) pro-
posed a sophisticated regular grammar to enforce
S →B0,0
Bi,j →[α(Bi,j+1 | Ci,j+1)];
Ci,j →xi (Ci+1,j | Ei+1,j);
Cn,j →En,j;
Ei,j+1 →](Ei,j | Bi,j);
En,j+1 →]En,j;
En,0 →ε;
whereα = (S | NP | VP | ...)
andxi ∈tokens
Figure 6: Sophisticated Regular Grammar for con-
stituency parsing.
the constraints of completeness, balanced brackets,
and label consistency as shown in Fig. 6.
The grammar falls into the category of regu-
lar grammar and is input-dependent.
it repro-
duces the input sentence, represented as a sequence
x = ⟨x0,...,xn−1⟩of words, in left-to-right order,
interspersing it with node labels and balanced
brackets. In order to guarantee balanced brackets,
the non-terminals Bi,j count the number of opened
left brackets [ using the second subscript index
j, and the rules ensure that the number of closed
brackets can never exceed the number of previously
opened brackets.
F
Data contamination risk
There is a rising concern over the data contam-
ination risk of evaluating LLMs on downstream
tasks. The datasets of our experiments are pub-
licly available on internet so there is a risk that the
models may have seen the data, such as the ground
true parse tree of Penn Treebank during pretraining.
However, the risk of data contamination is indepen-
dent of our method and doesn’t affect the validity
of our conclusions.
Method
Bracket-Prec
Bracket-Recall
Bracket-F1
Tag Accuracy
Valid Tag
Valid Tree
GPT-4
76.6
67.7
71.9
95.4
93.6
86.0
+ Lite Context-Free Grammar
75.8
67.8
71.5
95.3
100
92.5
+ Sophisticated Regular Grammar
69.3
63.1
66.1
98.5
100
100
GPT-3.5-Turbo
68.2
55.5
61.2
93.1
91.7
76.9
+ Lite Context-Free Grammar
68.7
56.6
62.1
92.6
100
81.5
+ Sophisticated Regular Grammar
61.2
49.4
54.7
96.0
100
100
Claude
73.1
63.1
67.7
94.5
95.1
62.6
+ Lite Context-Free Grammar
71.6
62.9
66.9
93.4
100
68.7
+ Sophisticated Regular Grammar
52.1
45.4
48.5
75.9
100
99.2
Claude-instant
71.3
59.1
64.7
89.6
91.7
56.6
+ Lite Context-Free Grammar
66.6
57.4
61.6
87.9
100
67.8
+ Sophisticated Regular Grammar
59.6
49.2
53.9
84.9
100
99.5
Table 9: Constituency parsing with two different grammar constraints, measured in terms of bracketing recall,
precision, F1-score, and tag accuracy (with bootstrapped 95% confidence intervals) †Only subset of samples whose
ground-truth parse trees are shorter than 128 tokens(LLaMAtokenizer) are considered, which accounts for shortest
25% of the samples.