Question:
Which statements are correct about CodeGRAG’s soft prompting and its cross-lingual performance characteristics reported in the paper?
A. During finetuning, the retrieved knowledge includes an expert-encoded graphical view injected via a token embedding; LoRA (rank=8, α=16, dropout=0.05) is applied to Q and V with 8-bit quantization and AdamW.
B. The hard Meta-Graph prompt yielded higher Pass@1 than Code Block in C++ and matched Code Block in Python on HumanEval-X, while multi-lingual Code Block reduced Python performance.
C. CodeGRAG requires multiple LLM calls per generation to dynamically inject per-step graph signals, increasing inference latency.
D. Extraction rates are high: in Python, CodeGRAG achieved a higher extraction rate (≈96.95%) than Code-RAG (≈91.46%), with a smaller but positive gain in C++.

Answer: ABD

Finetuning with the Expert Soft Signal. To help
the backbone LLMs to digest the graphical views,
we tune the LLMs with the expert soft signal using
supervised finetuning. The prompt for finetuning
consists of the system prompt, retrieved knowledge
where the expert encoded graphical view is contained using a token embedding, and task prompt,
which is illustrated in Figure 5 in the Appendix.
Inference. After the finetuning stage, we used
the tuned models to generate codes using the soft
prompting template as illustrated in Figure 5 in the
Appendix.

Table 1: Results of Hard Meta-Graph Prompt on Humaneval-X. (Pass@1)
Model
Retrieved Knowledge
C++
Python
GPT-3.5-Turbo
N/A
57.93
71.95
Code Block (Nashid et al., 2023; Lu et al., 2022)
60.37
72.56
Meta-Graph
62.20
72.56
(Multi-Lingual) Code-Block (Nashid et al., 2023; Lu et al., 2022)
62.20
70.12
(Multi-Lingual) Meta-Graph
64.02
77.44
GPT-4omini
N/A
63.41
78.66
Code Block (Nashid et al., 2023; Lu et al., 2022)
65.24
78.66
Meta-Graph
65.85
79.88
(Multi-Lingual) Code-Block (Nashid et al., 2023; Lu et al., 2022)
65.85
79.27
(Multi-Lingual) Meta-Graph
67.07
80.49

Table 5: The extraction rate of the generated results
passing the graphical extraction process.
Generated Codes
Pass@1
Extraction Rate
(C++) Code-RAG
62.20
92.07
(C++) CodeGRAG
64.02
92.68
(Python) Code-RAG
71.95
91.46
(Python) CodeGRAG
77.44
96.95

For the fintuning details, the learning rate and
weight decay for the expert GNN training is 0.001
and 1e-5, repectively. We apply 8-bit quantization
and use LoRA for parameter-efficient fine-tuning.
The rank of the low-rank matrices in LoRA is uni-
formly set to 8, alpha set to 16, and dropout is set
to 0.05. The LoRA modules are uniformly applied
to the Q and V parameter matrices of the attention
modules in each layer of the LLM. All the three
models are optimized using the AdamW optimizer.

CodeGRAG: Bridging the Gap between Natural Language and
Programming Language via Graphical Retrieval Augmented Generation
Kounianhua Du1, Jizheng Chen1, Renting Rui1, Huacan Chai1, Lingyue Fu1,
Wei Xia2, Yasheng Wang2, Ruiming Tang2, Yong Yu1, Weinan Zhang1
1Shanghai Jiao Tong University, 2 Huawei Noah’s Ark Lab
Shanghai, China
{kounianhuadu, wnzhang}@sjtu.edu.cn
Abstract
Utilizing large language models to generate
codes has shown promising meaning in
software development revolution.
Despite
the intelligence shown by the general large
language models, their specificity in code
generation can still be improved due to the
syntactic gap and mismatched vocabulary
existing among natural language and different
programming languages.
In this paper, we
propose CodeGRAG, a Graphical Retrieval
Augmented Code Generation framework to en-
hance the performance of LLMs. CodeGRAG
builds the graphical view of code blocks based
on the control flow and data flow of them to fill
the gap between programming languages and
natural language, which can facilitate natural
language based LLMs for better understanding
of code syntax and serve as a bridge among
different programming languages. To take the
extracted structural knowledge into the founda-
tion models, we propose 1) a hard meta-graph
prompt template to transform the challenging
graphical
representation
into
informative
knowledge for tuning-free models and 2) a soft
prompting technique that injects the domain
knowledge of programming languages into the
model parameters via finetuning the models
with the help of a pretrained GNN expert
model.
CodeGRAG significantly improves
the code generation ability of LLMs and can
even offer performance gain for cross-lingual
code generation. Implementation is available
at
https://anonymous.4open.science/r/Code-
5970/ .
1
Introduction
In recent years, large language models (LLMs)
(Achiam et al., 2023; Touvron et al., 2023a) have
shown great impact in various domains. Automated
code generation emerges as a captivating frontier
(Zheng et al., 2023; Roziere et al., 2023; Shen et al.,
2023), promising to revolutionize software develop-
ment by enabling machines to write and optimize
string string_xor(string a, string b){
string output="";
for (int i=0;(i<a.length() and i<b.length());i++){
if (i<a.length() and i<b.length()){
if (a[i]== b[i]) output+='0';
else output+='1';}
else{
if (i>=a.length()) output+=b[i];
else output+=a[i];}}
return output;}
Input are two strings a and b consisting only of 1s and 0s. Perform binary XOR
on these inputs and return results also as a string.
Start:
string_xor
Vocabulary Mismatch
for, int, =, <, ++, ...
consisting, binary
Syntactic Gap
Complex & Structural
Easier & Sequential
Programming
Language
Natural
Language
Init output,
minlen
For: i
range(minlen)
If: i<minlen
If:
a[i]=b[i]
yes
If:
i>=len(a)
yes
no
output
+= '0'
output
+= '1'
Extracted
Graph
no
yes
no
output
+= b[i]
Figure 1: Illustration of the gap between the program-
ming language and the natural language.
code with minimal human intervention.
However, syntactic gap and mismatched vocab-
ulary among natural language (NL) and program-
ming languages (PL) exist, hindering LLM’s per-
formance on code generation. As illustrated in Fig-
ure 1, programming language (marked in blue) con-
tains special tokens such as “int” or “++” that nat-
ural language (marked in yellow) doesn’t possess,
leading to vocabulary mismatch. Besides, the re-
lations between tokens in programming languages
are often structural, e.g., the complex branching
and jumps, whereas natural language is arranged
simply in sequential manner, leading to syntactic
gap. For example, in the control flow graph of
the raw code (marked in pink), two “if” blocks
(marked in purple) are adjacent and are executed
sequentially under certain condition, but they ap-
pear to be intervaled in raw textual code.
As discussed above, the innate structures of pro-
gramming languages are different from that of the
sequential-based natural language. The challenges
of enhancing a general-purposed large language
models for code-related tasks can be summarized
into two folds.
(C1) How to solve the gap between different
languages and better interpret the inherent logic
1
arXiv:2405.02355v3  [cs.SE]  8 Nov 2024
of code blocks. Code, unlike natural language,
possesses a well-defined structure that governs its
syntax and semantics. This structure provides valu-
able information about the relationships between
different parts of the code, the flow of execution,
and the overall organization of the functions (Jiang
et al., 2021; Guo et al., 2020). General-purpose
LLMs regard a code block as a sequence of tokens.
By ignoring the inherent structure of codes, they
miss out on essential cues that could help them
better understand and generate code. In addition,
the multi-lingual code generation abilities of LLMs
is challenging due to the gap among different pro-
gramming languages.
(C2) How to inject the innate knowledge of pro-
gramming languages into general purpose large lan-
guage models for enhancement. Despite the well
representation of the programming knowledge, the
ways to inject the knowledge into the NL-based
foundation models is also challenging. The struc-
tural representation of codes could be hard to un-
derstand, which poses a challenge to the capability
of the foundation models.
To solve the above challenges, we propose Code-
GRAG, a graphical retrieval augmented generation
framework for code generation. For (C1), we pro-
pose to interpret the code blocks using the com-
posed graph based on the data-flow and control-
flow of the code block, which extracts both the
semantic level and the logical level information
of the code. The composed graphical view could
1) better capture the innate structural knowledge
of codes for NL-based language models to under-
stand and 2) model the innate function of code
blocks that bridging different programming lan-
guages. For (C2), we propose a meta-graph prompt-
ing technique for tuning-free models and a soft-
prompting technique for tuned models. The meta-
graph prompt summarizes the overall information
of the extracted graphical view and transforms the
challenging and noisy graphical representation into
informative knowledge. The soft-prompting tech-
nique deals with the graphical view of codes with a
pretrained GNN expert network and inject the pro-
cessed knowledge embedding into the parameters
of the general-purpose foundation models with the
help of supervised finetuning.
The main contributions of the paper can be sum-
marized as follows:
• Novel GraphRAG framework for code gener-
ation. We propose CodeGRAG that bridges the
gap among natural language and programming
languages, transfers knowledge among different
programming languages, and enhances the abil-
ity of LLMs for code generation. CodeGRAG
requires only one calling of LLMs and can offer
multi-lingual enhancement.
• Effective graphical view to inform and stimu-
late the structural programming knowledge of
LLMs. We propose an effective graphical view
to purify the semantic and logic knowledge from
the code space, which offers more useful informa-
tion than the raw code block and can summarize
the cross-lingual knowledge.
• Effective soft prompting technique to preserve
the programming domain knowledge and in-
ject it into LLMs parameters. We propose
an effective soft prompting technique, which in-
jects the domain knowledge of programming lan-
guages into the model parameters via finetuning
LLMs with the assistance of a pretrained GNN
expert model.
2
Methodology
2.1
Overview
In this paper, we leverage both generative models
and retrieval models to produce results that are
both coherent and informed by the expert graphical
knowledge of programming language. The overall
process of CodeGRAG is illustrated in Figure 2,
which mainly consists of three stages: graphical
knowledge base preparation, knowledge querying,
and graphical knowledge augmented generation.
2.2
Graphical Knowledge Base Preparation
In this section, we discuss how to extract informa-
tive graphical views for code blocks. We analyze
the syntax and control information of code blocks
and extract their graphical views to better repre-
sent the codes. This process can be formulated as,
∀ci ∈Dpool:
gi ←−GraphExtractor(ci),
(1)
KB.append(⟨ci, gi⟩),
(2)
where ci is the raw code block and gi is the corre-
sponding extracted graphical view.
To capture both the semantic and the logical
information, we propose to combine the data flow
graph (Aho et al., 2006) and the control flow graph
(Allen, 1970) with the read-write signals (Long
2
External Code
Knowledge Base
Knowledge
Document
std::vector<int>
twoSum(std::vector<int>& nums, int
target) { .....
Composed Syntax Graph
Control Flow
Data Flow
Raw Code
Compact
Knowledge
Document
Knowledge
Document
Node Feature
Edge Feature
Meta-Graph Structure
Structure Preserving Contrastive Learning
Code
Graphical
View
NL
Alignment Contrastive Learning
Expert GNN
GraphEmb
Composed
Syntax Graph
Composed
Syntax Graph
Graphical View
Ret ur n a st r i ng cont ai ni ng space- del i mi t ed
number s st ar t i ng f r om 0 upt o n i ncl usi ve.
>>> st r i ng_sequence( 0)
" 0"
>> st r i ng_sequence( 5)
0 1 2 3 4 5
#i ncl ude<st di o. h>
#i ncl ude<st r i ng>
usi ng namespace st d;
st r i ng st r i ng_sequence( i nt  n) {
INPUT: Task instruction prompt
uery Extraction
Retriever
#include<stdio.h>
#include<math.h>
#include<string>
using namespace std;
#include<algorithm>
#include<stdlib.h>
string string_sequence(int n){
    string out="0";
    for (int i=1;i<=n;i++)
    out=out+" "+to_string(i);
    return out;
}
OUTPUT: Generated Code
External Code
Knowledge Base
Index
Graphical View
GraphEmb
General Purpose LLM
General Purpose LLM
Soft Prompting
Hard Meta-Graph
Prompt
Knowledge  Preparation
Retrieval Augmented Generation
(Soft &  Hard)
Hard
Graphical View
Soft
GraphEmb
Figure 2: Overview of CodeGRAG. (Top) Knowledge Preparation. We extract the control flow and data flow of
each external code block and compose them using the read-write signal to obtain the semantic and logical expression
of each code block, which is then abstracted into graphical view as hard knowledge document and embedded into
GraphEmb as soft knowledge document. The GraphEmb is encoded by a pretrained GNN expert model constrained
by the alignment and structure preserving objectives. (Bottem) Retrieval Augmented Generation. We extract
query from the task input and retrieve from the external corpus. For tuning free models, we use the hard graphical
view to stimulate the structural programming knowledge of LLMs for enhanced generation. For tunable models, we
use the soft GraphEmb and inject the programming domain knowledge into LLMs parameters via finetuning them
with the GNN expert signals. The expert signals informed LLMs can then produce enhanced generation.
et al., 2022) to represent the code blocks, both of
them are constructed on the base of the abstract
syntax tree.
Abstract Syntax Tree (AST). An abstract syntax
tree (AST) is a tree data structure that represents
the abstract syntactic structure of source code. An
AST is constructed by a parser, which reads the
source code and creates a tree of nodes. Each node
in the tree represents a syntactic construct in the
source code, such as a statement, an expression, or
a declaration. ASTs have good compactness and
can represent the structure of the source code in a
clear and concise way.
Data Flow Graph (DFG). The data flow graph
(DFG) is a graphical representation of the flow of
data dependencies within a program. It is a directed
graph that models how data is transformed and
propagated through different parts of a program. In
DFG, nodes are operands and edges indicate data
flows. Two types of edges are considered: 1) opera-
tion edges that connect the nodes to be operated and
the nodes that receive the operation results; 2) func-
tion edges that indicate data flows for function calls
and returns. These edges connect nodes, including
non-temporary operands and temporary operands,
which refer to variables and constants that explic-
itly exist in the source code, and variables existing
only in execution, respectively.
Control Flow Graph (CFG). The control flow
graph (CFG) is a graphical representation of the
flow of control or the sequence of execution within
a program. It is a directed graph that models the
control relationships between different parts of a
program. Based on compiler principles, we slightly
adjust the design of CFG to better capture the key
information of the program. Nodes in CFG are
operations in the source code, including standard
operations, function calls and returns. Edges indi-
cate the execution order of operations.
Composed Syntax Graph. A composed syntax
graph composes the data flow graph and the control
flow graph with the read-write flow existing in the
code blocks. An illustration of the extracted com-
posed syntax graph is displayed in Figure 3. Dif-
ferent edge types along with their concrete names
are given in colors. As for the node names, the
3
Checks if given string is a palindrome.
     #include<stdio.h>
     #include<math.h>
     #include<string>
     using namespace std;
     #include<algorithm>
     #include<stdlib.h>
     bool is_palindrome(string text){
         string pr(text.rbegin(),text.rend());
         return pr==text;
     }
write
child
DeclStmtedge0
read
CXXOperatorCallExpredge1
CXXOperatorCallExpredge2
next
UserDefineFun
Function Desciption
Code Block
Figure 3: Illustration of the extracted composed syntax graph from the code block. The arrows in the bottom part
indicate the names of different edges, which are extracted based on the ASTs.
middle figure displays the concrete types of nodes
(operands) and the right figure displays the proper-
ties of nodes.
An illustration of the composed graphical view
is in Figure 3. After obtaining the composed syn-
tax graphs, we use them to inform the general-
purpose LLMs to bridge the gap between NL and
PLs, where both the semantic level and the logic
level information are preserved.
2.3
Knowledge Querying
Given a target problem to be completed, we gen-
erate informative query of it and use it to retrieve
graphical knowledge from the constructed knowl-
edge base.
We extract the problem description of each task
to reduce the ambiguity and then concatenate it
with the function declaration to serve as the query
content, where the functionality and input format of
the expected code block are contained. The query
of the retrieval includes problem description Qp
and function description Qc, while each content of
the retrieval pool includes raw code block Vc and
its graphical view Vg.
To expressively represent the components, we
use the encoder ϕ(·) of the pretrained NL2Code
model to represent the problem description and
code snippets. The retrieval function is:
hV = ϕ(Vc∥Vg),
(3)
hQ = ϕ(Qp∥Qc),
(4)
Distance = 1 −
hQ · hV
∥hQ∥· ∥hV∥.
(5)
2.4
Graphical Knowledge Augmented
Generation
After we obtain the returned graphical view, we in-
ject it to the foundation LLMs for graphical knowl-
edge augmented generation. Since the graphical
view is hard to understand, we propose 1) a meta-
graph template to transform the graphical view into
informative knowledge for tuning-free model and
2) a soft prompting technique to tune the founda-
tion models for their better understanding of the
graphical views with the assistance of an expert
GNN model.
2.4.1
Hard Meta-Graph Prompt
The original graphical view of a code block could
contain hundreds of nodes and edges. A full de-
scription of it could cost a overly long context,
along with the understanding challenge posed by
the long edge lists. Therefore, we propose to use
a meta-graph template to abstract the information
of the graphical view. The abstracted meta-graph
consists of the canonical edge types and node types,
which describes the basic topology of the graphical
view (Sun and Han, 2013), with the textual features
obtained from the ASTs contained in the node and
edge features.
Then we use the meta-graph template to trans-
form the retrieved graphical view into digestable
knowledge and insert it into the final prompt for
generation. As illustrated in Figure 4 in the Ap-
pendix, the final prompt consists of three compo-
nents: the system prompt illustrated in the blue
part, the retrieved knowledge and hints illustrated
in the green part, and the problem (including task
description, function declaration, etc.) illustrated
in the yellow part. The three parts are concatenated
to be fed into LLMs for knowledge augmented
generation.
2.4.2
Soft Prompting with the Expert
Directly hard prompt to the LLMs poses a chal-
lenge to the digesting capability of the backbone
4
LLMs, which could fail under the case where the
backbone LLMs cannot well understand the graph
components. To compress the graphical knowl-
edge into model parameters and help the backbone
LLMs to better understand the programming lan-
guage, we propose a soft prompting technique. The
overall procedure can summarized into expert en-
coding of graphical views, finetuning with the ex-
pert signal, and inference.
Expert Encoding of Graphical Views. We design
a graph neural network to preserve the semantic
and logical information of code blocks. The rep-
resentation of each node n(0)
i
and edge e(0)
i
are
first initialized with vectors corresponding to the
node text and edge text encoded by ϕ1. A message
passing process is first conducted to fuse the se-
mantic and structural information into each node
representation.
m(l)
ij = W(l)(n(l−1)
i
∥e(l−1)
ij
),
(6)
Q(l)
j
=WQ(l)n(l−1)
j
,
(7)
K(l)
ij = WK(l)m(l)
ij ,
V(l)
ij = WV(l)m(l)
ij ,
(8)
a(l)
ij = softmaxi∈N(j)(Q(l)
j K(l)
ij ),
(9)
n(l)
j
=
X
i∈N(j)
a(l)
ij V(l)
ij .
(10)
A global attention-based readout is then applied
to obtain the graph representation:
g =
X
i
softmax(fgate(nL
i ))ffeat(nL
i ).
(11)
The expert encoding network is optimized via
the contrastive learning based self-supervised train-
ing, which includes the intra-modality contrastive
learning and inter-modality contrastive learning.
The intra-modality constrastive learning serves
for preserving the modality information, while
the inter-modality contrastive learning serves for
modality alignment.
• Alignment Contrastive Learning. There are
two types of alignment to be ensured: 1) NL-
Code (NC) alignment and 2) Code-Graph (CG)
alignment. We define the positive pairs for NC
alignment purpose as I+
NC = {⟨hV
i , hQ
i ⟩|i ∈
Dtrain} and define the negative pairs for NC align-
ment purpose as I−
NC = {⟨hV
i , hQ
j ⟩|i ̸= j, i ∈
Dtrain, j ∈Dtrain}.
And we define the positive pairs for CG align-
ment purpose as I+
CG = {⟨ϕ1(ci), ϕ2(gi)⟩|i ∈
Dtrain} and define the negative pairs for CG align-
ment purpose as I−
CG = {⟨ϕ1(ci), ϕ2(gj)⟩|i ̸=
j, i ∈Dtrain, j ∈Dtrain}.
• Structure Preserving Contrastive Learning.
To preserve the structural information of the
graphical views, we perform intra-modality con-
trastive learning among the graphical views and
their corrupted views. Concretely, we corrupt
each of the graphical view gi with the edge
dropping operation to obtain its corrupted view
g′
i. The positive pairs for structure-preserving
purpose are then designed as I+
preserve
=
{⟨ϕ2(gi), ϕ2(g′
i)⟩|i ∈Dtrain}. The negative pairs
for structure preserving purpose are designed
as I−
preserve = {⟨ϕ2(gi), ϕ2(g′
j)⟩|i ̸= j, i ∈
Dtrain, j ∈Dtrain}.
Finetuning with the Expert Soft Signal. To help
the backbone LLMs to digest the graphical views,
we tune the LLMs with the expert soft signal using
supervised finetuning. The prompt for finetuning
consists of the system prompt, retrieved knowledge
where the expert encoded graphical view is con-
tained using a token embedding, and task prompt,
which is illustrated in Figure 5 in the Appendix.
Inference. After the finetuning stage, we used
the tuned models to generate codes using the soft
prompting template as illustrated in Figure 5 in the
Appendix.
3
Experiments
RQ1 Does the proposed CodeGRAG offer perfor-
mance gain against the base model?
RQ2 Does the proposed graph view abstract more
informative knowledge compared with the
raw code block?
RQ3 Can soft prompting enhance the capability of
the backbone LLMs? Does finetuning with
the soft prompting outperforms the simple
supervised finetuning?
RQ4 Are the proposed pretraining objectives for
the GNN expert effective?
RQ5 What is the impact of each of the components
of the graphical view?
RQ6 How is the compatibility of the graphical
view?
5
Table 1: Results of Hard Meta-Graph Prompt on Humaneval-X. (Pass@1)
Model
Retrieved Knowledge
C++
Python
GPT-3.5-Turbo
N/A
57.93
71.95
Code Block (Nashid et al., 2023; Lu et al., 2022)
60.37
72.56
Meta-Graph
62.20
72.56
(Multi-Lingual) Code-Block (Nashid et al., 2023; Lu et al., 2022)
62.20
70.12
(Multi-Lingual) Meta-Graph
64.02
77.44
GPT-4omini
N/A
63.41
78.66
Code Block (Nashid et al., 2023; Lu et al., 2022)
65.24
78.66
Meta-Graph
65.85
79.88
(Multi-Lingual) Code-Block (Nashid et al., 2023; Lu et al., 2022)
65.85
79.27
(Multi-Lingual) Meta-Graph
67.07
80.49
Table 2: Results of Soft Prompting. (Pass@1)
Model
Finetune
CodeForce (C++)
APPS (Python)
Gemma 7b
N/A
12.83
5.09
SFT
14.76
21.09
Soft Prompting
19.13
26.15
Llama2 13b
N/A
9.61
7.29
SFT
11.88
12.06
Soft Prompting
13.62
12.74
CodeLlama 7b
N/A
5.20
24.41
SFT
9.87
26.15
Soft Prompting
11.09
30.26
3.1
Setup
In this paper, we evaluate CodeGRAG with the
widely used HumanEval-X (Zheng et al., 2023)
dataset, which is a multi-lingual code benchmark
and CodeForce dataset in which we collect real-
world programming problems from codeforces1
website. For CodeForce dataset we include prob-
lems categorized by different difficulty levels corre-
sponding to the website and select 469 problems of
difficulty level A for testing. We use greedy decod-
ing strategy to do the generation. The evaluation
metric is Pass@1. More details of the retrieval pool
and the finetuning setting can be found in Section A
in the Appendix.
3.2
Main Results
The main results are summarized in Table 1 and Ta-
ble 2. From the results, we can draw the following
conclusions.
RQ1. The proposed CodeGRAG could offer per-
formance gain against the base model, which val-
idates the effectiveness of the proposed graphical
retrieval augmented generation for code generation
1https://codeforces.com/
framework.
RQ2. The model informed by the meta-graph
(CodeGRAG) could beat model informed by the
raw code block. From the results, we can see that
the proposed graph view could summarize the use-
ful structural syntax information and filter out the
noises, which could offer more informative knowl-
edge hints than the raw code block. In addition,
inserting the intermediate representations of codes
into the prompt can stimulate the corresponding
programming knowledge of LLMs.
RQ3. From Table 2, we can see that finetuning
with the expert soft prompting could offer more per-
formance gain than that brought by simple super-
vised finetuning. This validates the effectiveness
of the designed pretraining expert network and the
technique of finetuning with soft prompting, which
injects the programming domain knowledge into
the LLMs parameters and inform the models with
the structural information for gap filling.
3.3
Impacts of the pretraining objectives for
the expert GNN (RQ4)
To study the effectiveness of the proposed pretrain-
ing objectives for the expert GNN, we remove each
6
Table 3: Ablation studies on the GNN pretraining losses.
Model
Finetune
CodeForce (C++)
APPS (Python)
Gemma 7b
Soft Prompting
19.13
26.15
w/o Alignment
7.88
28.58
w/o Structure-Preserving
11.70
21.50
Llama2 13b
Soft Prompting
13.62
12.74
w/o Alignment
11.79
10.76
w/o Structure-Preserving
5.50
11.09
CodeLlama 7b
Soft Prompting
11.09
30.26
w/o Alignment
10.92
29.45
w/o Structure-Preserving
10.66
26.59
objective to yield different expert GNNs. The re-
sults are in Table 3.
From the results, we could see that both the
Alignment and the Structure Preserving contribute
to the expressiveness of the expert GNN model.
The alignment pretraining objective helps to pro-
mote the alignment among natural language, pro-
gramming language, and their graphical views. The
structure preserving objective helps to preserve the
innate data-flows and control-flows information
of code blocks. The two objectives collaborate
with each other to yield expressive programming
domain knowledge GNN expert model, which en-
codes external programming knowledge and injects
the knowledge into LLMs parameters.
3.4
Impacts of the Components of the
Graphical View (RQ5)
In this section, we adjust the inputs of the graphical
components to the LLMs. Concretely, we study
the information contained in node names, edge
names, and the topological structure. The results
are presented in Table 4.
Table 4: The impacts of the graph components.
Datasets
Python
C++
Edge Type Only
73.78
61.59
Edge Type + Node Name
75.00
59.76
Edge Type + Node Type
75.61
59.15
Edge Type + Topological
77.44
64.02
The edge type refers to the type of flows between
operands (child, read, write, etc.), the node type
refers to the type of operands (DeclStmt, temp,
etc.), the node name refers to the name of the inter-
mediate variables, and the topological information
refers to the statistics of the concrete numbers of
different types of edges. From the results, we can
observe that 1) the edge features matter the most
in constructing the structural view of code blocks
for enhancement, 2) the type of nodes expresses
the most in representing operands information, and
3) the overall structure of the graphical view also
gives additional information.
3.5
Compatibility Discussion of the Graphical
Views(RQ6)
Despite the effectiveness of the proposed graphical
views to represent the code blocks, the flexibility
and convenience of applying the graphical views
extraction process is important for wider applica-
tion of the proposed method. In this section, we
discuss the compatibility of CodeGRAG.
First of all, the extraction process of all the graph-
ical views are front-end. Therefore, this extraction
process applies to a wide range of code, even error
code. One could also use convenient tools to refor-
mulate the code and improve the pass rate of the
extraction process.
In addition, we give the ratio of generated results
that can pass the graphical views extraction process,
which is denoted by Extraction Rate. The Pass@1
and the Extraction Rate of the generated results
passing the graphical extraction process are given
in Table 5.
Table 5: The extraction rate of the generated results
passing the graphical extraction process.
Generated Codes
Pass@1
Extraction Rate
(C++) Code-RAG
62.20
92.07
(C++) CodeGRAG
64.02
92.68
(Python) Code-RAG
71.95
91.46
(Python) CodeGRAG
77.44
96.95
From the results, we could see that the extraction
rates are high for codes to pass the graphical views
extraction process, even under the situation where
the Pass@1 ratios of the generated results are low.
7
This indicates that the application range of the pro-
posed method is wide. In addition, as the code
RAG also offers performance gains, one could use
multiple views as the retrieval knowledge.
4
Related Work
LLMs for NL2Code. The evolution of the Natural
Language to Code translation (NL2Code) task has
been significantly influenced by the development
of large language models (LLMs). Initially, gen-
eral LLMs like GPT-J (Radford et al., 2023), GPT-
NeoX (Black et al., 2022), and LLaMA (Touvron
et al., 2023a), despite not being specifically tailored
for code generation, showed notable NL2Code ca-
pabilities due to their training on datasets contain-
ing extensive code data like the Pile (Gao et al.,
2020) and ROOTS (Laurençon et al., 2022). To
further enhance these capabilities, additional pre-
training specifically focused on code has been em-
ployed. PaLM-Coder, an adaptation of the PaLM
model (Chowdhery et al., 2023), underwent further
training on an extra 7.8 billion code tokens, signifi-
cantly improving its performance in code-related
tasks. Similarly, Code LLaMA (Roziere et al.,
2023) represents an advancement of LLaMA2 (Tou-
vron et al., 2023b), benefiting from extended train-
ing on over 500 billion code tokens, leading to
marked improvements over previous models in
both code generation and understanding. These
developments underscore the potential of adapting
generalist LLMs to specific domains like NL2Code
through targeted training, leading to more effective
and efficient code translation solutions.
Code Search. The code search methods can be
summarized into three folds. Early methods uti-
lizes sparse search to match the query and codes
(Hill et al., 2011; Yang and Huang, 2017), which
suffers from mismatched vocabulary due to the
gap between natural language and codes. Neural
methods (Cambronero et al., 2019; Gu et al., 2021)
then focus on mapping the query and codes into
a joint representation space for more accurate re-
trieval. With the success of pretrained language
models, many methods propose to use pretraining
tasks to improve the code understanding abilities
and align different language spaces. For example,
CodeBERT (Feng et al., 2020) is pretrained on
NL-PL pairs of 6 programming languages with the
masked language modeling and replaced token de-
tection task. CodeT5 (Wang et al., 2021) supports
both code-related understanding and generation
tasks through bimodal dual generation. UniXcoder
(Guo et al., 2022) integrates the aforementioned
pretraining tasks, which is a unified cross-modal
pre-trained model. As retrieval augmented genera-
tion (RAG) shows its significance in promoting the
quality of LLMs generation, works in code RAG
start to accumulate. (Nashid et al., 2023; Lu et al.,
2022) utilize the code blocks as the retrieved knowl-
edge to inform the LLMs with similar code blocks
for enhancement. (Zhou et al., 2022) uses the pro-
gramming related document to serve as the retrieval
content, injecting auxiliary external programming
knowledge into the LLMs generation.
Code Representation. Early methods regard code
snippets as sequences of tokens, assuming the ad-
jacent tokens will have strong correlations. This
line of methods (Harer et al., 2018; Ben-Nun et al.,
2018; Feng et al., 2020; Ciniselli et al., 2021) take
programming languages as the same with the nat-
ural language, using language models to encode
the code snippets too. However, this ignoring of
the inherent structure of codes leads to a loss of
expressiveness. Methods that take the structural in-
formation of codes into consideration then emerge.
Mou et al. (2016) used convolution networks over
the abstract syntax tree (AST) extracted from codes.
Alon et al. (2019) encoded paths sampled from the
AST to represent codes. Further exploration into
the graphical representation of codes (Allamanis
et al., 2017) is conducted to better encode the struc-
tures of codes, where more intermediate states of
the codes are considered.
5
Conclusion
Despite the expanding role of LLMs in code gen-
eration, there are inherent challenges pertaining to
their understanding of code syntax. General large
language models trained mainly on sequential-
based natural language cannot well understand the
structural-based programming language, e.g., the
branching and jumping in codes. This paper pro-
poses an effective way to build a graphical view
of codes to better inform LLMs for code genera-
tion. To take the challenging structural graphical
knowledge into LLMs, a meta-graph prompt is pro-
posed for tuning-free models and a soft-prompting
technique is proposed to inject the structural pro-
gramming domain knowledge into the parameters
of LLMs. By integrating external structural knowl-
edge, CodeGRAG enhances LLMs’ comprehen-
sion of code syntax and empowers them to generate
code with improved accuracy and fluency.
8
Limitations
In this paper, we propose a graphical retrieval aug-
mented generation method that can offer enhanced
code generation. Despite the efficiency and effec-
tiveness, there are also limitations within this work.
For example, dependency on the quality of the ex-
ternal knowledge base could be a potential concern.
The quality of the external knowledge base could
be improved with regular expression extraction on
the noisy texts and codes.
Acknowledgments
This document has been adapted by Emily All-
away from the instructions for earlier ACL and
NAACL proceedings, including those for NAACL
2024 by Steven Bethard, Ryan Cotterell and
Rui Yan, ACL 2019 by Douwe Kiela and Ivan
Vuli´c, NAACL 2019 by Stephanie Lukin and Alla
Roskovskaya, ACL 2018 by Shay Cohen, Kevin
Gimpel, and Wei Lu, NAACL 2018 by Margaret
Mitchell and Stephanie Lukin, BibTEX suggestions
for (NA)ACL 2017/2018 from Jason Eisner, ACL
2017 by Dan Gildea and Min-Yen Kan, NAACL
2017 by Margaret Mitchell, ACL 2012 by Mag-
gie Li and Michael White, ACL 2010 by Jing-
Shin Chang and Philipp Koehn, ACL 2008 by Jo-
hanna D. Moore, Simone Teufel, James Allan, and
Sadaoki Furui, ACL 2005 by Hwee Tou Ng and
Kemal Oflazer, ACL 2002 by Eugene Charniak and
Dekang Lin, and earlier ACL and EACL formats
written by several people, including John Chen,
Henry S. Thompson and Donald Walker. Addi-
tional elements were taken from the formatting
instructions of the International Joint Conference
on Artificial Intelligence and the Conference on
Computer Vision and Pattern Recognition.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama
Ahmad, Ilge Akkaya, Florencia Leoni Aleman,
Diogo Almeida, Janko Altenschmidt, Sam Altman,
Shyamal Anadkat, et al. 2023. Gpt-4 technical report.
arXiv preprint arXiv:2303.08774.
Alfred V Aho, Monica S Lam, Ravi Sethi, and Jeffrey D
Ullman. 2006. Compilers: Principles techniques and
tools. 2007. Google Scholar Google Scholar Digital
Library Digital Library.
Miltiadis Allamanis, Marc Brockschmidt, and Mah-
moud
Khademi.
2017.
Learning
to
repre-
sent programs with graphs.
arXiv preprint
arXiv:1711.00740.
Frances E Allen. 1970. Control flow analysis. ACM
Sigplan Notices, 5(7):1–19.
Uri Alon, Meital Zilberstein, Omer Levy, and Eran
Yahav. 2019. code2vec: Learning distributed rep-
resentations of code. Proceedings of the ACM on
Programming Languages, 3(POPL):1–29.
Tal Ben-Nun, Alice Shoshana Jakobovits, and Torsten
Hoefler. 2018. Neural code comprehension: A learn-
able representation of code semantics. Advances in
Neural Information Processing Systems, 31.
Sid Black, Stella Biderman, Eric Hallahan, Quentin
Anthony, Leo Gao, Laurence Golding, Horace He,
Connor Leahy, Kyle McDonell, Jason Phang, et al.
2022. Gpt-neox-20b: An open-source autoregressive
language model. arXiv preprint arXiv:2204.06745.
Jose Cambronero, Hongyu Li, Seohyun Kim, Koushik
Sen, and Satish Chandra. 2019. When deep learning
met code search. In Proceedings of the 2019 27th
ACM Joint Meeting on European Software Engineer-
ing Conference and Symposium on the Foundations
of Software Engineering, pages 964–974.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin,
Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul
Barham, Hyung Won Chung, Charles Sutton, Sebas-
tian Gehrmann, et al. 2023. Palm: Scaling language
modeling with pathways. Journal of Machine Learn-
ing Research, 24(240):1–113.
Matteo Ciniselli, Nathan Cooper, Luca Pascarella,
Denys Poshyvanyk, Massimiliano Di Penta, and
Gabriele Bavota. 2021. An empirical study on the
usage of bert models for code completion. In 2021
IEEE/ACM 18th International Conference on Mining
Software Repositories (MSR), pages 108–119. IEEE.
Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xi-
aocheng Feng, Ming Gong, Linjun Shou, Bing Qin,
Ting Liu, Daxin Jiang, et al. 2020. Codebert: A
pre-trained model for programming and natural lan-
guages. arXiv preprint arXiv:2002.08155.
Leo Gao, Stella Biderman, Sid Black, Laurence Gold-
ing, Travis Hoppe, Charles Foster, Jason Phang, Ho-
race He, Anish Thite, Noa Nabeshima, et al. 2020.
The pile: An 800gb dataset of diverse text for lan-
guage modeling. arXiv preprint arXiv:2101.00027.
Jian Gu, Zimin Chen, and Martin Monperrus. 2021.
Multimodal representation for neural code search. In
2021 IEEE International Conference on Software
Maintenance and Evolution (ICSME), pages 483–
494. IEEE.
Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming
Zhou, and Jian Yin. 2022. Unixcoder: Unified cross-
modal pre-training for code representation. arXiv
preprint arXiv:2203.03850.
Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu
Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey
9
Svyatkovskiy, Shengyu Fu, et al. 2020. Graphcode-
bert: Pre-training code representations with data flow.
arXiv preprint arXiv:2009.08366.
Jacob A Harer, Louis Y Kim, Rebecca L Russell, Onur
Ozdemir, Leonard R Kosta, Akshay Rangamani,
Lei H Hamilton, Gabriel I Centeno, Jonathan R Key,
Paul M Ellingwood, et al. 2018. Automated software
vulnerability detection with machine learning. arXiv
preprint arXiv:1803.04497.
Emily Hill, Lori Pollock, and K Vijay-Shanker. 2011.
Improving source code search with natural language
phrasal representations of method signatures. In 2011
26th IEEE/ACM International Conference on Auto-
mated Software Engineering (ASE 2011), pages 524–
527. IEEE.
Xue Jiang, Zhuoran Zheng, Chen Lyu, Liang Li, and
Lei Lyu. 2021. Treebert: A tree-based pre-trained
model for programming language. In Uncertainty in
Artificial Intelligence, pages 54–63. PMLR.
Hugo Laurençon, Lucile Saulnier, Thomas Wang,
Christopher Akiki, Albert Villanova del Moral, Teven
Le Scao, Leandro Von Werra, Chenghao Mou, Ed-
uardo González Ponferrada, Huu Nguyen, et al. 2022.
The bigscience roots corpus: A 1.6 tb composite mul-
tilingual dataset. Advances in Neural Information
Processing Systems, 35:31809–31826.
Ting Long, Yutong Xie, Xianyu Chen, Weinan Zhang,
Qinxiang Cao, and Yong Yu. 2022. Multi-view graph
representation for programming language process-
ing: An investigation into algorithm detection. In
Proceedings of the AAAI Conference on Artificial
Intelligence, volume 36, pages 5792–5799.
Shuai Lu, Nan Duan, Hojae Han, Daya Guo, Seung-
won Hwang, and Alexey Svyatkovskiy. 2022. Reacc:
A retrieval-augmented code completion framework.
arXiv preprint arXiv:2203.07722.
Lili Mou, Ge Li, Lu Zhang, Tao Wang, and Zhi Jin. 2016.
Convolutional neural networks over tree structures
for programming language processing. In Proceed-
ings of the AAAI conference on artificial intelligence,
volume 30.
Noor Nashid, Mifta Sintaha, and Ali Mesbah. 2023.
Retrieval-based prompt selection for code-related
few-shot learning. In 2023 IEEE/ACM 45th Interna-
tional Conference on Software Engineering (ICSE),
pages 2450–2462. IEEE.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brock-
man, Christine McLeavey, and Ilya Sutskever. 2023.
Robust speech recognition via large-scale weak su-
pervision. In International Conference on Machine
Learning, pages 28492–28518. PMLR.
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten
Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi,
Jingyu Liu, Tal Remez, Jérémy Rapin, et al. 2023.
Code llama: Open foundation models for code. arXiv
preprint arXiv:2308.12950.
Bo Shen, Jiaxin Zhang, Taihong Chen, Daoguang Zan,
Bing Geng, An Fu, Muhan Zeng, Ailun Yu, Jichuan
Ji, Jingyang Zhao, et al. 2023. Pangu-coder2: Boost-
ing large language models for code with ranking feed-
back. arXiv preprint arXiv:2307.14936.
Yizhou Sun and Jiawei Han. 2013. Mining heteroge-
neous information networks: a structural analysis
approach. ACM SIGKDD explorations newsletter,
14(2):20–28.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier
Martinet, Marie-Anne Lachaux, Timothée Lacroix,
Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal
Azhar, et al. 2023a.
Llama:
Open and effi-
cient foundation language models. arXiv preprint
arXiv:2302.13971.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Al-
bert, Amjad Almahairi, Yasmine Babaei, Nikolay
Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti
Bhosale, et al. 2023b.
Llama 2: Open founda-
tion and fine-tuned chat models.
arXiv preprint
arXiv:2307.09288.
Yue Wang, Weishi Wang, Shafiq Joty, and Steven CH
Hoi. 2021.
Codet5:
Identifier-aware unified
pre-trained encoder-decoder models for code un-
derstanding and generation.
arXiv preprint
arXiv:2109.00859.
Yangrui Yang and Qing Huang. 2017. Iecs: Intent-
enforced code search via extended boolean model.
Journal of Intelligent & Fuzzy Systems, 33(4):2565–
2576.
Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan
Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang,
Yang Li, et al. 2023. Codegeex: A pre-trained model
for code generation with multilingual evaluations on
humaneval-x. arXiv preprint arXiv:2303.17568.
Shuyan Zhou,
Uri Alon,
Frank F Xu,
Zhiruo
Wang, Zhengbao Jiang, and Graham Neubig. 2022.
Docprompting: Generating code by retrieving the
docs. arXiv preprint arXiv:2207.05987.
A
Implementation Details
For the size of retrieval pool, we use 11,913 C++
code snippets and 2,359 python code snippets. Due
to the limited access, we do not use a large re-
trieval corpus for our experiment, which can be
enlarged by other people for better performance.
We also attach the graph extraction codes for both
languages and all other expeirment codes here:
https://anonymous.4open.science/r/Code-5970/
For the fintuning details, the learning rate and
weight decay for the expert GNN training is 0.001
and 1e-5, repectively. We apply 8-bit quantization
and use LoRA for parameter-efficient fine-tuning.
The rank of the low-rank matrices in LoRA is uni-
formly set to 8, alpha set to 16, and dropout is set
10
to 0.05. The LoRA modules are uniformly applied
to the Q and V parameter matrices of the attention
modules in each layer of the LLM. All the three
models are optimized using the AdamW optimizer.
For the CodeContest dataset, totally 10609 data-
points are used, and for APPS dataset, 8691 data
samples are used to train the model.
B
Prompt Template
  System Prompt
  Please continue to complete the [lang] function
  according to the requirements and function
  declarations. You are not allowed to modify the
  given code and do the completion only.\n
   Retrieved Knowledge
   The syntax graph of a similar code might be:\n
   [composed syntax graph desciption]
   You can refer to the above knowledge to do the
   completion. \n
   Problem
   The problem:\n
   [problem prompt]
Figure 4: Hard meta-graph prompt.
Soft Prompt for Knowledge Augmented Generation
System Prompt
Please use [lang] to write a correct solution to a program-
ming problem. You should give executable completed code
and nothing else.\n
Retrieved Knowledge
We also have the syntax graph embedding of a similar prob-
lem encoded in <GraphEmb> for you to refer to.\n
Problem
The problem:\n
[problem prompt]
Figure 5: Soft prompting.
11


Question:
Based on the intrinsic evaluation metrics, which of the following effects are observed when applying the 'least tokens' inference strategy across tokenizers?
A. It consistently improves cognitive plausibility metrics by aligning segmentations with human lexical decision patterns.
B. It reduces the average number of tokens per word more effectively than likelihood-based or dropout merge strategies.
C. It introduces significant morphological misalignment compared to deterministic merge rules for BPE tokenizers.
D. It underperforms longest prefix inference for SaGe tokenizers in retaining contextualized morphological boundaries during segmentation.

Answer: AB

We present the results on our benchmark for the 40K vocabularies in Table 3. Results for other sizes are presented in Appendix A. A breakdown of individual evaluation subsets is provided in Appendix B.

Inference methods
Within each tokenizer, we find that the default (‘intended’) strategy is often outperformed by others on some measures. We observe a significant difference in morphological alignment when using merge rules-based inference methods. Qualitative analysis showed the findings illustrated in Table 1, where early merge rules such as ‘i-n’, ‘a-m’, or ‘o-n’ cross morphological boundaries. We notice a similar trend for likelihood-based inference, where frequently-used tokens possess very high likelihood values, sometimes exceeding those of the gold-standard segments. We find that the least tokens strategy fares well not only on the token count metric, which is mostly by-design, but also on cognitive measures, suggesting an effect of human preference to minimal word segmentation. Finally, we observe that likelihood-based inference performs poorly in terms of Rényi efficieny, contrary to its stated purpose. Dropout, on the other hand, performs well on this measure, in line with its goal. longest suffix performs poorly across the board, possibly due to the suffixing nature of the English language, which has complementarily been shown to affect character-level sequential modeling (Pinter et al., 2019). Notably, all our key observations are consistent across vocabulary sizes, as shown in Appendix A.

Inter-tokenizer results
Our results align with Bostrom and Durrett (2020)’s finding that BPE is inferior to UnigramLM on morphology alignment. However, we show that some of this gap can be attributed not to the vocabulary but to the inference method. In addition, we find that SaGe is most aligned to morphology by a substantial margin, indicating that its contextualized objective succeeds in retaining meaningful tokens in the vocabulary during ablation. It is important to note that our evaluation is limited to English, a language with relatively low morphological complexity. Previous studies have identified significant tokenization challenges in non-English languages (Mager et al., 2022). Therefore, any definitive conclusions regarding the effectiveness of tokenization methods should ideally encompass a diverse array of languages. BPE and WordPiece, optimized for compression, unsurprisingly perform well above the likelihood-based vocabularies on the information measures. However, we note that this carries over to the cognitive benchmark as well, supporting Beinborn and Pinter (2023)’s findings.

Greed is All You Need: An Evaluation of Tokenizer Inference Methods
Omri Uzanβ
Craig W. Schmidtκ
Chris Tannerκµ
Yuval Pinterβ
β Department of Computer Science
κ Kensho Technologies
Ben-Gurion University of the Negev
µ Massachusetts Institute of Technology
Beer Sheva, Israel
Cambridge, MA, USA
{omriuz@post, uvp@cs}.bgu.ac.il
{craig.schmidt, chris.tanner}@kensho.com
Abstract
While subword tokenizers such as BPE and
WordPiece are typically used to build vocab-
ularies for NLP models, the method of de-
coding text into a sequence of tokens from
these vocabularies is often left unspecified, or
ill-suited to the method in which they were
constructed. We provide a controlled analysis
of seven tokenizer inference methods across
four different algorithms and three vocabulary
sizes, performed on a novel intrinsic evalua-
tion suite we curated for English, combining
measures rooted in morphology, cognition, and
information theory. We show that for the most
commonly used tokenizers, greedy inference
performs surprisingly well; and that SaGe, a
recently-introduced contextually-informed tok-
enizer, outperforms all others on morphological
alignment.
1
Introduction
Modern NLP systems, including large language
models (LLMs), typically involve an initial step
of mapping raw input text into sequences of sub-
word tokens. These tokens are selected from a
large vocabulary of candidates that were produced
from algorithms such as Byte-Pair Encoding (BPE;
Sennrich et al., 2016), WordPiece (Schuster and
Nakajima, 2012), or UnigramLM (Kudo, 2018).
This process, which we refer to as the inference
method of tokenization, is critical as it determines
how all text is represented and subsequently mod-
eled. Each inference method offers distinct map-
pings, and we assert that it is not well-understood
how these methods differ in performance. Fur-
thermore, popular implementation packages such
as Huggingface Tokenizers,1 SentencePiece,2 and
SubwordNMT3 often obfuscate or even restrict the
choice of inference methods, making it unclear if
1https://huggingface.co/docs/tokenizers
2https://pypi.org/project/sentencepiece
3https://github.com/rsennrich/subword-nmt
Tokenizerinference mode
Segmentation
BPEmerges
Ul tr am od ern
BPElongest prefix
Ultra modern
UnigramLMlikelihood
U nprecedented
UnigramLMlongest prefix
Un precedent ed
SaGelongest prefix
Inc once iva ble
SaGelikelihood
In conceiv able
Table 1: Examples of words being segmented differ-
ently by various tokenizers (vocab size 32,000) using
different inference modes on the same vocabulary. Each
tokenizer’s default mode is provided on top.
inference-time decoding is compatible with the al-
gorithm used to learn the tokenizer’s vocabulary.
Moreover, it is yet to be determined whether such
a match is ideal, or even necessary.
In Table 1 we present examples demonstrating
how the prescribed inference methods of BPE, Un-
igramLM, and SaGe (Yehezkel and Pinter, 2023)
do not necessarily provide the best segmentation
for complex English words, even when good seg-
ments are available in the vocabulary. BPE’s out-
of-the-box algorithm merges the cross-morphemic
am sequence at an early stage, preventing the con-
sideration of ultra and modern and condemn-
ing the downstream model to work with a repre-
sentation learned for the first-person present form
of ‘to be’. UnigramLM’s ablative algorithm en-
abled nprecedented (which crosses morpheme
boundaries) to remain in its final vocabulary of to-
kens, while SaGe’s greedy algorithm masks the
boundaries of both the prefix In and the suffix
able. In all cases, an alternative inference method
provides a more morphologically-aligned segmen-
tation over the same vocabulary.
Previous work regarding subword tokenization
mostly concerns developing vocabulary construc-
tion algorithms (Sennrich et al., 2016; Schuster and
Nakajima, 2012; Kudo, 2018; Mielke et al., 2021;
Yehezkel and Pinter, 2023), finding the optimal
arXiv:2403.01289v2  [cs.CL]  31 May 2024
vocabulary size (Gowda and May, 2020; Gutierrez-
Vasques et al., 2021), building multilingual vocab-
ularies (Liang et al., 2023), and using space posi-
tioning in the vocabulary tokens (Gow-Smith et al.,
2022; Jacobs and Pinter, 2022). Others analyze the
effects of vocabularies, finding intricate relations
between algorithm or vocabulary and downstream
performance (Bostrom and Durrett, 2020; Cognetta
et al., 2024a), information theory (Zouhar et al.,
2023; Cognetta et al., 2024b), cognitive plausibil-
ity (Beinborn and Pinter, 2023), impact on soci-
ety (Ovalle et al., 2024), or morphological align-
ment (Klein and Tsarfaty, 2020; Hofmann et al.,
2021, 2022; Gow-Smith et al., 2024; Batsuren et al.,
2024).
Research concerning inference methods has
been more scarce, and includes examination of ran-
dom effects on BPE merges (Provilkov et al., 2020;
Saleva and Lignos, 2023) and application of sophis-
ticated search algorithms (He et al., 2020). As far
as we know, there exists no comprehensive study
comparing inference methods across a variety of
vocabularies and sizes using diverse metrics.
In this work, we conduct a controlled experiment
isolating the effects of inference methods over four
tokenizers, introducing an evaluation suite aggre-
gating intrinsic benchmarks from various theoret-
ical realms.4 We find that greedy inference meth-
ods work surprisingly well for all four vocabular-
ies across morphological and information-theoretic
metrics. Furthermore, we demonstrate that SaGe
yields state-of-the-art performance according to
morphological metrics, and that inference methods
that minimize token count perform strongest by
cognitive metrics.
2
Inference Methods
Let V denote a vocabulary of subword tokens and
w denote a word (or ‘pretoken’), the output of a
pretokenizer. We define s(V, w) := (t1, ..., tk) as a
segmentation of w into k subword tokens such that
∀i, ti ∈V and that the concatenation of t1, ..., tk
results in w. We use the term segmentation to
denote the application of an inference method on a
text given a token vocabulary, as well as its result.
Current widely-employed tokenization sched-
ules couple together the tokenizer vocabulary with
the inference method. However, we advocate for
decoupling them, as they are independent pro-
4We release our code and data at https://github.com/
MeLeLBGU/tokenizers_intrinsic_benchmark.
cesses. Specifically, given a fixed token vocabulary
produced from pre-training data, one could subse-
quently use any applicable inference method for
the task at hand. Thus, in our experiments, we
use various intrinsic metrics to analyze the impact
and performance of the several classes of inference
methods:
Greedy inference methods
only consider and
produce one token at each step. We test three
greedy approaches: Longest prefix, which Word-
Piece uses by default (Wu et al., 2016), selects
the longest token in V that is a prefix of w, and
then continues to iteratively segment the remain-
ing text. Longest suffix selects the longest token
that is a suffix of w and continues iteratively (Ja-
cobs and Pinter, 2022; Bauwens, 2023). Since
this strategy diverges from English Morphology,
we consider it an intriguing baseline for assessing
the impact of linguistic structure on the inference
method. Longest token selects the longest token
that is contained in w, adds it to the generated
segmentation, and then iteratively segments each
remaining character sequence. This was proposed
by Hofmann et al. (2022) to approximate words
by their k longest tokens. They showed that it pre-
serves morphological structure of words and leads
to performance gains on some downstream tasks.
Merge rules-based inference methods
begin
with a word’s character sequence and iteratively
apply token-forming merge rules learnt by the tok-
enizer at the vocabulary creation phase, until none
can be applied. This is BPE’s default inference
mode.5 In our experiments we test two variants for
BPE: The deterministic merge strategy recursively
applies the first applicable BPE merge rule by its
order in the trained merge list. Dropout (Provilkov
et al., 2020) applies each valid merge rule with
probability p, leading to a regularization effect
where rare tokens surface more often and their em-
beddings can be better trained. It has been shown
to improve machine translation performance.
Likelihood-based inference methods
use indi-
vidual likelihood values assigned to tokens in order
to find a segmentation for w where the total likeli-
hood is maximized (Kudo, 2018; He et al., 2020).
Default uses likelihood values learned during vo-
cabulary construction and considers the likelihood
5While ostensibly also compatible with WordPiece, we
found no implementation of the model that provides an ordered
list of its merges.
Resource
Type
Size
Reference
License
LADEC
Morphological
7,804
Gagné et al. (2019)
CC BY-NC 4.0 DEED
MorphoLex
Morphological
12,029
Sánchez-Gutiérrez et al. (2018)
CC BY-NC-SA 4.0 DEED
MorphyNet
Morphological
219,410
Batsuren et al. (2021)
CC BY-SA 3.0 DEED
DagoBert
Morphological
279,443
Hofmann et al. (2020)
Not specified—citation based
UniMorph
Morphological
143,454
Batsuren et al. (2022)
CC BY 4.0 DEED
UnBlend
Morphological
312
Pinter et al. (2020)
GPL-3.0
CompoundPiece
Morphological
22,896
Minixhofer et al. (2023)
Not specified—citation based
Cognitive data
Cognitive
55,867
Beinborn and Pinter (2023)
MIT
tokenization-scorer
Information Theory
—
Zouhar et al. (2023)
Not specified—citation based
Table 2: Size, Reference and License details of the resources in our benchmark.
of a segmentation to be the product of individ-
ual likelihoods (from which UnigramLM gets its
name). Least tokens assigns a constant likelihood
value to all tokens, effectively selecting a segmen-
tation where the number of tokens is minimized.
While not suggested so far as a standalone infer-
ence method, this obecjtive is proposed for both
vocabulary training and inference in the PathPiece
algorithm (Schmidt et al., 2024).
3
Intrinsic Benchmark
Some analyses of tokenizers rely on training lan-
guage models or translation models and evaluating
their performance on downstream tasks. Using
this process to isolate effects of tokenization hy-
perparameters, such as inference method, is both
time- and resource-consuming, as well as unstable
due to the introduction of multiple sources of ran-
domness throughout the LM/TM pre-training and
fine-tuning phases. Few measures have been intro-
duced that are intrinsic to vocabularies and their
direct application to corpora, and fewer still avoid
conflating the measures with the objectives used
in the vocabulary construction process itself. As
a result, the body of work focused on improving
tokenization schemes is still relatively small.
We create and release a benchmark made to in-
trinsically evaluate subword tokenizers. We col-
lected word-level datasets and information mea-
sures which have been shown, or hypothesized, to
correlate with the performance of language mod-
els on various downstream tasks. Details on these
resources are provided in Table 2. At present, the
benchmark is focused on the English language, al-
though corresponding datasets exist for others as
well.
Morphological alignment
It is commonly as-
sumed that, for a given tokenizer, alignment of
word segments to morphological gold-standard seg-
mentations is a predictor of the ability of a language
model that uses the given tokenizer to represent
words, especially ‘complex’ ones that are made up
of several roots or contain multiple morphological
affixes (Schick and Schütze, 2019; Nayak et al.,
2020; Hofmann et al., 2021; Gow-Smith et al.,
2022). We follow Gow-Smith et al. (2022) and eval-
uate our tokenizers’s alignment with morphological
annotations found in LADEC (Gagné et al., 2019),
MorphoLex (Sánchez-Gutiérrez et al., 2018), Mor-
phyNet (Batsuren et al., 2021), and DagoBert (Hof-
mann et al., 2020). We augment these datasets
with morpheme segmentation data (Batsuren et al.,
2022), novel blend structure detection data (Pinter
et al., 2020), and compound separation data (Minix-
hofer et al., 2023). The number of words in each
resource can be found in Table 2. We compare
the segmentations generated by the tokenizers with
each inference method to gold-standard morpho-
logical segmentations using the metric introduced
by Creutz and Linden (2004), and report the macro-
averaged F1 score over the different resources.
Cognitive Plausibility
We use the benchmark
and data from Beinborn and Pinter (2023) to mea-
sure the correlation of a tokenizer’s output with
the response time and accuracy of human partici-
pants in a lexical decision task, predicated on the
hypothesis that a good tokenizer struggles with
character sequences that humans find difficult, and
vice versa. We report the average of the absolute
value correlation scores across the four linguistic
setups (word/nonword × accuracy/response time).
Tokens distribution statistics
We report the
Rényi efficiency of different segmentations across
a corpus (Zouhar et al., 2023). This measure penal-
izes token distributions dominated by either very
high- and/or very low-frequency tokens, and was
shown to correlate strongly with BLEU scores
for machine translation systems trained on the re-
spective tokenizers. Recent work (Cognetta et al.,
Vocab
Inference
Morphological
Cognitive
Rényi
Tokens
Decoding
method
alignment
plausibility
efficiency
per word
diff
BPE
longest prefix
.8584
.3266
.4482
1.4273
.0502
longest suffix
.6467
.3170
.4482
1.4286
.0417
longest token
.8738
.3302
.4474
1.4261
.0484
least tokens
.7544
.3321
.4476
1.4237
.0382
det. merges
.6309
.3355
.4482
1.4308
—
dropout merge
.6081
.2925
.4537
1.5793
.1313
WordPiece
longest prefix
.8488
.3307
.4507
1.4430
—
longest suffix
.6288
.3198
.4502
1.4435
.0656
longest token
.8466
.3332
.4500
1.4411
.0216
least tokens
.7342
.3306
.4401
1.4319
.0682
UnigramLM
longest prefix
.9222
.2858
.3400
1.7577
.1187
longest suffix
.7520
.2690
.2897
1.7624
.0516
longest token
.8845
.2948
.3040
1.7353
.0406
least tokens
.8982
.2953
.2969
1.7219
.0328
likelihood
.9149
.2937
.2919
1.7314
—
SaGe
longest prefix
.9606
.2581
.3217
1.9445
—
longest suffix
.7370
.2471
.2832
1.9615
.1704
longest token
.9236
.2671
.3027
1.9236
.0887
least tokens
.9125
.2674
.2944
1.8895
.1318
likelihood†
.9515
.2664
.2937
1.9156
.1168
Table 3: Intrinsic Benchmark results on a vocab size of 40k. ‘Default’ decoding algorithms (used in vocabulary
construction) in italics. Not all methods are applicable to all tokenizers. Decoding diff presents the share of
pretokens in the MiniPile test set that are differently tokenized using the method, compared with the default. We
present correlation scores for performance over the various metric families in Appendix C.
†For SaGe, likelihood is only based on unigram scores obtained before further vocabulary ablation.
2024b) reveals a misalignment between Rényi ef-
ficiency and downstream performance in certain
cases, reinforcing the necessity of an evaluation
suite grounded in diverse domains and disciplines,
as advocated in this work. We also measure the
average number of tokens per word over a corpus,
as a proxy for compression quality (Gallé, 2019).
We omit the popular measure of character-length
distribution of the tokens in the vocabulary, as it
does not vary with segmentation strategy.
Lastly, we report the proportion of pretokens that
are segmented different from the default across our
reference corpus.
4
Experiments
We evaluate inference methods for the following
tokenizer vocabularies: BPE, UnigramLM, Word-
Piece and SaGe. We use the train split of the MiniP-
ile (Kaddour, 2023) dataset to construct the tok-
enizer vocabularies. We train vocabularies of sizes
32,768, 40,960, and 49,152, using the HuggingFace
Tokenizers library, with identical pre-tokenization,
representing the text at byte level. UnigramLM
and SaGe require an initial vocabulary for their
top-down algorithms; for the former, we used the
default implementation of one million top n-grams,
while SaGe was initialized with a 262K-size Uni-
gramLM vocabulary. This initial vocabulary also
provided us with token likelihood scores for infer-
ence, although a more exact implementation would
also incorporate the contextual SaGe objective.
Token distribution statistics measurements and
decoding diff rates were computed over the test
split of the MiniPile dataset. We measure the Rényi
efficiency using the tokenization-scorer package6
with α = 2.5. For each tokenizer, all experiments
ran within several minutes on a personal laptop
computer, highlighting the usefulness of our bench-
mark as an efficient tool for in-loop hyperparamter
tuning.
We present the results on our benchmark for
the 40K vocabularies in Table 3. Results for other
sizes are presented in Appendix A. A breakdown
of individual evaluation subsets is provided in Ap-
pendix B.
Inference methods
Within each tokenizer, we
find that the default (‘intended’) strategy is often
outperformed by others on some measures. We
observe a significant difference in morphological
alignment when using merge rules-based inference
methods. Qualitative analysis showed the findings
6https://github.com/zouharvi/tokenization-scorer
illustrated in Table 1, where early merge rules such
as ‘i-n’, ‘a-m’, or ‘o-n’ cross morphological bound-
aries. We notice a similar trend for likelihood-
based inference, where frequently-used tokens pos-
sess very high likelihood values, sometimes exceed-
ing those of the gold-standard segments. We find
that the least tokens strategy fares well not only on
the token count metric, which is mostly by-design,
but also on cognitive measures, suggesting an effect
of human preference to minimal word segmenta-
tion. Finally, we observe that likelihood-based in-
ference performs poorly in terms of Rényi efficieny,
contrary to its stated purpose. Dropout, on the other
hand, performs well on this measure, in line with
its goal. longest suffix performs poorly across the
board, possibly due to the suffixing nature of the
English language, which has complementarily been
shown to affect character-level sequential model-
ing (Pinter et al., 2019). Notably, all our key obser-
vations are consistent across vocabulary sizes, as
shown in Appendix A.
Inter-tokenizer results
Our results align with
Bostrom and Durrett (2020)’s finding that BPE
is inferior to UnigramLM on morphology align-
ment. However, we show that some of this gap
can be attributed not to the vocabulary but to the
inference method. In addition, we find that SaGe
is most aligned to morphology by a substantial
margin, indicating that its contextualized objective
succeeds in retaining meaningful tokens in the vo-
cabulary during ablation. It is important to note
that our evaluation is limited to English, a language
with relatively low morphological complexity. Pre-
vious studies have identified significant tokeniza-
tion challenges in non-English languages (Mager
et al., 2022). Therefore, any definitive conclusions
regarding the effectiveness of tokenization meth-
ods should ideally encompass a diverse array of
languages. BPE and WordPiece, optimized for
compression, unsurprisingly perform well above
the likelihood-based vocabularies on the informa-
tion measures. However, we note that this carries
over to the cognitive benchmark as well, supporting
Beinborn and Pinter (2023)’s findings.
Finally, we note that the two likelihood-based
vocabularies follow the exact same within-vocab
trends, and those for the two information-based
vocabularies are also very close. This highlights
the consistency and robustness of our benchmark,
although some results are relatively close to each
other, which can be expected considering that some
inference methods do not change much of the token
sequences (see rightmost column of Table 3).
5
Conclusion
In this work, we curated an aggregated benchmark
for intrinsic evaluation of subword tokenizers and
used it to show the importance of selecting an infer-
ence method suited for a vocabulary given a task.
Given its computational efficiency, we hope the
benchmark can be used in LM training efforts as a
fruitful first step to improve tokenization schemes,
or to select inference methods on-line. Concretely,
our findings suggest that greedy inference is a good
choice, especially for morphologically-motivated
tasks, even for tokenizers trained on other objec-
tives. Considering its ease of implementation and
faster inference, this is an encouraging finding.
In the future, we plan to examine the correlation
between our benchmark and various downstream
tasks, as well as expand our experimentation to
other languages and new algorithms.
Limitations
Our paper contains evaluation of models in the En-
glish language. This was done mostly in order to
focus this short paper’s contribution, and to be able
to control for as many possibly-confounding vari-
ables such as training data. Nevertheless, a more
complete followup would have to include attempts
to replicate our findings on other languages, aiming
for a set as diverse as possible mostly in terms of
typology and script.
Our evaluation is limited to intrinsic measures.
While this makes development of tokenizers easier,
we acknowledge that the body of work correlating
success on these measures with performance of
downstream models on end-tasks is incomplete.
Ethical Considerations
Details for human annotation for the cognitive
benchmark are documented in the source bench-
mark’s paper (Beinborn and Pinter, 2023), from
which we took the data as-is.
Acknowledgments
We would like to thank Charlie Lovering, Varshini
Reddy, and Haoran Zhang for comments on early
drafts of this paper.
We thank the anonymous
reviewers for their comments on our submission.
This research was supported in part by the Israel
Science Foundation (grant No. 1166/23) and by
a Google gift intended for work on Meaningful
Subword Text Tokenization.
References
Khuyagbaatar Batsuren, Gábor Bella, and Fausto
Giunchiglia. 2021. MorphyNet: a large multilingual
database of derivational and inflectional morphology.
In Proceedings of the 18th SIGMORPHON Workshop
on Computational Research in Phonetics, Phonology,
and Morphology, pages 39–48, Online. Association
for Computational Linguistics.
Khuyagbaatar Batsuren, Omer Goldman, Salam Khal-
ifa, Nizar Habash, Witold Kiera´s, Gábor Bella,
Brian Leonard, Garrett Nicolai, Kyle Gorman, Yusti-
nus Ghanggo Ate, Maria Ryskina, Sabrina Mielke,
Elena Budianskaya, Charbel El-Khaissi, Tiago Pi-
mentel, Michael Gasser, William Abbott Lane,
Mohit Raj, Matt Coler, Jaime Rafael Montoya
Samame, Delio Siticonatzi Camaiteri, Esaú Zu-
maeta Rojas, Didier López Francis, Arturo Once-
vay, Juan López Bautista, Gema Celeste Silva Vil-
legas, Lucas Torroba Hennigen, Adam Ek, David
Guriel, Peter Dirix, Jean-Philippe Bernardy, An-
drey Scherbakov, Aziyana Bayyr-ool, Antonios
Anastasopoulos, Roberto Zariquiey, Karina Sheifer,
Sofya Ganieva, Hilaria Cruz, Ritván Karahóˇga,
Stella Markantonatou, George Pavlidis, Matvey Plu-
garyov, Elena Klyachko, Ali Salehi, Candy An-
gulo, Jatayu Baxi, Andrew Krizhanovsky, Natalia
Krizhanovskaya, Elizabeth Salesky, Clara Vania, Sar-
dana Ivanova, Jennifer White, Rowan Hall Maud-
slay, Josef Valvoda, Ran Zmigrod, Paula Czarnowska,
Irene Nikkarinen, Aelita Salchak, Brijesh Bhatt,
Christopher Straughn, Zoey Liu, Jonathan North
Washington, Yuval Pinter, Duygu Ataman, Marcin
Wolinski, Totok Suhardijanto, Anna Yablonskaya,
Niklas Stoehr, Hossep Dolatian, Zahroh Nuriah,
Shyam Ratan, Francis M. Tyers, Edoardo M.
Ponti, Grant Aiton, Aryaman Arora, Richard J.
Hatcher, Ritesh Kumar, Jeremiah Young, Daria
Rodionova, Anastasia Yemelina, Taras Andrushko,
Igor Marchenko, Polina Mashkovtseva, Alexandra
Serova, Emily Prud’hommeaux, Maria Nepomni-
ashchaya, Fausto Giunchiglia, Eleanor Chodroff,
Mans Hulden, Miikka Silfverberg, Arya D. Mc-
Carthy, David Yarowsky, Ryan Cotterell, Reut Tsar-
faty, and Ekaterina Vylomova. 2022. UniMorph 4.0:
Universal Morphology. In Proceedings of the Thir-
teenth Language Resources and Evaluation Confer-
ence, pages 840–855, Marseille, France. European
Language Resources Association.
Khuyagbaatar Batsuren, Ekaterina Vylomova, Verna
Dankers, Tsetsuukhei Delgerbaatar, Omri Uzan, Yu-
val Pinter, and Gábor Bella. 2024. Evaluating sub-
word tokenization: Alien subword composition and
oov generalization challenge.
Thomas Bauwens. 2023. BPE-knockout: Systematic
review of BPE tokenisers and their flaws with appli-
cation in Dutch morphology. Master’s thesis, KU
Leuven.
Lisa Beinborn and Yuval Pinter. 2023. Analyzing cogni-
tive plausibility of subword tokenization. In Proceed-
ings of the 2023 Conference on Empirical Methods
in Natural Language Processing, pages 4478–4486,
Singapore. Association for Computational Linguis-
tics.
Kaj Bostrom and Greg Durrett. 2020. Byte pair encod-
ing is suboptimal for language model pretraining. In
Findings of the Association for Computational Lin-
guistics: EMNLP 2020, pages 4617–4624, Online.
Association for Computational Linguistics.
Marco Cognetta, Tatsuya Hiraoka, Naoaki Okazaki,
Rico Sennrich, and Yuval Pinter. 2024a. An anal-
ysis of bpe vocabulary trimming in neural machine
translation.
Marco Cognetta, Vilém Zouhar, Sangwhan Moon, and
Naoaki Okazaki. 2024b. Two counterexamples to to-
kenization and the noiseless channel. In Proceedings
of the 2024 Joint International Conference on Compu-
tational Linguistics, Language Resources and Evalu-
ation (LREC-COLING 2024), pages 16897–16906,
Torino, Italia. ELRA and ICCL.
Mathias Creutz and Bo Krister Johan Linden. 2004.
Morpheme segmentation gold standards for finnish
and english.
Christina L. Gagné, Thomas L. Spalding, and Daniel
Schmidtke. 2019.
Ladec: The large database of
english compounds. Behavior Research Methods,
51:2152 – 2179.
Matthias Gallé. 2019. Investigating the effectiveness of
BPE: The power of shorter sequences. In Proceed-
ings of the 2019 Conference on Empirical Methods
in Natural Language Processing and the 9th Inter-
national Joint Conference on Natural Language Pro-
cessing (EMNLP-IJCNLP), pages 1375–1381, Hong
Kong, China. Association for Computational Linguis-
tics.
Edward Gow-Smith, Dylan Phelps, Harish Tayyar Mad-
abushi, Carolina Scarton, and Aline Villavicencio.
2024. Word boundary information isn’t useful for
encoder language models.
Edward Gow-Smith, Harish Tayyar Madabushi, Car-
olina Scarton, and Aline Villavicencio. 2022. Improv-
ing tokenisation by alternative treatment of spaces.
In Proceedings of the 2022 Conference on Empiri-
cal Methods in Natural Language Processing, pages
11430–11443, Abu Dhabi, United Arab Emirates. As-
sociation for Computational Linguistics.
Thamme Gowda and Jonathan May. 2020. Finding the
optimal vocabulary size for neural machine transla-
tion. In Findings of the Association for Computa-
tional Linguistics: EMNLP 2020, pages 3955–3964,
Online. Association for Computational Linguistics.
Ximena Gutierrez-Vasques, Christian Bentz, Olga Sozi-
nova, and Tanja Samardzic. 2021. From characters
to words: the turning point of BPE merges. In Pro-
ceedings of the 16th Conference of the European
Chapter of the Association for Computational Lin-
guistics: Main Volume, pages 3454–3468, Online.
Association for Computational Linguistics.
Xuanli He, Gholamreza Haffari, and Mohammad
Norouzi. 2020. Dynamic programming encoding
for subword segmentation in neural machine transla-
tion. In Proceedings of the 58th Annual Meeting of
the Association for Computational Linguistics, pages
3042–3051, Online. Association for Computational
Linguistics.
Valentin Hofmann, Janet Pierrehumbert, and Hinrich
Schütze. 2020. DagoBERT: Generating derivational
morphology with a pretrained language model. In
Proceedings of the 2020 Conference on Empirical
Methods in Natural Language Processing (EMNLP),
pages 3848–3861, Online. Association for Computa-
tional Linguistics.
Valentin Hofmann, Janet Pierrehumbert, and Hinrich
Schütze. 2021. Superbizarre is not superb: Deriva-
tional morphology improves BERT’s interpretation
of complex words. In Proceedings of the 59th Annual
Meeting of the Association for Computational Lin-
guistics and the 11th International Joint Conference
on Natural Language Processing (Volume 1: Long
Papers), pages 3594–3608, Online. Association for
Computational Linguistics.
Valentin Hofmann, Hinrich Schuetze, and Janet Pierre-
humbert. 2022. An embarrassingly simple method
to mitigate undesirable properties of pretrained lan-
guage model tokenizers. In Proceedings of the 60th
Annual Meeting of the Association for Computational
Linguistics (Volume 2: Short Papers), pages 385–393,
Dublin, Ireland. Association for Computational Lin-
guistics.
Cassandra L Jacobs and Yuval Pinter. 2022. Lost in
space marking. arXiv preprint arXiv:2208.01561.
Jean Kaddour. 2023.
The minipile challenge for
data-efficient language models.
arXiv preprint
arXiv:2304.08442.
Stav Klein and Reut Tsarfaty. 2020. Getting the ##life
out of living: How adequate are word-pieces for mod-
elling complex morphology? In Proceedings of the
17th SIGMORPHON Workshop on Computational
Research in Phonetics, Phonology, and Morphology,
pages 204–209, Online. Association for Computa-
tional Linguistics.
Taku Kudo. 2018. Subword regularization: Improv-
ing neural network translation models with multiple
subword candidates. In Proceedings of the 56th An-
nual Meeting of the Association for Computational
Linguistics (Volume 1: Long Papers), pages 66–75,
Melbourne, Australia. Association for Computational
Linguistics.
Davis Liang, Hila Gonen, Yuning Mao, Rui Hou, Na-
man Goyal, Marjan Ghazvininejad, Luke Zettle-
moyer, and Madian Khabsa. 2023. XLM-V: Over-
coming the vocabulary bottleneck in multilingual
masked language models. In Proceedings of the 2023
Conference on Empirical Methods in Natural Lan-
guage Processing, pages 13142–13152, Singapore.
Association for Computational Linguistics.
Manuel Mager, Arturo Oncevay, Elisabeth Mager,
Katharina Kann, and Thang Vu. 2022. BPE vs. mor-
phological segmentation: A case study on machine
translation of four polysynthetic languages. In Find-
ings of the Association for Computational Linguistics:
ACL 2022, pages 961–971, Dublin, Ireland. Associa-
tion for Computational Linguistics.
Sabrina J Mielke, Zaid Alyafeai, Elizabeth Salesky,
Colin Raffel, Manan Dey, Matthias Gallé, Arun Raja,
Chenglei Si, Wilson Y Lee, Benoît Sagot, et al. 2021.
Between words and characters: a brief history of
open-vocabulary modeling and tokenization in nlp.
arXiv preprint arXiv:2112.10508.
Benjamin Minixhofer, Jonas Pfeiffer, and Ivan Vuli´c.
2023. CompoundPiece: Evaluating and improving
decompounding performance of language models.
In Proceedings of the 2023 Conference on Empiri-
cal Methods in Natural Language Processing, pages
343–359, Singapore. Association for Computational
Linguistics.
Anmol Nayak, Hariprasad Timmapathini, Karthikeyan
Ponnalagu, and Vijendran Gopalan Venkoparao.
2020. Domain adaptation challenges of BERT in
tokenization and sub-word representations of out-
of-vocabulary words. In Proceedings of the First
Workshop on Insights from Negative Results in NLP,
pages 1–5, Online. Association for Computational
Linguistics.
Anaelia Ovalle, Ninareh Mehrabi, Palash Goyal, Jwala
Dhamala, Kai-Wei Chang, Richard Zemel, Aram Gal-
styan, Yuval Pinter, and Rahul Gupta. 2024. Tok-
enization matters: Navigating data-scarce tokeniza-
tion for gender inclusive language technologies.
Yuval Pinter, Cassandra L. Jacobs, and Jacob Eisenstein.
2020. Will it unblend? In Findings of the Associa-
tion for Computational Linguistics: EMNLP 2020,
pages 1525–1535, Online. Association for Computa-
tional Linguistics.
Yuval Pinter, Marc Marone, and Jacob Eisenstein. 2019.
Character eyes: Seeing language through character-
level taggers. In Proceedings of the 2019 ACL Work-
shop BlackboxNLP: Analyzing and Interpreting Neu-
ral Networks for NLP, pages 95–102, Florence, Italy.
Association for Computational Linguistics.
Ivan Provilkov, Dmitrii Emelianenko, and Elena Voita.
2020. BPE-dropout: Simple and effective subword
regularization. In Proceedings of the 58th Annual
Meeting of the Association for Computational Lin-
guistics, pages 1882–1892, Online. Association for
Computational Linguistics.
Jonne Saleva and Constantine Lignos. 2023.
What
changes when you randomly choose BPE merge op-
erations? not much. In Proceedings of the Fourth
Workshop on Insights from Negative Results in NLP,
pages 59–66, Dubrovnik, Croatia. Association for
Computational Linguistics.
Claudia H. Sánchez-Gutiérrez, Hugo Mailhot, S. Hélène
Deacon, and Maximiliano A. Wilson. 2018. Mor-
pholex: A derivational morphological database for
70,000 english words. Behavior Research Methods,
50:1568–1580.
Timo Schick and Hinrich Schütze. 2019. Rare words:
A major problem for contextualized embeddings and
how to fix it by attentive mimicking.
Craig W. Schmidt, Varshini Reddy, Haoran Zhang, Alec
Alameddine, Omri Uzan, Yuval Pinter, and Chris Tan-
ner. 2024. Tokenization is more than compression.
Mike Schuster and Kaisuke Nakajima. 2012. Japanese
and korean voice search. In 2012 IEEE international
conference on acoustics, speech and signal process-
ing (ICASSP), pages 5149–5152. IEEE.
Rico Sennrich, Barry Haddow, and Alexandra Birch.
2016. Neural machine translation of rare words with
subword units. In Proceedings of the 54th Annual
Meeting of the Association for Computational Lin-
guistics (Volume 1: Long Papers), pages 1715–1725,
Berlin, Germany. Association for Computational Lin-
guistics.
Yonghui Wu, Mike Schuster, Z. Chen, Quoc V. Le,
Mohammad Norouzi, Wolfgang Macherey, Maxim
Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff
Klingner, Apurva Shah, Melvin Johnson, Xiaobing
Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato,
Taku Kudo, Hideto Kazawa, Keith Stevens, George
Kurian, Nishant Patil, Wei Wang, Cliff Young, Ja-
son R. Smith, Jason Riesa, Alex Rudnick, Oriol
Vinyals, Gregory S. Corrado, Macduff Hughes, and
Jeffrey Dean. 2016. Google’s neural machine trans-
lation system: Bridging the gap between human and
machine translation. ArXiv, abs/1609.08144.
Shaked Yehezkel and Yuval Pinter. 2023. Incorporating
context into subword vocabularies. In Proceedings
of the 17th Conference of the European Chapter of
the Association for Computational Linguistics, pages
623–635, Dubrovnik, Croatia. Association for Com-
putational Linguistics.
Vilém Zouhar, Clara Meister, Juan Gastaldi, Li Du,
Mrinmaya Sachan, and Ryan Cotterell. 2023. To-
kenization and the noiseless channel. In Proceedings
of the 61st Annual Meeting of the Association for
Computational Linguistics (Volume 1: Long Papers),
pages 5184–5207, Toronto, Canada. Association for
Computational Linguistics.
A
Results on Different Vocabulary Sizes
Table 4 presents benchmark results on 32K-sized
and 49K-sized vocabularies.
B
Detailed Results
Table 5 breaks down the results (for 40K) on in-
dividual morphological datasets composing our
benchmark. Table 6 Provides the same for indi-
vidual cognitive measures.
C
Inter-Metric Correlations
Table 7 presents the Pearson correlation coeffi-
cients between the various intrinsic metrics used in
the benchmark. These correlations are calculated
based on the aggregated results across all vocabu-
lary sizes.
Vocab
Inference
Morphological
Cognitive
Rényi
Tokens
Decoding
method
alignment
plausibility
efficiency
per word
diff
BPE-32K
longest prefix
.8727
.3122
.4600
1.4511
.0581
longest suffix
.6496
.3018
.4602
1.4530
.0469
longest token
.8883
.3152
.4592
1.4498
.0558
least tokens
.7607
.3174
.4595
1.4469
.0426
det. merges
.6409
.3201
.4603
1.4551
—
dropout merge
.6149
.2795
.4656
1.6041
.1316
WordPiece-32K
longest prefix
.7819
.3185
.4630
1.4689
—
longest suffix
.5084
.3089
.4626
1.4698
.0744
longest token
.7764
.3212
.4622
1.4667
.0243
least tokens
.7394
.3185
.4508
1.4565
.0769
UnigramLM-32K
longest prefix
.9278
.2855
.3574
1.7803
.1171
longest suffix
.7610
.2679
.2961
1.7838
.0516
longest token
.8926
.2930
.3103
1.7534
.0395
least tokens
.9077
.2937
.3028
1.7418
.0303
likelihood
.9206
.2931
.2985
1.7501
—
SaGe-32K
longest prefix
.9613
.2610
.3454
1.9502
—
longest suffix
.7449
.2473
.2914
1.9736
.1653
longest token
.9348
.2685
.3113
1.9319
.0822
least tokens
.9212
.2691
.3035
1.9084
.1247
likelihood
.9579
.2679
.3026
1.9246
.1098
BPE-49K
longest prefix
.8440
.3371
.4391
1.4104
.0444
longest suffix
.6438
.3279
.4390
1.4112
.0379
longest token
.8637
.3404
.4384
1.4094
.0430
least tokens
.7464
.3421
.4385
1.4072
.0351
det. merges
.6208
.3461
.4390
1.4137
—
dropout merge
.5967
.2996
.4446
1.5610
.1310
WordPiece-49K
longest prefix
.7600
.3398
.4413
1.4245
—
longest suffix
.5133
.3309
.4407
1.4247
.0589
longest token
.7598
.3421
.4406
1.4228
.0194
least tokens
.7261
.3401
.4319
1.4145
.0615
UnigramLM-49K
longest prefix
.9157
.2818
.3467
1.7432
.1190
longest suffix
.7449
.2669
.2849
1.7486
.0516
longest token
.8750
.2915
.2994
1.7245
.0416
least tokens
.8908
.2926
.2924
1.7098
.0345
likelihood
.9095
.2911
.2871
1.7201
—
SaGe-49K
longest prefix
.9606
.2566
.3361
1.9414
—
longest suffix
.7355
.2466
.2783
1.9562
.1735
longest token
.9200
.2662
.2975
1.9192
.0912
least tokens
.9053
.2662
.2893
1.8947
.1353
likelihood
.9455
.2651
.2887
1.9111
.1194
Table 4: Aggregated results on 32K and 49K vocabularies.
Vocab
Inference
Ladec
Morpho-
Morphy-
Dago-
Uni-
UnBlend
Compound-
Lex
Net
Bert
Morph
Piece
BPE
longest prefix
.9210
.8091
.8511
.8013
.9956
.7404
.8904
longest suffix
.9497
.6222
.6524
.7116
.0316
.6095
.9502
longest token
.9147
.8125
.8953
.8618
.9705
.7711
.8905
least tokens
.9775
.7401
.8303
.8539
.2573
.6489
.9731
det. merges
.8160
.6781
.6132
.6195
.3233
.6097
.7568
dropout merge
.7666
.6557
.5871
.5953
.3128
.6213
.7178
WordPiece
longest prefix
.9333
.7625
.9114
.8659
.9963
.5569
.9153
longest suffix
.9447
.6005
.6289
.6844
.1059
.4838
.9535
longest token
.9275
.7568
.9124
.8765
.9666
.5749
.9112
least tokens
.9706
.7132
.8253
.8032
.2670
.5897
.9704
UnigramLM
longest prefix
.9551
.8800
.9291
.9087
.9973
.8553
.9299
longest suffix
.9248
.6387
.8206
.8407
.2777
.8076
.9536
longest token
.8855
.7534
.9313
.9378
.9135
.8571
.9130
least tokens
.9660
.8015
.9511
.9593
.7218
.9073
.9801
likelihood
.9341
.7903
.9645
.9782
.8423
.9205
.9743
SaGe
longest prefix
.9734
.9422
.9673
.9600
.9973
.9213
.9626
longest suffix
.9519
.5996
.7819
.8091
.2403
.8216
.9549
longest token
.9420
.8390
.9365
.9418
.9711
.8889
.9457
least tokens
.9856
.8394
.9533
.9632
.7269
.9318
.9877
likelihood
.9709
.8813
.9809
.9879
.9014
.9492
.9890
Table 5: Results on individual morphological resources.
Vocab
Inference
Words-RT
Words-ACC
nonwords-RT
nonwords-ACC
BPE
longest prefix
−.3136
.4035
.4111
−.1784
longest suffix
−.3102
.3890
.3987
−.1699
longest token
−.3164
.4086
.4130
−.1828
least tokens
−.3146
.4083
.4226
−.1828
det. merges
−.3285
.4138
.4163
−.1835
dropout merge
−.2562
.3505
.3908
−.1726
WordPiece
longest prefix
−.3198
.4029
.4119
−.1882
longest suffix
−.3132
.3863
.4028
−.1770
longest token
−.3226
.4067
.4134
−.1902
least tokens
−.3146
.4036
.4201
−.1842
UnigramLM
longest prefix
−.2292
.3391
.3920
−.1827
longest suffix
−.2308
.3235
.3645
−.1572
longest token
−.2493
.3590
.3904
−.1804
least tokens
−.2394
.3582
.3978
−.1860
likelihood
−.2424
.3577
.3926
−.1822
SaGe
longest prefix
−.1924
.2896
.3752
−.1754
longest suffix
−.1895
.2801
.3602
−.1585
longest token
−.2079
.3047
.3790
−.1767
least tokens
−.1978
.3034
.3864
−.1821
likelihood
−.2035
.3043
.3797
−.1780
Table 6: A breakdown of cognitive correlation results across vocabularies and inference methods.
Morphological
Cognitive
Rényi
Tokens
alignment
plausibility
efficiency
per word
Morphological alignment
1
−.5009
−.4799
.5726
Cognitive plausibility
—
1
.6470
−.9588
Rényi efficiency
—
—
1
−.6400
Tokens per word
—
—
—
1
Table 7: Correlations between the different intrinsic metrics.