Question:
Regarding strong negative samples, which of the following statements are correct:
A. Directly derived from the training dataset.
B. Aimed at addressing the challenges brought by the infinite answer space.
C. Provides core and precise preference contrast signals.
D. The use of strong negative samples works well.

Answer: AC

To introduce our loss, we first write the ideal loss in Eq.(9) in an equivalent form by removing the constant log(M) and moving a minus sign out of the logarithm:
˜L[θ|x] :=
1
N ΣN
i −log
exp[rθ(x, yi)]
exp[rθ(x, yi)] + ΣM
j exp[rθ(x, yi,j
−)]
(10)
Now we propose our loss function in this negative log soft-max form with a specific negative sampling strategy in mind.

4.1. Narrow definition
For the most classical setup, we assume we only have access to pair-wise preference data. In this setting, our loss for each mini-batch of B samples ({(xi, yi
w, yi
l)}B) is defined as follows:
LEPA =
1
B
B
X
i
−log
exp[rθ(xi, yi
w)]
P
j∈{i}∪Iwk
exp[rθ(xi, yj
w)] + exp[rθ(xi, yj
l )]
 (11)
where Iwk is a non-empty random subset of {1, 2, . . . , i − 1, i + 1, . . . , B}, introducing negative samples that are mismatched responses originally sampled for other prompts. Its size |Iwk| = N −
weak/2 ∈(0, B −1] is a hyperparameter. Note that our loss without Iwk reduces to the DPO loss. We justify our choice of positives and negatives in EPA as follows:
1. Why is yw a good approximation of a positive sam- ple from p(y|x)? For a yw in the dataset, it may not be the best y, but there is only a finite number of po- tentially possible better ones according to Assumption B.1. Also, since we know it is preferred over yl and infinitely many other arbitrary token sequences, it is a good approximation of a y that is preferred over all other samples up to a small error.
2. Why use both yl and mismatched responses as neg- atives? As stated at the end of section 3, we want to draw the negatives from a perturbation source that is both informative of the positives and of high variance at the same time. For the informativeness, we consider strong negatives yl because they are semantically close to yw. For the high variance, we consider weak negatives such as mismatched responses. We will show the effectiveness of such choice with ablation experiments in section 5.

4.2. General definition
Note that the number of strong negatives in Eq.(11) is lim- ited to 1 because of the given pair-wise data. This is not ideal for the approximation of IPM’s MLE because the number of negatives should be large enough to reduce the approxi- mation error (Theorem 3.3). Therefore, in order not to limit the power of IPM by the classical pair-wise data setup, we generalize our definition of EPA to circumstances where we can have access to more strong negatives (i.e., each yw is accompanied by {yl1, yl2, . . . } instead of just one yl). This is practically feasible because we can always sample less desirable responses from some LLM.

Hence, we define our loss in a more general form as follows:
Lgeneral
EPA
=
1
B
B
X
i
−log
exp[rθ(xi, yi
w)]
exp[rθ(xi, yiw)] + P
k∈Ist
exp[rθ(xi, yi
lk )] + P
j∈Iwk
exp[rθ(xi, yj
∗)]
(12)
where Ist contains N −
strong indices of available strong neg- atives; Iwk contains N −
weak indices of weak negatives; yj
∗
can be either yj
w or some yj
lk (j ̸= i). The justification for the choice of the positives, strong and weak negatives is the same as that of the narrow definition.

5.2.3. COMBINING STRONG AND WEAK NEGATIVES IS EFFECTIVE
We also run ablation and variants of EPA for different num- bers of strong and weak negatives. As shown in Table 2, we can see that although weak negatives are not competitive on its own, their presence together with strong negatives can greatly improve the alignment performance over alter- natives with only strong negatives. Moreover, we also find that weak negatives can even improve DPO, but DPO is still worse than EPA in that scenario (Table 5 in Appendix C).

Energy-Based Preference Model Offers Better Offline Alignment than the
Bradley-Terry Preference Model
Yuzhong Hong * Hanshan Zhang * Junwei Bao Hongfei Jiang Yang Song
Abstract
Since the debut of DPO, it has been shown that
aligning a target LLM with human preferences via
the KL-constrained RLHF loss is mathematically
equivalent to a special kind of reward modeling
task. Concretely, the task requires: 1) using the
target LLM to parameterize the reward model,
and 2) tuning the reward model so that it has a
1:1 linear relationship with the true reward. How-
ever, we identify a significant issue: the DPO loss
might have multiple minimizers, of which only
one satisfies the required linearity condition. The
problem arises from a well-known issue of the
underlying Bradley-Terry preference model: it
does not always have a unique maximum likeli-
hood estimator (MLE). Consequently, the mini-
mizer of the RLHF loss might be unattainable
because it is merely one among many mini-
mizers of the DPO loss. As a better alterna-
tive, we propose an energy-based model (EBM)
that always has a unique MLE, inherently satis-
fying the linearity requirement. To approximate
the MLE in practice, we propose a contrastive
loss named Energy Preference Alignment (EPA),
wherein each positive sample is contrasted against
one or more strong negatives as well as many
free weak negatives. Theoretical properties of our
EBM enable the approximation error of EPA to
almost surely vanish when a sufficient number of
negatives are used. Empirically, we demonstrate
that EPA consistently delivers better performance
on open benchmarks compared to DPO, thereby
showing the superiority of our EBM.
1. Introduction
Reinforcement Learning with Human Feedback (RLHF)
(Christiano et al., 2017) has been widely used to align a
large language model (LLM) with human preference. The
*Equal contribution
0
0.5
1
1.5
2
·104
−20
−15
−10
−5
0
Log Ratio Reward
DPO
rθ(x,yw)
rθ(x,yl)
rθ(x,y−
weak)
0
0.5
1
1.5
2
·104
−20
−15
−10
−5
0
#steps
Log Ratio Reward
EPA
rtrue(x,y−
weak)
rtrue(x,yl)
rtrue(x,yw)
True Reward
Log Ratio Reward
Figure 1. Samples are off from the slope-1 linearity (yellow lines)
after training with DPO. Given an extremely undesirable y−
weak
(i.e., it has very small rtrue), its rθ has to be as sufficiently small as
rtrue to attain the linearity.
canonical RLHF objective (Ziegler et al., 2019; Stiennon
et al., 2020; Ouyang et al., 2022; Perez et al., 2022) is
defined as follows (given x):
LRLHF =
−
E
πθ(y|x)[rtrue(x, y)] + βKL[πθ(y|x)||πref(y|x)]
(1)
where πθ(y|x) is the target LLM (i.e., the policy) to tune,
πref(y|x) a frozen LLM initialized identically as the target
LLM and rtrue(x, y) a reward to maximize.
The LRLHF as defined above is not differentiable w.r.t θ
(Ziegler et al., 2019; Rafailov et al., 2023), hence not SGD-
friendly. Luckily, it has been shown that the unique mini-
mizer of LRLHF can be analytically expressed (Korbak et al.,
2022a). Then, Rafailov et al. (2023) further reformulate the
analytical minimizer as the unique solution to the following
set of equations:



rθ(x, y)= β log πθ(y|x)
πref(y|x)
rθ(x, y)= rtrue(x, y) + C(x)
(2)
(3)
Eq.(2) defines rθ as the log ratio reward and Eq.(3) states
that there holds a slope-1 linearity between the log ratio
1
arXiv:2412.13862v1  [cs.LG]  18 Dec 2024
Energy-Based Preference Model Offers Better Offline Alignment
(Energy-Based) IPM

RLHF Loss
Energy
Discrepancy
DPO
EPA
Maximum
Likelihood
Estimation
: Not Differentiable
: Intractable
+ Loss Modification Tricks
RLHF Optimization
Vanilla Losses
Preference Models
Bradley-Terry
Maximum
Likelihood
Estimation
(ideal DPO)
(ideal EPA)
(Note: rθ(x, y) = β log πθ(y|x)
πref(y|x) )
p(y ≻any other y′|x)
=
exp [rtrue(x, y)]
∞
P
y′ exp [rtrue(x, y′)]
softmax ◦rθ
−log(·)
p(y ≻any other y′|x)
(Thm.3.2, 3.3)
p(yw ≻yl|x)
=
exp [rtrue(x, yw)]
P
y′∈{yw,yl}
exp [rtrue(x, y′)]
softmax ◦rθ
−log(·)
p(yw, yl|x) & p(yw ≻yl|x)
Figure 2. An illustration of the contributions of the paper. Our core argument is that an Energy-Based model (EBM) is a better alternative
to the Bradley-Terry model (BTM) due to its guaranteed unique existence of maximum likelihood estimator (MLE) (which is identical to
the minimizer of the RLHF loss). The advantage of our EBM comes from its intrinsic consideration of the infinity in the size of the space
of y|x, whereas BTM ignores issues caused by the pair sampling distribution (p(yw, yl|x)) in such infinite space. Hence we name our
EBM the Infinite Preference Model. Although approximating the MLE with our proposed EPA loss introduces inevitable error in practice,
we find that it is still empirically better performing than its counterpart – DPO, with or without loss modification techniques presented in
previous offline alignment literatures.
reward and the true reward. This formulation implies that as
long as we can find a differentiable objective function L(rθ)
to achieve Eq.(3) and rθ is parameterized by Eq.(2), we can
convert the RLHF problem into an offline supervised task.
This is the approach of interest in this paper, a drastically
different one from classical online RL methods such as PPO.
1.1. Background & Motivation
The poster child example of this offline approach is DPO
(Rafailov et al., 2023):
Lideal
DPO(rθ) =
−
E
p(yw,yl|x)
E
p(yw≻yl|x)[log σ(rθ(x, yw) −rθ(x, yl))]
(4)
where yw and yl are two responses and yw ≻yl means yw
is prefered to yl given the prompt x. The ideal1 DPO loss
is essentially the maximum likelihood estimation loss of
the Bradley-Terry model (BTM) who posits a sigmoidal
relationship between p(yw ≻yl|x) and rtrue(x, yw) −
rtrue(x, yl). If the maximum likelihood estimator (MLE)
uniquely exists, Rafailov et al. (2023) show that the MLE
will make the slope-1 linearity hold.
1“ideal” in the sense that the expectations in its loss function are
accurately computed. In practice, they can only be approximated,
which can introduce an additional error.
However, as alluded to by our framing, we argue that it
is false to conclude that the slope-1 linearity (i.e., the
minimizer of the RLHF loss) is guaranteed to be reached
with DPO. The reason is that the unique existence of BTM’s
MLE (i.e., the minimizer of LDPO) is not guaranteed without
some non-trivial constraints on the structure of p(yw, yl|x),
a well-known issue of BTM given an infinite candidate
space (i.e., that of y|x) in the literature on learning to rank
(Ford, 1957; Simons & Yao, 1999; Han et al., 2020; Hen-
drickx et al., 2020; Bong & Rinaldo, 2022; Wu et al., 2022).
It is also related to the theoretical issues around dataset
coverage in the RL literature (Kakade & Langford, 2002;
Munos & Szepesv´ari, 2008; Zhan et al., 2022). Moreover,
Tang et al. (2024) have shown that when offline data are
from πref (a usual practice), any pair-wise loss will cease
to correlate with LRLHF when πθ deviates enough from πref
due to reward maximization. This is the theoretical side of
our motivation.
A preliminary experiment also evidently shows that DPO
does not achieve the slope-1 linearity. Assuming it does,
we should expect rθ(x, yw) > rθ(x, yl) ≫rθ(x, y−
weak)
because the true rewards are certainly so if weak negatives
(y−
weak) are mismatched inputs and outputs as they are al-
most impossible to be preferred. However, as shown in
Figure 1, we find that their log ratio rewards are not sub-
2
Energy-Based Preference Model Offers Better Offline Alignment
stantially lower than the strong negatives (yl) when training
with DPO. This empirical phenomenon motivates us to find
a method that utilizes the free signal offered by y−
weak.
1.2. Contributions
Our proposal brings the theoretical and the empirical sides
together. As shown in Figure 2, we argue that an Energy-
Based model (EBM) called the Infinite Preference Model
(IPM) is superior to BTM in preference modeling for offline
alignment based on the following contributions:
• theoretically showing IPM has guaranteed unique ex-
istence of its MLE, equivalent to the minimizer of the
RLHF loss;
• the proposal of EPA, an offline contrastive loss to esti-
mate the MLE of IPM by explicitly using weak nega-
tives in addition to strong negatives;
• a new state of the art of offline alignment on open
benchmarks when given the same settings of training
data and usage of tricks to tweak the losses.
2. Related work
2.1. DPO and its recent improvements
The first approach to avoid DPO’s theoretical issue is to
use non-BTMs to model data distributions. Rafailov et al.
(2023) suggest the DPO’s counterpart for the Plackett-Luce
Model (we refer to it as DPO-PL), which is a generalized
version of BTM for K-wise comparison. IPO (Azar et al.,
2023) uses a different pair-wise preference model than BTM.
The loss derived from that model can be interpreted as: the
difference of log ratio rewards of the yw and yl regresses to
a constant. However, Tang et al. (2024) show that IPO is still
incapable of optimizing LRLHF, similar to DPO. Ethayarajh
et al. (2024) (KTO) point out some limitations of modeling
human preference with a pair-wise model. Instead, they
independently model a data distribution for desirable sam-
ples and another one for undesirable samples. However,
such data distributions do not reflect how most benchmark
datasets are sampled. This could be the reason why some
empirically driven studies find that KTO underperforms
DPO on these benchmarks (Meng et al., 2024; Zhou et al.,
2024).
The second approach is to tweak the DPO loss. Some loss-
tweaking tricks can be effective on their own. For example,
cDPO (Mitchell, 2023) uses label smoothing to alleviate
DPO’s overfitting problem. Park et al. (2024) (R-DPO) in-
troduce a length penalty on the log ratio reward to make
DPO less prone to the verbosity bias. Amini et al. (2024)
(ODPO) add a dynamic margin between yw and yl based on
the intuition that some pairs have stronger or weaker desir-
ability gaps than others. The most effective one discovered
so far is on-policy weighting (WPO) (Zhou et al., 2024). Its
idea is to approximate the on-policy training scenario by
assigning larger weights to the loss of samples closer the
current policy at each step and smaller weights to that of
less closer ones. Other tricks come in combinations. For ex-
ample, CPO (Xu et al., 2024) removes the reference model
in the log ratio reward and add an SFT loss at the same time.
ORPO (Hong et al., 2024) is an improvement over CPO by
adding yet another set of tricks: normalizing the policy to
the token level (length normalization) and then contrasting
the policy distribution with one minus itself. To separate
the wheat from the chaff, Meng et al. (2024) find the most
simple and effective recipe: removing the reference model,
adding a margin and applying length normalization, which
gives rise to SimPO.
There is also a hybrid approach of the above two: non-BTMs
+ tricks. For example, PRO (Song et al., 2024) is on top of
DPO-PL, and BCO (Jung et al., 2024) on top of KTO.
The problem with applying tricks is that there is usually a
lack of theoretical justification on how they are related to
the minimizer of the RLHF loss.
2.2. Fitting discrete EBMs
To provide a theoretical background for our proposal, we
give a concise review of the most related work on fitting
discrete EBMs.
Energy-based models (EBM) (LeCun et al., 2006) are gen-
erative models that posit a Boltzmann distribution of data,
i.e., p(x) ∝exp (−E(x)) where E(x) (called the energy
function) is a real-valued function to learn. An EBM is
called discrete when the data point x is defined on a discrete
space. To fit p(x) with maximum likelihood estimation re-
quires the computation of the normalizer Σ∞
x′ exp (−E(x
′))
(called the partition function), which is intractable. There-
fore, EBMs are usually learned with a tractable approxima-
tion.
The classical approach is to approximate the gradient of
maximum likelihood estimation by online sampling from
parameterized pθ(x) with MCMC (Song & Kingma, 2021).
Although they are ideally effective, it is usually difficult
or expensive to do such sampling, which harms practical
results. Therefore, there are also many MCMC-free meth-
ods (Meng et al., 2022; Hyv¨arinen, 2007; Dai et al., 2020;
Lazaro-Gredilla et al., 2021; Eikema et al., 2022). Recently,
Schr¨oder et al. (2023) have introduced the notion of energy
discrepancy, whose unique global minimizer is identical to
the MLE of the EMB in question. Hence, to find the MLE,
one can simply minimize the energy discrepancy, which is
feasible with SGD on offline data. For its simplicity, we
derive EPA based on their theoretical framework.
3
Energy-Based Preference Model Offers Better Offline Alignment
2.3. EBMs for RLHF
EBMs are not rare in the RLHF research. One of the re-
search directions is to formulate RLHF as a Distribution
Matching problem: minimization of the KL divergence be-
tween a target EBM that reflects human preference and
the policy. The typical example for this approach is Distri-
butional Policy Gradients (DPG) (Parshakova et al., 2019;
Khalifa et al., 2021). However, we would like to point out
that our EBM is different and used for a different purpose.
The EBM in DPG is a non-parametric one predefined as
the learning signal. Our EBM is a parametric one to fit the
distribution of data. The only connection between the two
EBMs is that they are used to find the same optimal policy
(Korbak et al., 2022a).
Deng et al. (2020) uses an EBM for language modeling.
Their work essentially solves the self-play-like RLHF prob-
lem (Chen et al., 2024b). They use the algorithm of Noise
Contrastive Estimation (NCE) of fit their EBM. Although
their EBM is also parametric, it fits the optimal policy dis-
tribution. Our EBM instead fits the preference distribution.
Chen et al. (2024a) proposes two methods – infoNCA and
NCA, based on the same EBM as that of Deng et al. (2020).
The NCA loss follows the same derivation of the loss pro-
posed by Deng et al. (2020) except that they parameterize
their energy function differently. The infoNCA loss exhibits
similarity to our loss. However, we will show that infoNCA
is just a worse-performing ablation version of EPA.
3. IPM: Our EBM for Preference Modelling
In the first subsection, we show that an energy-based model
(EBM) is guaranteed to have a unique MLE which is equiv-
alent to the minimizer of the RLHF objective. In the second
subsection, based on a framework by Schr¨oder et al. (2023),
we describe a general strategy to approximate the MLE us-
ing offline data. Using it will provide the theoretical account
for our proposal in section 4.
3.1. Theoretical guarantee
Given any x, it is obvious that the space of y|x is infinitely
large because y can be any token sequence of unlimited
length no matter how likely or unlikely it is a response to
x. This infinity is problematic for BTM. For example, if
there is a single y that is never sampled, it is easy to refute
the unique existence of BTM’s MLE (see Proposition B.5).
However, EBM can naturally take the infinity into account
to avoid the issue. Specifically, we model a one-to-infinite
preference (v.s. BTM and the more general Plakett-Luce
model only model a one-to-finite-number preference) as
follows:
p(y|x) := p(∀y′ ̸= y : y ≻y′|x)
(5)
Namely, p(y|x)2 is the probability that candidate y is pre-
ferred over all other candidates. Under mild assumptions
(Assumptions B.1 and B.2) that make an EBM applicable,
we define the Infinite Preference Model (IPM) to be the one
that posits that p(y|x) is a Boltzmann distribution induced
by the corresponding true reward (i.e., using −rtrue(x, y) as
the energy function):
p(y|x) =
exp[rtrue(x, y)]
Σ∞
y′ exp[rtrue(x, y
′)]
(6)
IPM is a better alternative to BTM because of the following
theorem (see Appendix B for proof).
Theorem 3.1. when we parameterize the IPM as follows,
the unique existence of the IPM’s MLE is guaranteed and
it will be reached if and only if the slope-1 linearity (i.e.,
Eq.(3)) holds between the log ratio reward and the true
reward.
qθ(y|x) =
exp[rθ(x, y)]
Σ∞
y′ exp[rθ(x, y
′)]
(7)
where rθ is defined as in Eq.(2).
Therefore, as long as we can find the MLE of the IPM pa-
rameterized as so, we are guaranteed to reach the minimizer
of LRLHF since it is the unique solution to Eq.(2) and Eq.(3).
On a side note, the IPM has been previously introduced by
other studies on RLHF for a different purpose: to theoreti-
cally equate the maximization of −LRLHF to the variational
inference of the optimal policy with πref as the prior (Korbak
et al., 2022b; Yang et al., 2024). However, to the best of our
knowledge, we are the first one to introduce IPM not just as
a theoretical toy, but as a tool (when parameterized by the
log ratio reward) to do proper RL-free RLHF.
3.2. Offline approximation of MLE
Despite the powerfulness of IPM, finding its MLE is a non-
trivial task. Directly finding it with the minimization of the
negative log likelihood −log qθ(y|x) is intractable because
of the infinity in the denominator.
There are good tractable approximations but usually with
complex online training algorithms. For simplicity and
scalability purposes, we choose to follow Schr¨oder et al.
(2023), who provide a general strategy that finds the optimal
EBM by simple SGD with offline training data. The strategy
is based on two theorems formally adapted for our purpose
as follows.
Theorem 3.2. For any random variable Z with the condi-
tional variance Var[Y |Z] being positive, the global unique
2One should not confuse p(y|x) with π(y|x) although both of
them are distributions over y given x. p(y|x) is how likely humans
would rate a y as the best whereas π(y|x) measures how likely y
is to be generated.
4
Energy-Based Preference Model Offers Better Offline Alignment
minimizer r∗(x, y) of the functional Energy Discrepancy
(ED) defined as follows is the optimal IPM (i.e., p(y|x) ∝
exp [r∗(x, y)]).
EDx,p(y|x),p(z|y)[r] :=
E
p(y|x)
E
p(z|y)[log Σy′p(z|y′) exp [r(x, y′) −r(x, y)]]
(8)
Theorem 3.3. For any random variable Z whose back-
ward and forward transition probabilities from Y solve the
equation Σyp(z|y)f(y) = Σyp(y|z)f(y) for an arbitrary
f, the estimation error of the following statistic estimate of
EDx,p(y|x),p(z|y)[r] vanishes almost surely when N −→∞
and M −→∞.
L[θ|x] :=
1
N ΣN
i log(1 + ΣM
j exp [rθ(x, yi,j
−) −rθ(x, yi)])
−log(M)
(9)
where {yi}N are samples from p(y|x), {yi,j
−}M from
p(y|z0), and z0 a single sample from p(z|yi).
A one-sentence interpretation of the above theorems is: if
we have a particular kind of negative sampling strategy by
perturbing observed preferred samples, we will learn the
optimal IPM by minimizing a contrastive loss between the
negatives and observed positives. Therefore, when the IPM
is parameterized by the log ratio reward, we will find the
exact minimizer of LRLHF with this loss function.
Note that the property of the negative sampling source Z
as described in Theorem 3.3 is just a sufficient condition as
opposed to a necessary one. This leaves room for empirical
discovery of better negative sampling strategies. A rule of
thumb as suggested by Schr¨oder et al. (2023) is that Z has to
be informative of Y and of high conditional variance at
the same time. This provides the intuition of our proposal
in section 4.
4. EPA: A Practical Approximation
To introduce our loss, we first write the ideal loss in Eq.(9)
in an equivalent form by removing the constant log(M) and
moving a minus sign out of the logarithm:
˜L[θ|x] :=
1
N ΣN
i −log
exp[rθ(x, yi)]
exp[rθ(x, yi)] + ΣM
j exp[rθ(x, yi,j
−)]
(10)
Now we propose our loss function in this negative log soft-
max form with a specific negative sampling strategy in mind.
4.1. Narrow definition
For the most classical setup, we assume we only have access
to pair-wise preference data. In this setting, our loss for
each mini-batch of B samples ({(xi, yi
w, yi
l)}B) is defined
as follows:
LEPA =
1
B
B
X
i
−log
exp[rθ(xi, yi
w)]
P
j∈{i}∪Iwk
exp[rθ(xi, yj
w)] + exp[rθ(xi, yj
l )]
 (11)
where Iwk is a non-empty random subset of {1, 2, . . . , i −
1, i + 1, . . . , B}, introducing negative samples that are
mismatched responses originally sampled for other prompts.
Its size |Iwk| = N −
weak/2 ∈(0, B −1] is a hyperparameter.
Note that our loss without Iwk reduces to the DPO loss.
We justify our choice of positives and negatives in EPA as
follows:
1. Why is yw a good approximation of a positive sam-
ple from p(y|x)? For a yw in the dataset, it may not
be the best y, but there is only a finite number of po-
tentially possible better ones according to Assumption
B.1. Also, since we know it is preferred over yl and
infinitely many other arbitrary token sequences, it is
a good approximation of a y that is preferred over all
other samples up to a small error.
2. Why use both yl and mismatched responses as neg-
atives? As stated at the end of section 3, we want to
draw the negatives from a perturbation source that is
both informative of the positives and of high variance
at the same time. For the informativeness, we consider
strong negatives yl because they are semantically close
to yw. For the high variance, we consider weak nega-
tives such as mismatched responses. We will show the
effectiveness of such choice with ablation experiments
in section 5.
4.2. General definition
Note that the number of strong negatives in Eq.(11) is lim-
ited to 1 because of the given pair-wise data. This is not ideal
for the approximation of IPM’s MLE because the number
of negatives should be large enough to reduce the approxi-
mation error (Theorem 3.3). Therefore, in order not to limit
the power of IPM by the classical pair-wise data setup, we
generalize our definition of EPA to circumstances where
we can have access to more strong negatives (i.e., each yw
is accompanied by {yl1, yl2, . . . } instead of just one yl).
This is practically feasible because we can always sample
less desirable responses from some LLM.
5
Energy-Based Preference Model Offers Better Offline Alignment
Hence, we define our loss in a more general form as follows:
Lgeneral
EPA
=
1
B
B
X
i
−log
exp[rθ(xi, yi
w)]
exp[rθ(xi, yiw)] + P
k∈Ist
exp[rθ(xi, yi
lk )] + P
j∈Iwk
exp[rθ(xi, yj
∗)]
(12)
where Ist contains N −
strong indices of available strong neg-
atives; Iwk contains N −
weak indices of weak negatives; yj
∗
can be either yj
w or some yj
lk (j ̸= i). The justification for
the choice of the positives, strong and weak negatives is the
same as that of the narrow definition.
4.3. Gradient analysis
Using the chain rule consecutively on the negative log soft-
max and the log ratio reward, one can easily find the gradient
of the EPA loss (the general one) as follows:
∇θLgeneral
EPA
=
−β
B
B
X
i
 X
k∈Ist
si
lk
∇θ log πθ(yi
w|xi) −∇θ log πθ(yi
lk|xi)

|
{z
}
strong contrast
+
X
j∈Iwk
sj∇θ log πθ(yi
w|xi) −∇θ log πθ(yj
∗|xi)

|
{z
}
weak contrast

(13)
where si
lk and sj are the softmax-ed values of the strong
negative log ratio rewards and the weak negative log ratio
rewards, respectively. They control the magnitude of the
strong and weak contrast. When there is no weak contrast,
the gradient reduces to that of the DPO loss if there is only
one strong negative. Therefore, one can interpret the weak
contrast as a regularization term added to DPO to prevent
θ from moving to a direction that undesirably increases the
likelihood of weak negatives.
5. Experiments
5.1. Experimental setup
5.1.1. TRAINING DATA
We consider the dataset of Ultrafeedback (Cui et al., 2024)
(denoted as ‘UF-all’) and a widely used pair-wise version
of it (Tunstall et al., 2023) (UF-binarized). The two datasets
are ideal for our purpose besides their popularity. Firstly, in
UF-all, there are 4 responses sampled from multiple sources
for each prompt. This will allow training with our general
version of EPA which can utilize multiple strong negatives.
Secondly, in both UF-all and UF-binarized, the positive
sample yw for each x is the best one out of the 4 responses.
This is an arguably close approximation to our assumption
that positives are sampled from p(∀y′ ̸= y : y ≻y′|x).
5.1.2. EVALUATION
Although Ultrafeedback is intended to reflect human pref-
erence, it is labeled by GPT-4 in reality. Consequently, we
consider MT-Bench (Zheng et al., 2024) which also uses
GPT-4 to score a response on a scale of 1-10. The metric is
the average score for 80 single-turn conversations and 80
multi-turn conversations. We also consider Alpaca-Eval 2.0
(Dubois et al., 2024) because of its high correlation with
human preference, the ultimate concern for RLHF. Its met-
rics are win-rates (with or without length control) against
GPT-4-turbo across 805 test samples with the judge being
GPT-4-turbo itself. We report them in the format of “length
controlled win-rate / win-rate” in the experiment results.
For evaluation on Alpaca Eval 2.0, we use the default de-
coding parameters in the Huggingface implementation. For
MT-Bench, we use the ones specifically required by the
benchmark.
5.1.3. BASELINES & LOSS MODIFICATION TRICKS
Training Data Method
AE 2.0 (%) MT-Bench
SFT
8.16 / 5.47
6.44
UF-binarized
+DPO
17.43 / 15.24
7.55
+IPO
12.97 / 10.13
7.31
+KTO
12.62 / 11.29
7.21
+NCA
14.64 / 11.27
7.39
+EPA
19.20 / 19.26
7.71
UF-all
+DPO-PL 15.95 / 14.68
7.57
+NCA
15.08 / 11.85
7.28
+infoNCA 17.30 / 16.25
7.50
+EPA
22.03 / 21.44
7.58
Table 1. EPA beats all other baselines either for pair-wise data or
for data with > 2 responses for each prompt.
For fair comparison, we only consider methods from the ap-
proach that explicitly aims to minimize LRLHF with specific
probabilistic models about data distributions. Therefore,
we consider DPO, IPO, KTO and NCA for the classical
pair-wise data setting. We consider DPO-PL, NCA, and
infoNCA for the general setting where there are multiple
responses for each prompt in the dataset.
Loss modification tricks are not considered as baselines
because they are orthogonal to our proposal. Comparing
BTM+tricks to EBM would be comparing apples to oranges.
Instead, we consider applying the tricks to both EPA (the
narrow one for fair comparison) and DPO to further verify
our core argument about EBM’s superiority over BTM. The
6
Energy-Based Preference Model Offers Better Offline Alignment
low
KL
high
KL
0
20
40
60
7.25
7.5
7.75
8
8.25
Avg. KL[πθ||πref]
Avg. Reward
(a)
EPA
DPO
0
1
2
3
6.75
7
7.25
7.5
7.75
#epochs
MT-Bench Score
(b)
EPA
DPO
Figure 3. DPO vs. EPA (1:1:2) from the perspective of (a) KL-Reward frontier and (b) training dynamics.
tricks in consideration are those used in SimPO, R-DPO,
CPO and WPO (Details in Table 4 in the Appendix C).
5.1.4. IMPLEMENTATION
We use mistral-7b-sft-beta3 as the reference model and for
the initialization of policy in our paper. We train all models
in this paper for 3 epochs with LoRA (r = 16, α = 16,
dropout = 0.05). Whenever comparing among different
methods, we pick the one out of the three checkpoints with
the best MT-Bench score for each method. For fair compar-
ison of baseline models, we fix β to 0.01. It is more of a
control variable than a hyperparameter because it is a given
component of the RLHF objective which all baselines are
aimed to optimize. We only vary β for them when probing
their KL-Reward frontiers. For comparison of loss modifi-
cation tricks, since the RLHF objective is not necessarily
the purpose, we use the best β and other hyperparameters
specific to each method as reported in previous work (e.g.,
the tricks used in SimPO are only competitive when β = 2.0
for the Mistral model). Learning rate is grid-searched for
each method among {1e −5, 5e −6, 1e −6}.
5.2. Results and analysis
5.2.1. EPA PERFORMS BETTER THAN BASELINES
As shown in Table 1, we can see EPA consistently achieves
the highest scores and hence a new state of the art. Note
that other baselines generally perform even less well than
DPO. This makes BTM the strongest baseline for EBM.
5.2.2. EPA >= DPO FOR THE OPTIMIZATION OF LRLHF
To compare our EBM with its most competitive baseline
BTM in detail, we come back to the starting point – op-
timizing LRLHF. We study from two perspectives of the
optimization problem. Both perspectives involve multiple
checkpoints beyond the single best one for each method
3huggingface.co/HuggingFaceH4/mistral-7b-sft-beta
(e.g., Table 1), offering a more comprehensive comparison.
First, we study how well each method balances the KL
term and the reward term in LRLHF with varying β ∈
{0.01, 0.02, 0.03, 0.04, 0.05, 0.1, 0.5}. Both terms are com-
puted on the 80 single-turn prompts in the MT-Bench dataset.
We estimate KL with 20 response samples per prompt from
each policy distribution. We use the GPT-4 score produced
by MT-Bench as an alias for the true reward. As shown in
Figure 3.(a), in the high-KL region, EPA generally achieves
higher reward than DPO. The two become indistinguishable
only in the low-KL region.
Second, to understand how EPA differs from DPO in terms
of the dynamics during the optimization process, we test the
MT-Bench score of the checkpoint at every 20% of an epoch.
As shown in Figure 3.(b), EPA is less prone to overfitting
and fits to the reward signal more steadily than DPO. The
performance of DPO starts to degenerate rapidly after the
first epoch. However, EPA reaches its peak performance
at the end of the second epoch and overfits more slowly
afterward. This is consistent with our gradient analysis in
Section 4 that EPA is more regularized than DPO.
5.2.3. COMBINING STRONG AND WEAK NEGATIVES IS
EFFECTIVE
We also run ablation and variants of EPA for different num-
bers of strong and weak negatives. As shown in Table 2, we
can see that although weak negatives are not competitive
on its own, their presence together with strong negatives
can greatly improve the alignment performance over alter-
natives with only strong negatives. Moreover, we also find
that weak negatives can even improve DPO, but DPO is still
worse than EPA in that scenario (Table 5 in Appendix C).
5.2.4. EBM+TRICKS >= BTM+TRICKS
Since most loss modification tricks presented in the re-
cent offline alignment literature are originally intended for
BTM/DPO and do not necessarily make sense to EBM/EPA,
7
Energy-Based Preference Model Offers Better Offline Alignment
Method
N+:N−
strong:N−
weak
AE 2.0 (%) MT-Bench
Ablation
1:1:0 (DPO)
17.43 / 15.24
7.55
1:0:2
9.37 / 6.74
6.57
EPA
1:1:1
21.14 / 20.55
7.29
1:1:2
19.20 / 19.26
7.71
1:1:6
16.63 / 15.78
7.57
Ablation
1:3:0 (infoNCA) 17.30 / 16.25
7.50
EPA
1:3:2
22.03 / 21.44
7.58
1:3:4
21.31 / 20.13
7.58
1:3:6
24.01 / 23.44
7.35
1:3:8
24.54 / 23.75
7.19
1:3:10
23.58 / 22.78
7.43
Table 2. Ablation and variants of EPA with varying number of
strong negatives (N −
strong) and varying number of weak negatives
(N −
weak) in addition to the 1 positive (N + = 1) in the denominator
of Lgeneral
EPA
.
Pref
Loss
AE 2.0 (%)
MT-
Model
Modification
Bench
BTM
N/A (DPO)
17.43 / 15.24 7.55
−ref + Lsft (CPO)
13.63 / 10.34 7.11
+lenp (R-DPO)
19.10 / 16.71 7.70
−ref + lenn + mc (SimPO) 20.57 / 20.19 7.61
+wop (WPO)
21.90 / 21.04 7.56
+mc
22.33 / 20.61 7.67
EBM
N/A (EPA)
19.20 / 19.26 7.71
+wop
22.80 / 22.26 7.61
+mc
23.00 / 22.47 7.68
Table 3. BTM vs. EBM with empirically effective loss modifica-
tion tricks.
0
2
4
6
8
10
12
14
16
18
20
15
17
19
21
23
exp [mc]
Alpaca Eval 2.0 (%)
N−
weak = 4
N−
weak = 2
N−
weak = 0
Figure 4. Performance of modified DPO (N −
weak = 0) and modi-
fied EPA (N −
weak > 0) with a margin mc added to rθ(yl|x). Solid
lines represent the length-controlled win-rates, and dotted lines
represent the raw win-rates.
we only consider two of them when applying to EPA. The
first one is a constant margin mc added to the logit of yl.
The trick can be viewed as a loose numerical approxima-
tion to the general EPA where there are multiple ylk. For
example, if mc = 1.4, we have exp [rθ(x, yl) + mc] =
exp [mc] × exp [rθ(x, yl)] ≈4 × exp [rθ(x, yl)]. The sec-
ond one is the on-policy weight proposed by Zhou et al.
(2024). It can be viewed as a curriculum learning technique
which prioritizes samples that closely relate to the current
policy distribution at each step.
As shown in Table 3, we find that both +wop and +mc
produce similar performance boost on EPA to DPO. Al-
though the marginal boost on EPA is generally smaller than
DPO, EPA with tricks is still better than DPO with tricks.
However, the fact these tricks can still work on EPA also
implies that there is still room for improvement. This may
come from EPA not necessarily being the best algorithm to
approximate our EBM’s MLE.
We also study in detail how the value of mc influences
DPO and EPA. As shown in Figure 4, we observe that
a combination of higher mc and higher N −
weak tends to
produce higher performance. A possible explanation for
this is that as mc scales up the logit of the strong negative
yl to loosely approximate the existence of multiple strong
negatives, we get closer to the performance of the general
EPA.
6. Conclusion
In this paper, we show both BTM and our EBM have the
property that their MLE, if uniquely exists, is equivalent to
the minimizer of the RLHF loss. However, the unique exis-
tence of EBM’s MLE is guaranteed whereas that of BTM’s
MLE is not. This theoretical advantage implies that as long
as the EBM’s MLE is accurately found, we are bound to
minimize the RLHF loss. But, the same claim does not
hold for BTM. Although EPA is just an empirical attempt to
approximate our EBM’s MLE, it is already sufficient to out-
perform its counterpart – DPO on open benchmarks, with or
without loss modification tricks presented in previous work.
However, EPA is far from perfect. For example, relatively
poorer computation and memory efficiency is a major handi-
cap of EPA. Foreseeable future work includes finding better
ways to perturb data or adopting more efficient methods to
approximate the MLE. Loss modification tricks particularly
tailored for EPA also remain to be explored.
References
Amini, A., Vieira, T., and Cotterell, R.
Direct pref-
erence optimization with an offset.
arXiv preprint
arXiv:2402.10571, 2024.
8
Energy-Based Preference Model Offers Better Offline Alignment
Azar, M. G., Rowland, M., Piot, B., Guo, D., Calandriello,
D., Valko, M., and Munos, R. A general theoretical
paradigm to understand learning from human preferences.
arXiv preprint arXiv:2310.12036, 2023.
Bong, H. and Rinaldo, A.
Generalized results for the
existence and consistency of the MLE in the bradley-
terry-luce model. In Chaudhuri, K., Jegelka, S., Song,
L., Szepesvari, C., Niu, G., and Sabato, S. (eds.), Pro-
ceedings of the 39th International Conference on Ma-
chine Learning, volume 162 of Proceedings of Machine
Learning Research, pp. 2160–2177. PMLR, 17–23 Jul
2022. URL https://proceedings.mlr.press/
v162/bong22a.html.
Chen, H., He, G., Su, H., and Zhu, J. Noise contrastive align-
ment of language models with explicit rewards. arXiv
preprint arXiv:2402.05369, 2024a.
Chen, Z., Deng, Y., Yuan, H., Ji, K., and Gu, Q. Self-play
fine-tuning converts weak language models to strong lan-
guage models. arXiv preprint arXiv:2401.01335, 2024b.
Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg,
S., and Amodei, D. Deep reinforcement learning from
human preferences. Advances in neural information pro-
cessing systems, 30, 2017.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H.,
Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano,
R., Hesse, C., and Schulman, J. Training verifiers to solve
math word problems. CoRR, abs/2110.14168, 2021. URL
https://arxiv.org/abs/2110.14168.
Cui, G., Yuan, L., Ding, N., Yao, G., He, B., Zhu, W., Ni, Y.,
Xie, G., Xie, R., Lin, Y., et al. Ultrafeedback: Boosting
language models with scaled ai feedback. In Forty-first
International Conference on Machine Learning, 2024.
Dai, H., Singh, R., Dai, B., Sutton, C., and Schuurmans,
D. Learning discrete energy-based models via auxiliary-
variable local exploration. Advances in Neural Informa-
tion Processing Systems, 33:10443–10455, 2020.
Deng, Y., Bakhtin, A., Ott, M., Szlam, A., and Ranzato,
M. Residual energy-based models for text generation. In
International Conference on Learning Representations,
2020. URL https://openreview.net/forum?
id=B1l4SgHKDH.
Dubois, Y., Galambosi, B., Liang, P., and Hashimoto, T. B.
Length-controlled alpacaeval: A simple way to debias
automatic evaluators. arXiv preprint arXiv:2404.04475,
2024.
Eikema, B., Kruszewski, G., Dance, C. R., Elsahar, H.,
and Dymetman, M. An approximate sampler for energy-
based models with divergence diagnostics. Transactions
on Machine Learning Research, 2022.
Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., and
Kiela, D. Kto: Model alignment as prospect theoretic
optimization, 2024.
Ford, L. R. Solution of a ranking problem from binary
comparisons. The American Mathematical Monthly, 64
(8):28–33, 1957.
ISSN 00029890, 19300972.
URL
http://www.jstor.org/stable/2308513.
Han, R., Ye, R., Tan, C., and Chen, K. Asymptotic theory
of sparse Bradley–Terry model. The Annals of Applied
Probability, 30(5):2491 – 2515, 2020. doi: 10.1214/
20-AAP1564. URL https://doi.org/10.1214/
20-AAP1564.
Hendrickx, J., Olshevsky, A., and Saligrama, V.
Mini-
max rate for learning from pairwise comparisons in the
BTL model. In III, H. D. and Singh, A. (eds.), Pro-
ceedings of the 37th International Conference on Ma-
chine Learning, volume 119 of Proceedings of Machine
Learning Research, pp. 4193–4202. PMLR, 13–18 Jul
2020. URL https://proceedings.mlr.press/
v119/hendrickx20a.html.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M.,
Song, D., and Steinhardt, J. Measuring massive multitask
language understanding. CoRR, abs/2009.03300, 2020.
URL https://arxiv.org/abs/2009.03300.
Hong, J., Lee, N., and Thorne, J. Orpo: Monolithic pref-
erence optimization without reference model.
arXiv
preprint arXiv:2403.07691, 2(4):5, 2024.
Hyv¨arinen, A. Some extensions of score matching. Com-
putational statistics & data analysis, 51(5):2499–2512,
2007.
Jung, S., Han, G., Nam, D. W., and On, K.-W. Binary clas-
sifier optimization for large language model alignment.
arXiv preprint arXiv:2404.04656, 2024.
Kakade, S. and Langford, J. Approximately optimal ap-
proximate reinforcement learning. In Proceedings of the
Nineteenth International Conference on Machine Learn-
ing, pp. 267–274, 2002.
Khalifa, M., Elsahar, H., and Dymetman, M.
A distri-
butional approach to controlled text generation. In In-
ternational Conference on Learning Representations,
2021. URL https://openreview.net/forum?
id=jWkw45-9AbL.
Korbak, T., Elsahar, H., Kruszewski, G., and Dymetman,
M. On reinforcement learning and distribution match-
ing for fine-tuning language models with no catastrophic
forgetting. Advances in Neural Information Processing
Systems, 35:16203–16220, 2022a.
9
Energy-Based Preference Model Offers Better Offline Alignment
Korbak, T., Perez, E., and Buckley, C. Rl with kl penalties
is better viewed as bayesian inference. In Findings of
the Association for Computational Linguistics: EMNLP
2022, pp. 1083–1091, 2022b.
Langley, P. Crafting papers on machine learning. In Langley,
P. (ed.), Proceedings of the 17th International Conference
on Machine Learning (ICML 2000), pp. 1207–1216, Stan-
ford, CA, 2000. Morgan Kaufmann.
Lazaro-Gredilla, M., Dedieu, A., and George, D. Perturb-
and-max-product: Sampling and learning in discrete
energy-based models. Advances in Neural Information
Processing Systems, 34:928–940, 2021.
LeCun, Y., Chopra, S., Hadsell, R., Ranzato, M., and Huang,
F. A tutorial on energy-based learning. Predicting struc-
tured data, 1(0), 2006.
Majerek, D., Nowak, W., and Zieba, W. Conditional strong
law of large number. Int. J. Pure Appl. Math, 20(2):
143–156, 2005.
Meng, C., Choi, K., Song, J., and Ermon, S. Concrete score
matching: Generalized score matching for discrete data.
Advances in Neural Information Processing Systems, 35:
34532–34545, 2022.
Meng, Y., Xia, M., and Chen, D. Simpo: Simple preference
optimization with a reference-free reward. arXiv preprint
arXiv:2405.14734, 2024.
Mitchell, E.
A note on dpo with noisy preferences
& relationship to ipo,
2023.
URL https://
ericmitchell.ai/cdpo.pdf.
Munos, R. and Szepesv´ari, C. Finite-time bounds for fitted
value iteration. Journal of Machine Learning Research, 9
(5), 2008.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C.,
Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A.,
et al. Training language models to follow instructions
with human feedback. Advances in neural information
processing systems, 35:27730–27744, 2022.
Park, R., Rafailov, R., Ermon, S., and Finn, C. Disentan-
gling length from quality in direct preference optimiza-
tion. arXiv preprint arXiv:2403.19159, 2024.
Parshakova, T., Andreoli, J.-M., and Dymetman, M.
Distributional reinforcement learning for energy-based
sequential models.
Optimization Foundations for
Reinforcement Learning Workshop at NeurIPS 2019,
2019.
URL https://optrl2019.github.io/
assets/accepted_papers/34.pdf.
Perez, E., Huang, S., Song, F., Cai, T., Ring, R., Aslanides,
J., Glaese, A., McAleese, N., and Irving, G. Red teaming
language models with language models. In Proceedings
of the 2022 Conference on Empirical Methods in Natural
Language Processing, pp. 3419–3448, 2022.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D.,
Ermon, S., and Finn, C. Direct preference optimization:
Your language model is secretly a reward model. In Thirty-
seventh Conference on Neural Information Processing
Systems, 2023. URL https://openreview.net/
forum?id=HPuSIXJaa9.
Schr¨oder, T., Ou, Z., Li, Y., and Duncan, A. B. Training
discrete EBMs with energy discrepancy. In ICML 2023
Workshop: Sampling and Optimization in Discrete Space,
2023. URL https://openreview.net/forum?
id=kFMpJh75Wo.
Simons, G. and Yao, Y.-C. Asymptotics when the number
of parameters tends to infinity in the bradley-terry model
for paired comparisons. The Annals of Statistics, 27(3):
1041–1060, 1999.
Song, F., Yu, B., Li, M., Yu, H., Huang, F., Li, Y., and Wang,
H. Preference ranking optimization for human alignment.
In Proceedings of the AAAI Conference on Artificial In-
telligence, volume 38, pp. 18990–18998, 2024.
Song, Y. and Kingma, D. P. How to train your energy-based
models. arXiv preprint arXiv:2101.03288, 2021.
Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R.,
Voss, C., Radford, A., Amodei, D., and Christiano,
P. F. Learning to summarize with human feedback. Ad-
vances in Neural Information Processing Systems, 33:
3008–3021, 2020.
Tang, Y., Guo, Z. D., Zheng, Z., Calandriello, D., Munos,
R., Rowland, M., Richemond, P. H., Valko, M., Pires,
B. ´A., and Piot, B. Generalized preference optimization:
A unified approach to offline alignment. arXiv preprint
arXiv:2402.05749, 2024.
Tikhonov, A. and Ryabinin, M. It’s all in the heads: Using
attention heads as a baseline for cross-lingual transfer in
commonsense reasoning. CoRR, abs/2106.12066, 2021.
URL https://arxiv.org/abs/2106.12066.
Tunstall, L., Beeching, E., Lambert, N., Rajani, N., Ra-
sul, K., Belkada, Y., Huang, S., von Werra, L., Fourrier,
C., Habib, N., et al. Zephyr: Direct distillation of lm
alignment. arXiv preprint arXiv:2310.16944, 2023.
Wu, W., Junker, B. W., and Niezink, N. Asymptotic compar-
ison of identifying constraints for bradley-terry models.
arXiv preprint arXiv:2205.04341, 2022.
10
Energy-Based Preference Model Offers Better Offline Alignment
Xu, H., Sharaf, A., Chen, Y., Tan, W., Shen, L., Van Durme,
B., Murray, K., and Kim, Y. J. Contrastive preference
optimization: Pushing the boundaries of llm performance
in machine translation. arXiv preprint arXiv:2401.08417,
2024.
Yang, J. Q., Salamatian, S., Sun, Z., Suresh, A. T., and
Beirami, A. Asymptotics of language model alignment,
2024.
Yuan, H., Yuan, Z., Tan, C., Wang, W., Huang, S.,
and Huang, F.
RRHF: Rank responses to align lan-
guage models with human feedback. In Thirty-seventh
Conference on Neural Information Processing Systems,
2023. URL https://openreview.net/forum?
id=EdIGMCHk4l.
Zhan, W., Huang, B., Huang, A., Jiang, N., and Lee, J. Of-
fline reinforcement learning with realizability and single-
policy concentrability. In Conference on Learning Theory,
pp. 2730–2775. PMLR, 2022.
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z.,
Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. Judging
llm-as-a-judge with mt-bench and chatbot arena. Ad-
vances in Neural Information Processing Systems, 36,
2024.
Zhou, W., Agrawal, R., Zhang, S., Indurthi, S. R., Zhao,
S., Song, K., Xu, S., and Zhu, C. Wpo: Enhancing rlhf
with weighted preference optimization. arXiv preprint
arXiv:2406.11827, 2024.
Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford,
A., Amodei, D., Christiano, P., and Irving, G. Fine-tuning
language models from human preferences. arXiv preprint
arXiv:1909.08593, 2019.
A. Proof of the equivalence between slope-1
linearity and minimizer of the RLHF loss
We do not claim any originality for the proofs given in this
section because they are largely just paraphrased versions of
the work by Korbak et al. (2022a); Rafailov et al. (2023) and
others. We include them just for reference and completeness
of the mathematical foundation shared by both DPO and
EPA.
Lemma A.1. The minimizer of the RLHF objective uniquely
exists.
Proof. We show the minimizer πr can be analytically
expressed by
1
Z(x)πref(y|x) exp 1
β r(x, y) where Z(x) =
Σ∞
y πref(y|x) exp 1
β r(x, y) (i.e., a normalizer to make πr
a probabilistic distribution).
From the property of Gibb’s inequality, we know:
πr =
1
Z(x)πref(y|x) exp [ 1
β r(x, y)]
⇔
πr = arg min
πθ
βKL[πθ(y|x)||
1
Z(x)πref(y|x) exp 1
β r(x, y)]
We will complete the proof by showing the βKL-Divergence
on the RHS of the above equation is the RLHF objective
itself plus a constant w.r.t θ:
βKL[πθ(y|x)||
1
Z(x)πref(y|x) exp 1
β r(x, y)]
= β
E
πθ(y|x)[log
πθ(y|x)
1
Z(x)πref(y|x) exp 1
β r(x, y)]
= β
E
πθ(y|x)[log
Z(x)
exp[ 1
β r(x, y)] + log πθ(y|x)
πref(y|x)]
= β
E
πθ(y|x)[log Z(x) −1
β r(x, y) + log πθ(y|x)
πref(y|x)]
= −
E
πθ(y|x)[r(x, y)] + βKL[πθ(y|x)||πref(y|x)]
+ β log Z(x)
= LRLHF (θ) + β log Z(x)
Definition A.2. We say a slope-1 linearity holds when:
rθ(x, y) = r(x, y) + C(x)
where rθ(x, y) = β log πθ(y|x)
πref(y|x).
Theorem A.3 (Theorem of necessity). If πθ = πr, then
slope-1 linearity holds.
Proof. If πθ = πr, then according to Lemma A.1, we have:
πθ =
1
Z(x)πref(y|x) exp 1
β r(x, y)
11
Energy-Based Preference Model Offers Better Offline Alignment
Take the logarithm of both sides of this equation, we have:
log πθ = log πref + 1
β r(x, y) −log Z(x)
After moving the two log terms to the same side, we get
slope-1 linearity:
β log πθ
πref
= r(x, y) −β log Z(x)
Theorem A.4 (Theorem of sufficiency). If slope-1 linearity
holds, then πθ = πr.
Proof. From the Theorem of necessity, we know:
β log πr
πref
= r(x, y) −β log Z(x)
Substracting the slope-1 linearity from this equation, we
get:
β log πr
πref
−β log πθ
πref
= −β log Z(x) −C(x)
Eliminating the non-zero β from both sides and taking the
exponential, we have:
πr
πθ
= f(x)
where f(x) =
1
Z(x) exp[ 1
β C(x)]. Moving πθ to the RHS, we
get:
πr = πθf(x)
Taking Σ∞
y for both sides, we can sum up both distributions
πr and πθ to one:
1 = f(x)
Therefore,
πr = πθf(x) = πθ
Note that from the above proof, we can easily get the follow-
ing corollary because f(x) = 1 ⇔C(x) = −β log Z(x).
Corollary A.5. when a rθ satisfies slope-1 linearity, it is
unique.
B. Theoretical Aspect of the Infinite
Preference Model
We will first give our proof of the guaranteed unique exis-
tence of our IPM’s MLE. Then, we will discuss how BTM
is flawed for an infinite space of y|x.
B.1. On IPM’s MLE
We make the following mild assumptions about the structure
of human preference:
Assumption B.1. D·|x = {y|p(y|x) > 0} is a finite set.
Assumption B.2. r(x, y) −→−∞for any y /∈D·|x and
r(x, y) < +∞for any y ∈D·|x.
Note that the above two assumptions are just one of many
sufficient assumptions that make the partition function exist
as a finite real number. Also note that the finity of D·|x and
the infinity of the space of y|x are two different things that
can certainly co-exist. Namely, D·|x is a subset of the space
of y|x. The number of y outside of D·|x is still infinitely
large. The two assumptions in plain words are simply that
we assume humans will only possibly prefer a finite set of
responses. Note that this does not mean that the finite set
D·|x cannot be very large.
Definition B.3. The maximum likelihood estimation objec-
tive of IPM is the negative log-likelihood of preference data
computed as follows:
−Σ∞
y p(y|x) log qθ(y|x)
where
p(y|x)
=
exp[r(x,y)]
Σ∞
y′ exp[r(x,y′)]
and
qθ(y|x)
=
exp[rθ(x,y)]
Σ∞
y′ exp[rθ(x,y′)].
Given the uniqueness in Corollary A.5, we can argue the
following:
Theorem B.4 (Theorem 3.1 in the main content of the pa-
per). The rθ that satisfies the slope-1 linearity is the unique
minimizer of the IPM’s maximum likelihood estimation ob-
jective.
Proof. Again, from the property of Gibb’s inequality, we
know:
qθ(y|x) = p(y|x)
⇔
qθ(y|x) = arg min
q
KL[p(y|x)||q(y|x)]
For the equation on the right, since p(y|x) is a constant w.r.t
θ, we can find qθ is the minimizer of IPM’s objective:
qθ(y|x) = arg min
q
KL[p(y|x)||q(y|x)]
= arg min
q
H[p(y|x)] −Σ∞
y p(y|x) log q(y|x)
= arg min
q
−Σ∞
y p(y|x) log q(y|x)
We then show that qθ(y|x) = p(y|x) is equivalent to slope-1
linearity to complete the proof by taking the logarithm of
12
Energy-Based Preference Model Offers Better Offline Alignment
both sides:
rθ(x, y) −C1(x) = r(x, y) −C2(x)
⇔
rθ(x, y) = r(x, y) + C(x)
where C1(x)
=
log Σ∞
y′ exp[rθ(x, y
′)],
C2(x)
=
log Σ∞
y′ exp[r(x, y
′)] and C(x) = C1(x) −C2(x)
Note that similar proof does not apply to BTM. The fun-
damental reason is that the C1(x) and C2(x) only become
constants when there is an infinity in the sum to cancel out
all y.
B.2. On Bradley-Terry Model’s Flaw
We will show in Proposition B.5 that a very likely choice
of p(yw, yl|x) will lead to multiple minimizers for the max-
imum likelihood estimation of BTM. There are also many
other choices of p(yw, yl|x) are known to cause the exis-
tence of multiple minimizers (Ford, 1957; Bong & Rinaldo,
2022), such as when there is no full connectivity of the
graph made by pairs from p(yw, yl|x), and when all y candi-
dates can only be paired with a single shared winning ˜y, etc.
Therefore, there is no guarantee for the MLE’s uniqueness
without imposing additional constraints on p(yw, yl|x) (i.e,
how the pairs are sampled for DPO). For the loosest suffi-
cient constraints discovered so far to ensure the uniqueness,
one can refer to Bong & Rinaldo (2022). However, to the
best of our knowledge, how such constraints can be applied
to DPO has never been studied in the offline alignment liter-
ature, which is also out of the scope of this paper. Moreover,
in the infinite-candidate scenario, a constraint that is both
necessary and sufficient for the uniqueness of BTM’s
MLE remains unknown to this day. What makes BTM
even more theoretically troublesome in the case of RLHF
is that there is also an infinity for the space of x as well.
Therefore, strictly speaking, there is an infinite number of
BTMs used in DPO. And, the p(yw, yl|x) for every x should
ensure the uniqueness, in order to make DPO really work as
expected. Interestingly, although our EPA loss also needs
an infinite number of IPMs in the strict sense, Theorem B.4
(3.1) ensures the MLE uniqueness of all the IPMs.
Proposition B.5. If there exists a y∗that will never be
sampled (i.e., p(y∗, ·|x) = 0 and p(·, y∗|x) = 0), then
whenever there is a minimizer for Bradley-Terry’s maximum
likelihood estimation, it is not unique.
Proof. Without losing generality, we set β = 1.
Given the log ratio reward parameterization, we have an
intrinsic constraint on rθ:
∞
X
y′
πref exp[rθ] =
∞
X
y′
πθ = 1
If we assume that there is a unique minimizer rˆθ to BTM’s
maximum likelihood estimation, it certainly satisfies the
above constraint:
∞
X
y′
πref exp[rˆθ] = 1
We will then show that another reward also follows the
constraint (hence a valid log ratio reward) and shares the
same expected data likelihood as rˆθ, which contradicts the
uniqueness of rˆθ. We define the other reward as:
˜rˆθ(x, y) =
(
log(exp[rˆθ(x, y) + A(x)] + 1−exp[A(x)]
πref
),
if y = y∗
rˆθ(x, y) + A(x),
otherwise
where A(x) can be any negative constant w.r.t y. This
reward satisfies the constraint because:
∞
X
y′
πref exp[˜rˆθ(x, y′)]
=
∞
X
y′̸=y∗
πref exp[˜rˆθ(x, y′)] + πref exp[˜rˆθ(x, y∗)]
=
∞
X
y′̸=y∗
πref exp[rˆθ(x, y′) + A(x)]
+ πref exp[˜rˆθ(x, y∗)]
= exp[A(x)]
∞
X
y′̸=y∗
πref exp[rˆθ(x, y′)]
+ πref exp[˜rˆθ(x, y∗)]
= exp[A(x)](1 −πref exp[rˆθ(x, y∗)])
+ πref exp[˜rˆθ(x, y∗)]
= exp[A(x)](1 −πref exp[rˆθ(x, y∗)])
+ πref exp[rˆθ(x, y∗) + A(x)] + (1 −exp[A(x)])
= exp[A(x)](1 −πref exp[rˆθ(x, y∗)])
+ exp[A(x)](πref exp[rˆθ(x, y∗)] −1) + 1
= 1
Note that the constraint makes ˜rˆθ correspond to a valid
policy ˜πˆθ = πref exp[˜rˆθ] that sums up to 1. The policy is
also in the range of [0, 1] because:
1) for y = y∗:
˜πˆθ = πref exp[˜rˆθ]
= πref exp[rˆθ + A(x)] + 1 −exp[A(x)]
= exp[A(x)]πref exp[rˆθ] + 1 −exp[A(x)]
= exp[A(x)]πref
πˆθ
πref
+ 1 −exp[A(x)]
= exp[A(x)](πˆθ −1) + 1
13
Energy-Based Preference Model Offers Better Offline Alignment
and since A(x) < 0 and πˆθ ∈[0, 1], we have:
1 ≥exp[A(x)](πˆθ −1) + 1
≥exp[0](πˆθ −1) + 1
= πˆθ ≥0
hence ˜πˆθ ∈[0, 1];
2) for y ̸= y∗:
˜πˆθ = πref exp[˜rˆθ]
= πref exp[rˆθ + A(x)]
= exp[A(x)]πref exp[rˆθ]
= exp[A(x)]πref
πˆθ
πref
= exp[A(x)]πˆθ
∈[0, 1]
Then, we show that this valid log ratio reward is indeed
another minimizer because it leads to the same expected
likelihood of data as rˆθ.
E
p(yw,yl|x)
E
p(yw≻yl|x)[log σ(˜rˆθ(x, yw) −˜rˆθ(x, yl))]
=
∞
X
yw,yl
p(yw, yl|x)
E
p(yw≻yl|x)[log σ(˜rˆθ(x, yw) −˜rˆθ(x, yl))]
=
∞
X
yw̸=y∗,yl̸=y∗
p(yw, yl|x)·
E
p(yw≻yl|x)[log σ(˜rˆθ(x, yw) −˜rˆθ(x, yl))]
=
∞
X
yw̸=y∗,yl̸=y∗
p(yw, yl|x)·
E
p(yw≻yl|x)[log σ(rˆθ(x, yw) + A(x) −rˆθ(x, yl) −A(x))]
=
∞
X
yw̸=y∗,yl̸=y∗
p(yw, yl|x)·
E
p(yw≻yl|x)[log σ(rˆθ(x, yw) −rˆθ(x, yl))]
=
∞
X
yw,yl
p(yw, yl|x)
E
p(yw≻yl|x)[log σ(rˆθ(x, yw) −rˆθ(x, yl))]
=
E
p(yw,yl|x)
E
p(yw≻yl|x)[log σ(rˆθ(x, yw) −rˆθ(x, yl))]
Finally, because the possible choices of A(x) are infinite and
by assuming the full representation capacity of a large neural
network, ˜πˆθ can be represented by some other π˜θ.
Note that if we are given the p(yw, yl|x) defined in Proposi-
tion B.5, even if we have an infinite amount of data, we still
do not have a unique MLE. This refutes a popular claim that
DPO will work better given enough data.
B.3. Fitting IPM with Energy Discrepancy
In our paper, since we simply adapt the general theorems
given by Schr¨oder et al. (2023) to fit the RLHF context, we
only provide proof sketches for quick reference and claim
no originality thereof. We encourage readers to refer to the
original full proof if more details are needed.
Proof sketch for Theorem 3.2. The first functional deriva-
tive of EDx,p(y|x),p(z|y)[E] (we use E = −r to conform to
the convention of EBMs) is given by:
d
dϵEDx,p(y|x),p(z|y)[E + ϵh]
=
E
x,p(y|x)[h(x, y)] −
E
x,p(y|x),p(z|y)
E
pE,ϵ(y′|z)[h(x, y′)]
where
pE,ϵ(y′|z) =
p(z|y′) · exp[−E(x, y′) −ϵh(x, y′)]
∞
P
y′′ p(z|y′′) · exp[−E(x, y′′) −ϵh(x, y′′)]
Then, setting ϵ = 0 and E = −rtrue, we get the first varia-
tion of ED at E = −rtrue to be 0 because the second term
in the derivative becomes identical to the first term:
E
x,p(y|x),p(z|y)
E
pE=−rtrue,ϵ=0(y′|z)[h(x, y′)]
=
E
x,p(y|x),p(z|y)
∞
X
y′
p(z|y′) · exp[rtrue(x, y′)]
∞
P
y′′ p(z|y′′) · exp[rtrue(x, y′′)]
· h(x, y′)
=
E
x,p(y|x),p(z|y)
∞
X
y′
p(z|y′)p(y′|x)
∞
P
y′′ p(z|y′′)p(y′′|x)
· h(x, y′)
= Ex
∞
X
y
X
z
p(y|x)p(z|y)
∞
P
y′ p(z|y′)p(y′|x)h(x, y′)
∞
P
y′′ p(z|y′′)p(y′′|x)
= Ex
∞
X
y′
p(y′|x)h(x, y′)
∞
X
y
X
z
p(y|x)p(z|y)p(z|y′)
∞
P
y′′ p(z|y′′)p(y′′|x)
= Ex
∞
X
y′
p(y′|x)h(x, y′)
X
z
p(z|y′)
∞
P
y
p(z|y)p(y|x)
∞
P
y′′ p(z|y′′)p(y′′|x)
= Ex
∞
X
y′
p(y′|x)h(x, y′)
X
z
p(z|y′)
= Ex
∞
X
y′
p(y′|x)h(x, y′)
= Ex,p(y|x)h(x, y)
14
Energy-Based Preference Model Offers Better Offline Alignment
With the first variation being 0, we will then only need
to show that the second variation of ED at E = −rtrue
is strictly positive to complete the proof that E = −rtrue
is the global unique minimizer of ED. This can be done
by showing that the second derivative becomes an expec-
tation of Varp(y|z)[h(y)] which cannot be negative or zero
because we assume Varp(y|z)[y] > 0 (i.e., Var[Y |Z] > 0).
Concretely, the second derivative of ED is given by:
d2
dϵ2 EDx,p(y|x),p(z|y)[E + ϵh]
= −
E
x,p(y|x),p(z|y)
d2
dϵ2
E
pE,ϵ(y′|z)[h(x, y′)]
= (see (Schr¨oder et al., 2023)’s Lemma 2)
E
x,p(y|x),p(z|y)
"
E
pE,ϵ(y′|z)[h2(x, y′)] −
E
pE,ϵ(y′|z)[h(x, y′)]
2
#
=
E
x,p(y|x),p(z|y)[VarpE,ϵ(y′|z)[h(x, y′)]]
Setting ϵ = 0 and E = −rtrue, we will get the positive
second variation of ED at E = −rtrue.
Proof sketch for Theorem 3.3. Given the property that
Σyp(z|y)f(y) = Σyp(y|z)f(y), the Σy′p(z|y′) in the defi-
nition of ED becomes an expectation over y, i.e., Ep(y′|z).
This enables a statistic estimate using the normal and a mod-
ified Strong Law of Large Numbers (Majerek et al., 2005)
for the expectations outside and inside of the logarithm,
respectively.
B.4. The Perspective of Functional Analysis
An elegant way to understand the theoretical advantage of
EBM/EPA over BTM/DPO is through the lens of functional
analysis. As illustrated in Figure 5, our EBM’s maximum
likelihood estimation loss, the energy discrepancy, and the
RLHF loss can be regarded as three functionals sharing the
same unique global minimizer. The EPA loss, being a sta-
tistical estimate4 of the energy discrepancy, is yet another
functional. However, the Strong Law of Large Numbers
ensures that the error between it and the energy discrepancy
will almost surely vanish with enough amount of negatives
drawn from a properly designed data perturbing source Z.
Therefore, one can view the EPA loss as a locally fuzzy
approximation of the energy discrepancy. The maximum
likelihood estimation loss of BTM is a functional that may
have multiple minimizers. Therefore, without explicit con-
straints to prevent this from happening, chances are that
optimizing the DPO loss (which is strictly speaking also
just a statistical estimate of the maximum likelihood of data,
4Strictly speaking, there is also a constant shift log M, which
we removed from the energy discrepancy to derive Eq.(10) because
it has no topological impact in terms of optimization problems.
LRLHF[r]
LIPM’s MLE[r]
LEPA[r]
EDZ[r]
LBTM’s MLE[r]
LDPO[r]
The global minimizer r∗(= rtrue + C) of LRLHF[r]
Figure 5. A simplified illustration of the topology of the function-
als mentioned in the paper’s theorems. The vertical axis represents
the value of each functional. The horizontal axis represents the
space of r.
probably with less fuzziness) will lead to a different solu-
tion than the minimizer of the RLHF loss. Increasing the
amount of data for DPO can certainly mitigate the fuzziness
but can do nothing to avoid the undesirable structure of
BTM’s maximum likelihood estimation loss governed by
p(yw, yl|x).
C. More Experimental Details
C.1. Implementation Details
C.1.1. MORE HYPERPARAMETER DETAILS
We use 8 A100/A800 GPUs (80G Memory) with ZeRO3
parallelism to train each model in this paper. Global batch
size is fixed to 64. For experiments in Figure 3, we run
two rounds with different seeds (0 and 1) for each model
configuration. Other experiments are only conducted with
seed 0.
C.1.2. LOSS MODIFICATIONS TRICKS
In Table 4, we list how the tricks are applied to the DPO
loss or the narrow EPA loss. For more detailed properties of
them, one can refer to the corresponding previous work. In
summary, length penalty (Park et al., 2024), length normal-
ization (Yuan et al., 2023; Hong et al., 2024; Meng et al.,
2024), constant or dynamic margins (Meng et al., 2024;
Amini et al., 2024), removal of the reference model (Xu
et al., 2024; Hong et al., 2024; Meng et al., 2024), on-policy
weighting (Zhou et al., 2024) and addition of SFT loss (Xu
et al., 2024; Hong et al., 2024), etc.
15
Energy-Based Preference Model Offers Better Offline Alignment
Notation
Trick function
−ref
log πθ(y|x)
πref(y|x) −→log πθ(y|x)
+Lsft
L −→L −log πθ(yw|x)
+lenp
rθ(y|x) −→rθ(y|x) −α|y|
+lenn
rθ(y|x) −→
1
|y|rθ(y|x)
+mc
rθ(yl|x) −→rθ(yl|x) + mc
+wop
L −→Π∗∈{w,l}(Πt
πθ(yt
∗|y0:t−1
∗
,x)
Σy′∈V ocπ2
θ(y′|y0:t−1
∗
,x))
1
|y| · L
Table 4. Notation and function of loss modification tricks proposed
in the offline alignment literature. We consider on-policy weighting
(+wop), length penalty (+lenp), length normalization (+lenn),
constant margin (+mc), removal of the reference model (−ref),
and addition of a SFT loss (+Lsft)
C.2. DPO vs. EPA from More Perspectives
C.2.1. USING WEAK NEGATIVES IN DPO
Training Data
Method
AE 2.0 (%) MT-Bench
UF-binarized
DPO
17.43 / 15.24
7.55
EPA (1:1:1) 21.14 / 20.55
7.29
EPA (1:1:2) 19.20 / 19.26
7.71
+ UF-weak×1
DPO
18.99 / 17.00
7.37
+ UF-weak×2
DPO
18.49 / 18.72
7.42
Table 5. Adding the same number of weak negatives to pair-wise
data for DPO does not show any advantage over EPA. +UF-
weak×1 means to add a copy of UF-binarized with the yl replaced
by a random weak negative for each prompt. +UF-weak×2 means
to add 2 such copies.
It is also interesting to compare DPO and EPA with the
computation complexity and number of weak samples as
strict control variables. We implement this by directly using
the same number of weak negatives as additional alternatives
for yl in pair-wise data for DPO. However, we would like
to emphasize that this is not meaningful for refuting or
supporting our core argument about our EBM’s superiority
over BTM as there is no obvious theoretical link between
such usage and approximating the MLE of either our EBM
or BTM. Suppressing the log ratio reward of weak negatives
is just a necessary condition for the slope-1 linearity as
opposed to a sufficient one. As shown in Table 5, we can
see that there is no major advantage of such usage over EPA.
Method GSM8k-5Shot
MMLU
Winograd
SFT
0.421 (0.014) 0.598 (0.004) 0.800 (0.006)
+DPO
0.463 (0.014)
0.593 (0.004) 0.790 (0.006)
+EPA
0.419 (0.014) 0.591 (0.004) 0.790 (0.006)
Table 6. EPA has slightly higher alignment tax than DPO.
C.2.2. ALIGNMENT TAX
Preference alignment is usually associated with an align-
ment tax: forgetting certain capabilities (e.g., math problem
solving) while enhancing others (e.g., safety, truthfulness,
and helpfulness). We acknowledge that EPA might exhibit
a higher alignment tax than DPO due to the results in Table
6, where we report metrics on GSM8k (Cobbe et al., 2021),
MMLU (Hendrycks et al., 2020), and Winograd (Tikhonov
& Ryabinin, 2021). However, a more comprehensive future
work on this issue is still necessary for a reliable conclu-
sion. For example, although GSM8k and MMLU are of
relatively higher correlation with human preference than
other Exact-Match-based benchmarks, they have poorer cor-
relations than the ones used in the main paper (MT-Bench
and Alpaca-Eval 2.0) (Dubois et al., 2024). Therefore, a
more aggressive alignment method with human preference
could cause the lower scores on GSM8k and MMLU.
C.2.3. PROBING THE SLOPE-1 LINEARITY
Reward models in the past only need to satisfy ranking con-
sistency with the corresponding true rewards. However, for
ideal offline optimization of the RLHF loss, a learned log
ratio reward and its corresponding true reward have to sat-
isfy the slope-1 linearity, a much stronger requirement than
ranking consistency. The reason is that, to satisfy ranking
consistency, the relationship between the two rewards can be
any monotonically increasing function, including those that
are non-linear or linear with the slope being any positive
number.
Therefore, metrics that are based on ranking consistency
to evaluate the log ratio reward (i.e., the learned reward
model) is not meaningful when it comes to offline alignment.
Instead, we evaluate the log ratio reward by probing how
well the linearity is approximated. For this purpose, we
need multiple samples of y (responses) given an x (prompt)
and their true rewards. The test split of the Ultrafeedback
data (Tunstall et al., 2023) can fulfill this purpose because
there are four y for each x and they are scored using the
same scoring scheme used for our training data (i.e., UF-all
and UF-binarized). We randomly sample 500 prompts from
the test split to speed up evaluation while preserving the
general reliability.
Firstly, we consider only how linear the relationship between
16
Energy-Based Preference Model Offers Better Offline Alignment
Method
Pearson (↑)
ˆϵ (↓)
slope-1: NA slope-1: ✓
linearity: ✓linearity: ✓
DPO
0.4693
5.78
EPA (1:1:2)
0.5808
5.26
EPA (1:3:2)
0.5754
5.01
Table 7. Metrics to probe the slope-1 linearity. Both the average
Pearson coefficient (∈[−1, 1]) and the average slope-1 linear
regression error ˆϵ show that EPA is closer to slope-1 linearity than
DPO.
the two rewards is, regardless of the slope. This is exactly
the essence of Pearson correlation analysis. We compute
the Pearson coefficient between the two rewards over the
four y for each prompt. Then, we compute the mean over
all 500 prompts as a metric. We report the results in the first
column of Table 7.
Secondly, we study both the “slope-1” and the “linearity”.
We do this via linear regression with the slope fixed to
1. Specifically, given a prompt, we need to fit a linear
regression model rlearned = 1 · rtrue + b to the 4 coordinates
of (rlearned, rtrue). With simple algebra, the optimal value
of b that minimize the linear regression error ϵ = Σ|rtrue −
rlearned + b|2 can be analytically expressed as ˆb = (1/4) ·
Σ(rlearned −rtrue). Thus, we use ˆb to compute the minimal
error for each given prompt, and then compute the average
minimal error ˆϵ over all 500 prompts as the metric (see
the second column of Table 7). We can also shift the four
rlearned for each prompt by the constant −ˆb, which should
move all regression lines for the slope-1 linearity to the
same location: the diagonal rlearned = 1 · rtrue. This allows
us to visualize the overall degree of how well the slope-1
linearity is approximated. As shown in Figure 6, we can see
that for the top 10% prompts with the best ˆϵ (i.e., smallest
ˆϵ), the slope-1 linearity is well approximated for both EPA
and DPO. However, we can observe that DPO is slightly off
from the slope-1 linearity for the medium 10% and much
so for the worst 10%. On the other hand, for EPA, although
the points are also gradually spreading out when we move
towards the worst 10%, they are still distributed along the
direction of the diagonal. This phenomenon means that EPA
is closer to the slope-1 linearity than DPO, especially for
the worst group of prompts.
True Reward (rtrue)
Log Ratio Reward (rlearned)
EPA (1:3:2)
EPA (1:1:2)
DPO
0
2
4
6
8
10
0
2
4
6
8
10
best 10% : ϵ = 0.30
0
2
4
6
8
10
0
2
4
6
8
10
best 10% : ˆϵ = 0.32
0
2
4
6
8
10
0
2
4
6
8
10
medium 10% : ˆϵ = 2.53
0
2
4
6
8
10
0
2
4
6
8
10
medium 10% : ˆϵ = 2.54
0
2
4
6
8
10
0
2
4
6
8
10
worst 10% : ˆϵ = 23.09
0
2
4
6
8
10
0
2
4
6
8
10
worst 10% : ˆϵ = 23.36
0
2
4
6
8
10
0
2
4
6
8
10
best 10% : ˆϵ = 0.29
0
2
4
6
8
10
0
2
4
6
8
10
medium 10% : ˆϵ = 2.45
0
2
4
6
8
10
0
2
4
6
8
10
worst 10% : ˆϵ = 20.47
Figure 6. EPA vs DPO in terms of the slope-1 linearity. Each
average slope-1 linear regression error ˆϵ is computed on 500 ×
10% = 50 prompts. Although the difference between EPA and
DPO only becomes noticeable for the “WORST 10%” group in
the visualization (i.e., how close the partially transparent red dots
are to the diagonal), the difference in ˆϵ is conspicuous.
17