Question:
The core contribution of the paper is the proposal of a very simple, modular and theoretically complete new method to solve the maximum entropy inverse reinforcement learning problem. Regarding the content of the article, the following statements are correct:
A. This method decomposes the complex IRL problem into a classification problem and a regression problem.
B. The paper adopts the MaxEnt IRL proposed by Li Xiaran et al. in 2008 as the baseline model to demonstrate the advantages of the new method.
C. When the reward function is linear and the traditional methods still use nonlinear models (incorrect model setting), the new method outperforms the traditional methods significantly in reward recovery and policy matching through the flexible neural network function approximator.
D. The paper closely links IRL with the DDC model in economics and the value function fitting (Fitted Q-Iteration) in reinforcement learning, playing the role of a bridge.

Answer: AD

Inverse reinforcement learning (IRL) aims to explain observed behavior by uncovering an
underlying reward. In the maximum-entropy or Gumbel-shocks-to-reward frameworks, this
amounts to fitting a reward function and a soft value function that together satisfy the soft
Bellman consistency condition and maximize the likelihood of observed actions. While this
perspective has had enormous impact in imitation learning for robotics and understanding dynamic
choices in economics, practical learning algorithms often involve delicate inner-loop optimization,
repeated dynamic programming, or adversarial training, all of which complicate the use of modern,
highly expressive function approximators like neural nets and boosting. We revisit softmax IRL
and show that the population maximum-likelihood solution is characterized by a linear fixed-
point equation involving the behavior policy. This observation reduces IRL to two off-the-shelf
supervised learning problems: probabilistic classification to estimate the behavior policy, and
iterative regression to solve the fixed point. The resulting method is simple and modular across
function approximation classes and algorithms. We provide a precise characterization of the
optimal solution, a generic oracle-based algorithm, finite-sample error bounds, and empirical
results showing competitive or superior performance to MaxEnt IRL.

...

In this paper, we instead first solve a simpler but highly under-specified optimization problem
given by dropping the reward normalization. We show one trivial solution (among many optimal
solutions) is the logarithm of the behavior policy. Moreover, all other solutions are given via a
transformation due to the invariance of observed behavior to the introduction of a state-level potential
function. Therefore, to solve the original problem, it remains to find the state potential function
that would transform the trivial solution in order to also satisfy the previously-relaxed reward
normalization. Overall, this is given by a linear integral equation involving the log behavior policy
and suggests a simple algorithm: classify to learn the behavior policy and iterate a regression a few
times (namely, log(n) times for n observations) to solve the equation.

This yields a meta-algorithm with a simple implementation that just requires calling off-the-
shelf supervised learning routines a few times. The approach is a model-free, completely avoiding
parameterizing the MDP or imposing special structure like linear rewards, and permits flexible
function approximation via flexible supervised learning subroutines, e.g., a neural net regression.

Inverse Reinforcement Learning Using
Just Classification and a Few Regressions
Lars van der Laan1, Nathan Kallus2,3, and Aur´elien Bibaut2
1University of Washington
2Netflix
3Cornell University
Abstract
Inverse reinforcement learning (IRL) aims to explain observed behavior by uncovering an
underlying reward. In the maximum-entropy or Gumbel-shocks-to-reward frameworks, this
amounts to fitting a reward function and a soft value function that together satisfy the soft
Bellman consistency condition and maximize the likelihood of observed actions. While this
perspective has had enormous impact in imitation learning for robotics and understanding dynamic
choices in economics, practical learning algorithms often involve delicate inner-loop optimization,
repeated dynamic programming, or adversarial training, all of which complicate the use of modern,
highly expressive function approximators like neural nets and boosting. We revisit softmax IRL
and show that the population maximum-likelihood solution is characterized by a linear fixed-
point equation involving the behavior policy. This observation reduces IRL to two off-the-shelf
supervised learning problems: probabilistic classification to estimate the behavior policy, and
iterative regression to solve the fixed point. The resulting method is simple and modular across
function approximation classes and algorithms. We provide a precise characterization of the
optimal solution, a generic oracle-based algorithm, finite-sample error bounds, and empirical
results showing competitive or superior performance to MaxEnt IRL.
1
Introduction
Behavioral data are abundant in robotics, human–computer interaction, healthcare, and economics.
Inverse reinforcement learning (IRL) aims to determine the underlying and unobserved rewards that
rationalize this behavior. Doing so provides a useful lens for explaining observed behavior, learning
system structure, optimizing algorithmic policies, or choosing changes to the system that induce more
preferable behavior. Overall, IRL is a key approach to gleaning generalizable insights from purely
behavioral data, but it is not always so easy to operationalize.
Maximum entropy (MaxEnt) IRL is an especially appealing framework where the behaving agent
is assumed to maximize cumulative rewards plus a bonus for policy entropy (Ziebart et al., 2008).
It is equivalent to dynamic discrete choice (DDC) models assuming a rational agent facing random
Gumbel-distributed shocks to immediate rewards (Rust, 1987). This induces softmax-like behavior,
smoothing out otherwise brittle argmax behavior and allowing to explain observed behavior that may
not always be exactly optimal. Yet, many DDC or MaxEnt IRL algorithms that realize this elegant
theory are difficult to scale, generalize, and tune: nested optimization problems, restrictions to linear
reward models, sensitivity to initialization and step sizes, and complex adversarial training loops.
Much of this complexity arises from too-directly tackling the IRL problem: find the Markov
decision process (MDP) that matches observed transitions, satisfies a reward normalization for
1
arXiv:2509.21172v1  [cs.LG]  25 Sep 2025
identifiability, and maximizes likelihood of observed actions under the policy that maximizes reward
(equivalently, mean reward plus entropy in the MaxEnt persepctive) in this MDP. Many algorithms
directly optimize this problem (Audiffren et al., 2015; Levine et al., 2011; Snoswell et al., 2020;
Ziebart et al., 2008). For example, Ziebart et al. (2008) use gradient ascent in this optimization
problem using a linear parametrization of the reward. Other algorithms nest or alternate searching
over MDPs and solving the RL problem implied by each (Fu et al., 2018; Ho and Ermon, 2016;
Rust, 1987; Wulfmeier et al., 2016). For example, Rust (1987) nests a policy iteration loop inside a
parameter search. (A complete literature review is given in section 6.)
In this paper, we instead first solve a simpler but highly under-specified optimization problem
given by dropping the reward normalization. We show one trivial solution (among many optimal
solutions) is the logarithm of the behavior policy. Moreover, all other solutions are given via a
transformation due to the invariance of observed behavior to the introduction of a state-level potential
function. Therefore, to solve the original problem, it remains to find the state potential function
that would transform the trivial solution in order to also satisfy the previously-relaxed reward
normalization. Overall, this is given by a linear integral equation involving the log behavior policy
and suggests a simple algorithm: classify to learn the behavior policy and iterate a regression a few
times (namely, log(n) times for n observations) to solve the equation.
This yields a meta-algorithm with a simple implementation that just requires calling off-the-
shelf supervised learning routines a few times. The approach is a model-free, completely avoiding
parameterizing the MDP or imposing special structure like linear rewards, and permits flexible
function approximation via flexible supervised learning subroutines, e.g., a neural net regression.
Theoretically, we can characterize the statistical behavior for general function approximation and
obtain bounds from assuming high-level PAC bounds on the supervised learning routines, which we
can in particular instantiate for nonparametric least squares under a Bellman completeness assumption
on the hypothesis class and functional complexity measures. Empirically, the the algorithm is simple
and effective.
1.1
Contributions
Our main technical and methodological contributions are:
1. Linear characterization via a normalization equation. We show that all maximum-likelihood
solutions correspond to potential-based reward shapings of (r, v) = (log π, 0), where π(a | s) is
the behavior policy. Imposing a per-state normalization yields a unique solution characterized by
a simple linear equation in a state potential.
2. A generic two-oracle algorithm. Given access to any probabilistic classifier for π(a | s) and
any regression routine for conditional expectations, a short fitted fixed-point loop solves the
normalization equation and outputs estimates (ˆr, ˆv).
3. Finite-sample guarantees. We establish a data-free oracle inequality that controls the error
of approximate fixed-point iterates in terms of classification and regression inaccuracies. We
then derive high-probability (finite-sample) guarantees via sample-splitting, showing that the
estimation error of (ˆr, ˆv) decays at the statistical rates of the classifier and regressor.
2
Problem set up
We review the softmax IRL setting from both structural discrete choice and maximum-entropy
viewpoints, then formalize the optimization problem central to our analysis. We observe data
2
{(si, ai, s′
i)}n
i=1 of state–action-next-state transitions sampled from a distribution P representing the
observed behavior of an agent. The action space A is assumed finite (|A| < ∞), and the state space S
is a measurable spaces and can each be discrete or continuous (in our paper the complexity/cardinality
of both spaces is captured wholly through the complexity of hypothesis classes of functions on them).
The goal of IRL is to infer a reward function r : S ×A →R consistent with the apparent behavior
policy π(a | s) = P(a | s) being optimal in some sense in the corresponding MDP with a given
discount factor γ ∈[0, 1). The data specifies the transition distribution P(s′ | s, a). What remains to
specify an MDP is a reward distribution.
We now discuss in what sense π should be optimal in the recovered MDP. One perspective is that
of DDC (Aguirregabiria and Mira, 2010; Hotz and Miller, 1993; Rust, 1987). In these econometric
models of sequential discrete decisions, at each time t, an agent sees rewards r(st, a)+εt(a) for each
action a, where εt(at) are mean-zero idiosyncratic shocks (of known distribution) to the (unknown
and to-be-inferred) mean reward function r(st, at). In this setting, the value function is
V (s) = E
hP
t≥0 γt(r(st, at) + εt(at)) | s0 = s
i
,
where expectation is over trajectories s0, a0, s1, a1, . . . with distribution π(a0 | s0)P(s1 | a0, s0)π(a1 |
s1) · · · and independent reward shocks εt(a). We can also define
v(s, a) = PV (s, a),
Q(s, a) = r(s, a) + γv(s, a),
using the shorthand that, for any state function f(s) we define
Pf(s, a) = Es′∼P (·|s,a)[f(s′)] =
R
f(s′)P(ds′ | s, a).
Another handy shorthand we will use is that, for any state-action functions f(s, a) and µ(a | s)
(usually a policy), we define the µ-expectation and the log-sum-exp over actions, repsectively, as
µf(s) = P
a µ(a | s)f(s, a),
Ξf(s) = log(P
a exp(f(s, a))).
The rational agent maximizes the value function from any starting point. When εt(at) are
chosen to be i.i.d. Gumbel (type-I generalized extreme value distribution), the resulting rational
behavior is exactly to softmax the Q-function: π∗(a | s) ∝eQ(s,a). And, averaging over the Gumbel
idyosyncracies that will get maxed, the reward to go from state s is ΞQ(s). Thus, we obtain the soft
Bellman equation: v = PΞ(r + γv). Or, written out:
v(s, a) =
R
log P
a′ exp(r(s′, a′) + γ v(s′, a′)) P(ds′ | s, a)
∀s, a.
A wholly equivalent perspective is given by MaxEnt IRL (Ziebart et al., 2008; 2010). Here, one
assumes that the agent seeks to maximizes the cumulative long-run discounted-average rewards plus
a bonus for the entropy of the policy. This also leads to the same π∗being optimal. And, the rewards
plus entropy bonus from state s onward is again ΞQ(s), leading to the same soft Bellman equation.
In view of this, the aim is then to find rewards that induce an MDP that best matches the data in
terms of the (average conditional) likelihood assigned by the induced π∗to observing the behavior
we in fact see, of playing ai in state si (given si and averaging over it). In other words, minimize the
Kullback-Leibler (KL) divergence between π(· | s) and π∗(· | s), averaged over states s ∼P.
While this is of course optimized when π∗= π, it is insufficient to determine rewards: the
softmax policy is invariant to translating the Q-function by an arbitrary state-dependent function. To
resolve this ambiguity, we must impose some reward normalization. For example, in discrete choice
modeling (dynamic or otherwise), to anchor things, it is often assumed that a specific action has
3
zero reward (e.g., the “outside option” of not choosing anything on offer in the context of a product
offering) or alternatively that the sum (equivalently, uniform-weighted mean) of rewards is zero.
Inferred rewards are understood as relative. In MaxEnt IRL it is common to anchor the value of the
behavior policy. In this paper, we consider a general reward normalization requiring the reward to
integrate to zero against a reference conditional measure µ(· | s) (e.g., a point mass at a reference
action, a uniform measure, or the behavior policy). That is, we enforce P
a µ(a | s)r(s, a) = 0 for
all s, or µr = 0
Main problem: Put together, the main problem of interest is to maximize over state-action
functions r, v the conditional log likelihood, subject to soft Bellman and reward normalization:
arg maxr,v
E(a,s)∼P [ r(s, a) + γ v(s, a) −log P
a′ exp(r(s, a′) + γ v(s, a′))]
s.t.
v = PΞ(r + γv)
(soft Bellman),
µr = 0
(reward normalized)
(1)
The remainder of the paper shows that solving (1) reduces to first finding π and then solving
a linear equation to find a state potential function, and then operationalizing this idea in a simple
algorithm.
3
Characterization via a relaxed problem
To highlight the underlying structure, we first drop the normalization and analyze the relaxed problem.
Relaxed problem: We relax reward normalization in eq. (1) but still enforce soft Bellman
consistency:
arg maxr,v
E(a,s)∼P [ r(s, a) + γ v(s, a) −log P
a′ exp(r(s, a′) + γ v(s, a′))]
s.t.
v = PΞ(r + γv)
(soft Bellman)
(2)
This optimization problem is highly under-specified (has many solutions) because the likelihood
objective is flat along any state-potential reshaping of the reward. This is the exact invariance we will
use to back out a solution to eq. (1), after finding just one easy-to-identify optimal solution.
3.1
A trivial solution to the relaxed problem
When the normalization is absent, there is a trivial but informative solution: set the reward to the log
behavior policy, r(s, a) = log π(a | s), and the soft action-value function to zero, v(s, a) = 0. This
choice exactly maximizes the conditional log-likelihood and satisfies the soft Bellman relation, since
the next-state log-partition vanishes under the policy normalization (P
a exp(r(s, a)) = 1).
In the rest of the paper we will define the log-behavior-policy as
u⋆(s, a) = log π(a | s).
Lemma 1 (Trivial optimum for the relaxed problem). (r, v) = (u⋆, 0) is an optimal solution to (2).
In the next subsection, we show that this trivial solution already suffices to compare policy values.
3.2
Aside: Policy comparison
Any feasible solution to eq. (2), including the trivial one (u⋆, 0), suffices for one of IRL’s most funda-
mental tasks: comparing policies. The following theorem establishes that policy value differences are
invariant to potential-based shaping, and hence constant over the entire solution set of eq. (2).
4
Given a reward r, the Q-function of a policy π1 is defined by the fixed point Qπ1
r
= r+γπ1PQπ1
r .
The policy’s value function is V π1
r
= π1Qπ1
r , and we are often interested in comparing values of
policies.
Theorem 1 (Identification of policy value differences). Let (r, v) be an optimal solution to eq. (2).
Then, for any policy π1,
Qπ1
u⋆(s, a) = Qπ1
r (s, a) −Ξ(r + γv)(s).
Consequently, for any two policies π1, π2,
V π1
u⋆(s) −V π2
u⋆(s) = V π1
r (s) −V π2
r (s).
This identification mirrors the central observation of Hotz and Miller (1993): differences in
action-specific value functions are revealed by log-odds of the observed behavior policy (for Gumbel
idiosyncratic shocks to rewards). In our setting, u(s, a) = log π(a | s) plays the same role, showing
that the trivial solution suffices for recovering policy value differences.
Nonetheless, precise reward recovery is required when evaluating policies under different dynam-
ics (e.g., shifting P), different time horizons (e.g., changing γ for long-term vs. short-term planning),
or when estimating structural parameters of r and v in economic models. In such cases, solving
the normalization-constrained problem eq. (1) is essential to identify the true reward, which is the
focus of the remainder of the paper (and, differently from Hotz and Miller, 1993; Hotz et al., 1994,
we will fit v directly and possibly non-parametrically using function approximators, rather than use
simulation to infer moments on a parametrization of r that would be solved by a method of moments).
3.3
An invariance among solutions
The relaxed problem is invariant to potential-based shaping: adding a state-only potential c : S →R
shifts all logits in the softmax by the same amount per state, leaving both feasibility and likelihood
unchanged. This is the entropy-regularized analogue of reward shaping in classical RL (Ng et al.,
1999) and explains why the relaxed objective is flat along an affine subspace.
Lemma 2 (Potential-based shaping invariance). Let (r, v) be feasible in eq. (2) (i.e., satisfies soft
Bellman) and let c : S →R be arbitrary. Define ˜r = r + c −γPc, ˜v = v + Pc, or explicitly
˜r(s, a) = r(s, a) + c(s) −γ
R
c(s′)P(ds′ | s, a),
˜v(s, a) = v(s, a) +
R
c(s′)P(ds′ | s, a).
Then (˜r, ˜v) is also feasible in eq. (2), where it obtains the same objective value as (r, v).
3.4
Solving the original (normalization-constrained) problem
Lemma 1 gives us one optimal solution to the relaxed problem, eq. (2), and lemma 2 gives us a way
to transform it to obtain a variety of other also-optimal solutions. If eq. (1) and eq. (2) have the same
optimal value (which they do), then optimizers of eq. (1) are the optimizers of eq. (2) that satisfy
reward normalization. What remains in order to solve the original problem, (1), is to transform the
trivial solution until we also satisfy the reward normalization, which amounts to a linear equation in
the state potential.
Theorem 2 (IRL as a linear equation). Equation (1) admits a unique optimal solution r⋆, v⋆, where
r⋆= u⋆−v⋆+ µ(γv⋆−u⋆) and v⋆is the unique bounded solution to the fixed point
v⋆= Pµ(γv⋆−u⋆),
5
or, written explicitly, v⋆(s, a) = Es′∼P (·|s,a)[P
a′ µ(a′ | s′)(γv⋆(s′, a′) −u⋆(s′, a′)) | s, a] ∀s, a,
and r⋆(s, a) = u⋆(s, a) −v∗(s, a) + P
a′ µ(a′ | s)(γv∗(s, a′) −u⋆(s, a′)).
The solutions r⋆, v⋆can also be written in the form of lemma 2 as r⋆= u⋆+c⋆−γPc⋆, v⋆= Pc⋆,
where the “right” state potential c⋆: S →R is the unique bounded solution to c⋆−γµPc⋆= −µu⋆.
This theorem is the fulcrum of the paper. It reduces IRL to computing u⋆= log π and then
solving a linear equation (involving u⋆) for c⋆. The reward and soft value functions are then given by
these alone. We next show how to operationalize this with a minimal algorithm.
4
A generic algorithm
The solution characterization suggests a two-oracle procedure: learn ˆu by probabilistic classification,
then solve for ˆv by fitted fixed-point regression. This is described in algorithm 1 below.
Algorithm 1 CLASSIFY-THEN-REGRESS IRL: a simple IRL solver with general function approxi-
mation via blackbox classification and regression oracles
1: Inputs: transitions {(si, ai, s′
i)}n
i=1; reference measure µ(a | s); discount γ; a classification
algorithm; a regression algorithm
2: Classify: fit ˆπ(a | s) by classifying yi = ai given xi = si
3: ˆu(s, a) ←log ˆπ(a | s)
4: Initialize ˆv(0)(s, a) = 0
5: for k = 1, . . . , K do
6:
Regress: fit ˆv(k) by regressing yi = P
a µ(a | s′
i)
γ ˆv(k−1)(s′
i, a)−ˆu(s′
i, a)

on xi = (si, ai)
7: end for
8: Return ˆr(s, a) = ˆu(s, a) + P
a′ µ(a′ | s)
γ ˆv(K)(s, a′) −ˆu(s, a′)

−γˆv(K)(s, a), ˆv = ˆv(K)
We can understand the algorithm as approximating a fixed point iteration. Define for any u the
map
Tuv = Pµ(γv −u).
Then, v⋆is the unique fixed point of Tu⋆. We approximate Tu⋆by plugging in ˆu and by replacing the
map P with a regression algorithm. We then iterate this approximation a few times to obtain ˆv. As
we will argue in the next section, only a modest number of iterations (roughly K ≈log n) are needed
for the iteration error to be negligible relative to statistical error. This is because the operator Tu is a
γ-contraction and therefore fixed point iterations converge exponentially fast.
Our procedure is deliberately blackbox: any calibrated classifier and regressor can be used in
practice. For example, one may use neural networks or gradient boosting with cross-entropy loss
for classification and with mean-squared error for regression. Practically, the regression step also
need not be solved fully. We could, as in Deep Q-Networks (Mnih et al., 2013), parametrize ˆv by a
neural net and, for each “Regress” step, only take a single (or more) stochastic gradient step in the
mean-squared error loss before updating the target network ˆv for the next gradient step.
5
Theoretical Guarantees
This section develops a two-layer analysis of the proposed procedure. We first present a deterministic
(data-free) oracle inequality that bounds the error of any K-step approximate fixed-point iterate in
terms of two primitive quantities: the input mismatch ν := ∥ˆu−u⋆∥2 (the quality of the classification
6
step) and the per-iteration inexactness ηk := ∥ˆv(k) −Tˆuˆv(k−1)∥2 (the quality of each regression
step). Having established this, we then provide high-probability instantiations of these quantities via
sample-splitting: ˆu is learned on one half of the data, and the K regression steps are performed on K
disjoint subfolds of the other half to obtain {ˆv(k)}K
k=1.
Notations for this section.
In the following, let vu be the unique bounded fixed point of Tu, which
exists because Tu is an affine γ-contraction in the supremum norm. Note that v⋆= vu⋆. Let λ be
a state distribution satisfying the stationary λPµ = λ. We write ∥c∥2,λ⊗µ := (Es∼λ[c(s)2])1/2,
∥g∥2,µ⊗λ := (E(s,a)∼λ⊗µ[g(s, a)2])1/2, and ∥g∥2 := (E(s,a)∼P [g(s, a)2])1/2. We assume that the
density ratio λ⊗µ
P (a, s) is uniformly bounded with κ
:=
ess sups,a
(λ⊗µ)(s,a)
P (s,a)
<
∞, so that
∥· ∥2,λ⊗µ ≲∥· ∥2. All subsequent results remain valid if ∥· ∥2,λ⊗µ and ∥· ∥2,λ⊗µ are replaced by
the supremum norms.
5.1
Generic analysis of approximate fixed-point iterates
We begin by isolating two structural facts about the operator Tu and its fixed points. The first shows
that the map u 7→vu is Lipschitz. The second quantifies how inexact iterations accumulate error.
Lemma 3 (Lipschitz stability of the fixed-point map). Let vu and vu′ be the unique fixed points of
Tu and Tu′, respectively. Then
∥vu −vu′∥2,λ⊗µ ≤
1
1−γ ∥u −u′∥2,λ⊗µ.
We next quantify the impact of per-iteration inexactness when iterating Aˆu.
Lemma 4 (Inexact iterations of Tˆu). Fix ˆu. Consider any sequence {ˆv(k)}K
k=0 such that, for k ≥1,
∥ˆv(k) −Tˆuˆv(k−1)∥2,λ⊗µ ≤ηk.
Then, ∥vˆu −ˆv(K)∥2,λ⊗µ ≤γK ∥vˆu −ˆv(0)∥2,λ⊗µ + PK
t=1 γK−t ηt.
Combining the two lemmas with the triangle inequality yields a deterministic bound in terms of
input error (classification error) and iteration error (regression error).
Theorem 3 (Deterministic inequality for the K-step iterate). Fix ˆu satisfying error ν := ∥ˆu −
u⋆∥2,λ⊗µ and a sequence {ˆv(k)}K
k=0 with errors ηk := ∥ˆv(k) −Tˆuˆv(k−1)∥2,λ⊗µ. Then
∥ˆv(K) −v⋆∥2,λ⊗µ ≤
1
1−γ ν + γK ∥vˆu −ˆv(0)∥2,λ⊗µ + PK
k=1 γK−k ηk.
The next lemma shows that this L2(λ ⊗µ) control suffices to control the reward error ∥ˆr −r⋆∥2.
Lemma 5. Let ess sups,a
P (s,a)
λ(s)µ(a|s) < ∞. Then, ∥ˆr(K)−r⋆∥2 ≲∥ˆu−u∥2+∥ˆv(K−1)−v⋆∥2,λ⊗µ+
ηK.
5.2
High-probability instantiation via sample splitting
We now instantiate the deterministic oracle inequality using a sample-splitting scheme and generic
supervised learning PAC guarantees. The n samples are divided into two halves. On the first half, we
estimate ˆu = log ˆπ by probabilistic classification over a function class U ⊆{u : S × A →R} of
log-policies. On the second half, we further partition the data into K disjoint folds of size ⌊n/(2K)⌋.
7
On each fold, we fit the value estimate ˆv(k) by least-squares regression of µ(γˆv(k−1) −ˆu)(s′
i) on
(si, ai) over a convex class V ⊆{v : S × A →R}, and then compute ˆc(k) via a Bellman-like update.
This structured specialization of Algorithm 1 is summarized in Algorithm 2 in Appendix A, and
Lemma 6 provides the corresponding high-probability bounds for ∥ˆu −u⋆∥2 and the per-iteration
inexactness ∥ˆv(k) −Tˆuˆv(k−1)∥2.
To study generalization, assume both U and V are uniformly bounded by some constant B < ∞,
which ensures that ˆu and ˆv are bounded. We assume that the fitted policy ˆπ input to Alg. 2 satisfies a
PAC-type guarantee with respect to a (possibly data-dependent) function ˆρU
 n
2 , δ

(see, e.g., van de
Geer, 2000 Chapter 7 for such guarantees on cross entropy over a generic nonparmateric function
class of conditional probabilities).
Assumption 1 (Policy generalization). For all δ ∈(0, 0.5), ˆu, u ∈U and
Es∼P

KL
π(· | s) ∥ˆπ(· | s)

≤{ˆρU
 n
2 , δ

}2
with probability at least 1 −δ.
The sample-split least-squares instantiation enables a direct application of standard learning
theory to obtain a generalization bound for ˆv(k) as an estimator of Tˆuˆv(k−1). For a sample of size n
and radius r > 0, the localized empirical Rademacher complexity of V is defined as ˆRn(V, r) :=
Eε
h
supv,w∈V: ∥v−w∥2≤r
1
n
Pn
i=1 εi(v −w)(si, ai)
i
where εi
i.i.d.
∼Rad(±1). The PAC-type bound
we will prove for ˆv(k) is expressed in terms of the empirical critical radius:
ˆρV(n, δ) := ˆrV(n) +
q
log(1/δ)
n
,
ˆrV(n) := inf{r > 0 : ˆRn(V, r) ≲r2},
with constants depending only on B. Finally, we assume correct specification of the function classes
through Bellman completeness.
Assumption 2 (Bellman completeness). For any u ∈U and v ∈V, one has Tuv ∈V.
Under Assumption 2, each regression target Tˆuˆv(k−1) belongs to V. Thus, nonparametric least
squares incurs estimation but no approximation error on each fold. The following lemma collects the
resulting high-probability bounds.
Lemma 6 (High-probability events for least-squares classification and K regressions). Suppose
Algorithm 2 is executed (i.e., split the sample for classification and each regression step). Then, for
any δ ∈(0, 1), there exists a constant C = C(B) such that, with probability at least 1 −δ,
∥ˆu −u∥2 ≤C ˆρU
 n
2 , δ
2

,
∥ˆv(k) −Tˆuˆv(k−1)∥2 ≤C ˆρV
j
n
2K
k
,
δ
2K

.
Plugging these events into the deterministic oracle inequality gives the desired finite-sample
guarantee.
Theorem 4 (Sample complexity for the K-step iterate). Let κ := ess sup(s,a)
d(λ⊗µ)
dP
(s, a) < ∞.
Under Assumption 2, for any δ ∈(0, 1) there exists C = C(B, κ) such that, with probability at least
1 −δ, the K-step output satisfies
∥ˆv(K) −v⋆∥2,λ⊗µ ≤
1
1 −γ C ˆρU
 n
2 , δ
2

+ γK ∥vˆu −ˆv(0)∥2,λ⊗µ + 1 −γK
1 −γ C ˆρV
j
n
2K
k
,
δ
2K

.
In particular, taking K ≃c log n (so that γK is negligible) yields
∥ˆv(K) −v⋆∥2,λ⊗µ ≲
1
1 −γ
n
ˆρU
 n
2 , δ
2

+ ˆρV
j
n
2K
k
,
δ
2K
o
.
8
The theorem reduces statistical analysis of the algorithm to well-understood localized complexities
for the chosen function classes. For instance, if U and V are parametric with pseudo-dimensions dU
and dV, then typically ˆρU(m, δ), ˆρV(m, δ) ≍
p
(d + log(1/δ))/m, and the bound becomes
∥ˆv(K) −v⋆∥2,λ⊗µ ≲
1
1 −γ
 r
dU + log(1/δ)
n
+
r
log n (dV + log(log n/δ))
n
!
.
6
Related literature
We expand on connections to structural econometrics, IRL/imitation, and entropy-regularized control.
Structural dynamic discrete choice (DDC).
The nested fixed point approach of Rust (1987)
and the two-step estimator of Hotz and Miller (1993) underlie much of empirical dynamic choice.
Identification in DDC with unobserved heterogeneity and exclusion restrictions is studied by Magnac
and Thesmar (2002). Surveys by Aguirregabiria and Mira (2010) and Arcidiacono and Ellickson
(2011) detail computational and statistical strategies. Our work is closest to softmax DDC with
Gumbel shocks, where the choice probabilities have a logit form and the soft Bellman equation
governs continuation values. The novelty here is to side-step nested policy iterations, simulation
procedures, and generalized methods of moments, and instead leverage generic supervised learn-
ing for nonparametric, flexible estimation using a fitted iteration akin to model-free value-based
reinforcement learning.
Classical IRL and MaxEnt IRL.
Foundational IRL work framed reward recovery as feature
expectation matching (Abbeel and Ng, 2004; Ng and Russell, 2000). Maximum (causal) entropy IRL
(Ziebart, 2010; Ziebart et al., 2008) introduced a probabilistic foundation with convexity and a soft
Bellman consistency. Relative-entropy and maximum-margin variants (Ratliff et al., 2006) provided
alternative regularizations and learning signals. Bayesian IRL (Ramachandran and Amir, 2007)
quantified uncertainty at the cost of heavier computation. Deep IRL extensions include GP-based
nonlinear rewards (Levine et al., 2011) and deep energy-based models (Wulfmeier et al., 2016). Many
related works focus on imitation: GAIL (Ho and Ermon, 2016) casts imitation as occupancy measure
matching via GAN training; AIRL (Fu et al., 2018) recovers a reward up to shaping by jointly training
a discriminator and policy. These methods scale well but bring adversarial optimization challenges
and do not directly target IRL for recovering underlying reward structure. Our work shows how to
characterize the structural maximum-likelihood solution as a solution to an affine fixed point and
leverage supervised learning with off-the-shelf generalization guarantees.
Entropy-regularized RL and control as inference.
Linearly solvable MDPs and path-integral
control (Kappen, 2005; Todorov, 2009) connect entropy-regularization to tractable value computations.
Path-consistency learning (Nachum et al., 2017) enforces soft Bellman identities along trajectories,
and SAC (Haarnoja et al., 2018) brings maximum entropy to off-policy actor–critic learning. Uehara
et al. (2023) study entropy-regularized offline RL and show how slowly tempering regularization
yields strong guarantees for vanilla offline RL. Levine (2018) surveys RL-as-inference, situating
many of these methods under a common umbrella. Our work, like MaxEnt IRL, is complementary to
this: it targets the inverse problem.
9
Value-based offline RL.
Fitted Q-iteration (FQI) is a standard algorithm for offline RL that uses
a regression oracle to approximate and iterate the Bellman optimality operator (Ernst et al., 2005;
Mnih et al., 2013; Munos, 2005). Aside from our handling of the inexactness in ˆu, our analysis of
the inexact fixed point iteration steps is overall similar to the approaches in analyses of FQI (Fan
et al., 2020; Hu et al., 2025; Munos and Szepesv´ari, 2008). This literature also reveals the minimality
of Bellman completeness to avoid suffering exponential-in-horizon terms (Amortila et al., 2020;
Foster et al., 2021; Wang et al., 2021), which strongly suggests it is similarly unavoidable in our
value-based IRL setup without another assumption in its place, as well as possible remedies such as
representation learning (Chang et al., 2022; Pavse et al., 2024) or adversarial estimators that replace
Bellman completeness with an assumption of rich-enough critic class (Chen and Jiang, 2019; Uehara
et al., 2020; 2021).
7
Experiments
We aim to show that accurate reward recovery (up to normalization) and expert-level policies can
be obtained without complicated optimization, matching MaxEnt IRL when the function class is
correctly specified and improving upon it when generic function approximation is required.
We evaluate our approach in three gridworld domains with discounted demonstrations (γ = 0.97).
We compare (i) MaxEnt IRL with gradient-based reward fitting, and (ii) Ours, which first estimates
the trivial solution u = log π using a simple neural classifier, and then computes the normalized
solution by solving the corresponding fixed-point equation. For the MaxEnt baseline we implement
differentiable soft value iteration (temperature = 1), trained with Adam, learning-rate scheduling,
gradient clipping, and early stopping. Existing implementations in the imitation package and in
qzed/irl-maxent only support action-independent rewards and were not applicable. All methods
use the true transition kernel. Reward recovery is evaluated using Q-differences Q(s, a) −Q(s, 0),
which are invariant to normalization and isolate recovery up to potential shaping (reported as RMSE
and correlation). Policy quality is measured by KL divergence, total variation (TV), and top-1
accuracy. All results are averaged over 100 reruns.
Easy identifiable. On a 4 × 4 torus gridworld with linear rewards, both methods achieve
near-perfect recovery (Corr ≥0.99) and indistinguishable policies (KL ≤0.006).
Identifiable. In an 8 × 8 gridworld with tabular linear rewards, MaxEnt IRL is competitive, but
Ours yields lower error (RMSE 0.0086 vs. 0.1081) with equally accurate policies.
Misspecified. With nonlinear rewards on the same 8 × 8 gridworld, the linear MaxEnt baseline
underfits (Corr 0.83, KL 0.049), while Ours with a neural policy head improves both recovery and
policy matching (Corr 0.73, KL 0.0032, TV 0.0083).
Table 1: Results in three gridworld domains. Q-diff (RMSE), KL, and TV: lower is better. Corr and
Top-1: higher is better. Entries are mean ± SE across reruns.
Exp.
Method
RMSE ↓
Corr ↑
KL ↓
TV ↓
Top-1 ↑
Easy
MaxEnt
0.17 ± 0.03
0.996 ± 0.005
0.0057 ± 0.0007
0.045 ± 0.005
1.00
Ours
0.13 ± 0.02
0.992 ± 0.006
0.0011 ± 0.0002
0.016 ± 0.002
1.00
Ident.
MaxEnt
0.27 ± 0.04
0.981 ± 0.005
0.0016 ± 0.0005
0.018 ± 0.004
0.86 ± 0.07
Ours
0.017 ± 0.003
0.999 ± 0.000
0.0000
0.0016 ± 0.0002
0.93 ± 0.07
Hard
MaxEnt
0.83 ± 0.02
0.561 ± 0.031
0.029 ± 0.002
0.094 ± 0.004
0.50 ± 0.03
Ours
0.23 ± 0.01
0.980 ± 0.001
0.0011 ± 0.0001
0.018 ± 0.001
0.90 ± 0.02
10
8
Conclusion, extensions, and future work
We have shown how the softmax IRL problem can be reframed as finding the state potential function
that transform a trivial v-free solution to satisfy the given reward normalization. This suggested
a simple algorithm: fit the behavior policy to find this trivial solution, iterate a few regressions to
make it match the reward normalization and hence solve the problem. The algorithm is simple to
implement, leverage off-the-shelf supervised learning routines, and admits guarantees that allow
flexible nonparametric function approximation. Beyond suggesting a particular algorithm, we hope
this work provides a lens to better understand IRL: if observing an expert then logits of the behavior
policy already must capture long-term value, the rest is just attributing it over time to fit constraints.
We here focused on Gumbel reward shocks (equivalently, entropy regularized agent) as it is the
most common. An extension would be to generalize this to reward shocks with any given cumulative
distribution function F. This change would propagate to appropriately updating the soft Bellman
equation and the trivial u⋆solution, but many of the primary insights and algorithmic structure would
remain. Another possible extension is to replace the fixed point regression iteration with a minimax
estimator, analogous to minimax Q-estimators in offline RL (Uehara et al., 2021). This can allow
relaxing Bellman completeness assumptions by decoupling V from the critic class, so richer classes
only improve performance. Finally, one may consider alternative reward normalizations. This would
lead to a different constraint to pin down the right element in the equivalence class from lemma 2 of
the optimal solution from lemma 1.
References
Pieter Abbeel and Andrew Y. Ng. Apprenticeship learning via inverse reinforcement learning. In
ICML, 2004.
Victor Aguirregabiria and Pedro Mira. Dynamic discrete choice structural models: A survey. Journal
of Econometrics, 156(1):38–67, 2010.
Philip Amortila, Nan Jiang, and Tengyang Xie. A variant of the wang-foster-kakade lower bound for
the discounted setting. arXiv preprint arXiv:2011.01075, 2020.
Peter Arcidiacono and Paul Ellickson. Practical methods for estimation of dynamic discrete choice
models. Annual Review of Economics, 3:363–394, 2011.
Julien Audiffren, Michal Valko, Alessandro Lazaric, and Mohammad Ghavamzadeh. Maximum
entropy semi-supervised inverse reinforcement learning. In International joint conference on
artificial intelligence, 2015.
Jonathan Chang, Kaiwen Wang, Nathan Kallus, and Wen Sun. Learning bellman complete repre-
sentations for offline policy evaluation. In International Conference on Machine Learning, pages
2938–2971. PMLR, 2022.
Jinglin Chen and Nan Jiang. Information-theoretic considerations in batch reinforcement learning. In
International conference on machine learning, pages 1042–1051. PMLR, 2019.
Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning.
Journal of Machine Learning Research, 6, 2005.
Jianqing Fan, Zhaoran Wang, Yuchen Xie, and Zhuoran Yang. A theoretical analysis of deep
q-learning. In Learning for dynamics and control, pages 486–489. PMLR, 2020.
11
Dylan J Foster, Akshay Krishnamurthy, David Simchi-Levi, and Yunzong Xu. Offline reinforcement
learning: Fundamental barriers for value function approximation. arXiv preprint arXiv:2111.10919,
2021.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse reinforce-
ment learning. In ICLR, 2018.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy
maximum entropy deep reinforcement learning with a stochastic actor. In ICML, 2018.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In NeurIPS, 2016.
V. Joseph Hotz and Robert A. Miller. Conditional choice probabilities and the estimation of dynamic
models. The Review of Economic Studies, 60(3):497–529, 1993.
V Joseph Hotz, Robert A Miller, Seth Sanders, and Jeffrey Smith. A simulation estimator for dynamic
models of discrete choice. The Review of Economic Studies, 61(2):265–289, 1994.
Yichun Hu, Nathan Kallus, and Masatoshi Uehara. Fast rates for the regret of offline reinforcement
learning. Mathematics of Operations Research, 50(1):633–655, 2025.
Hilbert J. Kappen. Linear theory for control of nonlinear stochastic systems. Physical Review Letters,
95(20):200201, 2005.
Vladimir Koltchinskii. Oracle inequalities in empirical risk minimization and sparse recovery
problems: Ecole D’Et´e de Probabilit´es de Saint-Flour XXXVIII-2008, volume 2033. Springer,
2011.
Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review.
arXiv preprint arXiv:1805.00909, 2018.
Sergey Levine, Zoran Popovi´c, and Vladlen Koltun. Nonlinear inverse reinforcement learning with
gaussian processes. In NeurIPS, 2011.
Thierry Magnac and David Thesmar. Identifying dynamic discrete decision processes. Econometrica,
70(2):801–816, 2002.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan
Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint
arXiv:1312.5602, 2013.
R´emi Munos. Error bounds for approximate value iteration. In Proceedings of the National Conference
on Artificial Intelligence, volume 20, page 1006. Menlo Park, CA; Cambridge, MA; London;
AAAI Press; MIT Press; 1999, 2005.
R´emi Munos and Csaba Szepesv´ari. Finite-time bounds for fitted value iteration. Journal of Machine
Learning Research, 9(5), 2008.
Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Bridging the gap between
value and policy based reinforcement learning. In NeurIPS, 2017.
Andrew Y. Ng and Stuart J. Russell. Algorithms for inverse reinforcement learning. In ICML, 2000.
12
Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations:
Theory and application to reward shaping. In Icml, volume 99, pages 278–287. Citeseer, 1999.
Brahma S Pavse, Yudong Chen, Qiaomin Xie, and Josiah P Hanna. Stable offline value function
learning with bisimulation-based representations. arXiv preprint arXiv:2410.01643, 2024.
Deepak Ramachandran and Eyal Amir. Bayesian inverse reinforcement learning. In IJCAI, 2007.
Nathan Ratliff, J. Andrew Bagnell, and Martin Zinkevich. Maximum margin planning. In ICML,
2006.
John Rust.
Optimal replacement of gmc bus engines: An empirical model of harold zurcher.
Econometrica: Journal of the Econometric Society, pages 999–1033, 1987.
Aaron J Snoswell, Surya PN Singh, and Nan Ye. Revisiting maximum entropy inverse reinforcement
learning: New perspectives and algorithms. In 2020 IEEE Symposium Series on Computational
Intelligence (SSCI), pages 241–249. IEEE, 2020.
Emanuel Todorov. Efficient computation of optimal actions. Proceedings of the National Academy of
Sciences, 106(28):11478–11483, 2009.
Masatoshi Uehara, Jiawei Huang, and Nan Jiang. Minimax weight and q-function learning for off-
policy evaluation. In International Conference on Machine Learning, pages 9659–9668. PMLR,
2020.
Masatoshi Uehara, Masaaki Imaizumi, Nan Jiang, Nathan Kallus, Wen Sun, and Tengyang Xie. Finite
sample analysis of minimax offline reinforcement learning: Completeness, fast rates and first-order
efficiency. 2021.
Masatoshi Uehara, Nathan Kallus, Jason D Lee, and Wen Sun. Offline minimax soft-q-learning
under realizability and partial coverage. Advances in Neural Information Processing Systems, 36:
12797–12809, 2023.
Sara van de Geer. Empirical Processes in M-estimation, volume 6. Cambridge university press, 2000.
Yuanhao Wang, Ruosong Wang, and Sham Kakade. An exponential lower bound for linearly
realizable mdp with constant suboptimality gap. Advances in Neural Information Processing
Systems, 34:9521–9533, 2021.
Markus Wulfmeier, Peter Ondruska, and Ingmar Posner. Maximum entropy deep inverse reinforce-
ment learning. In AAAI, 2016.
Brian D. Ziebart. Modeling Purposeful Adaptive Behavior With the Principle of Maximum Causal
Entropy. PhD thesis, Carnegie Mellon University, 2010.
Brian D. Ziebart, Andrew Maas, J. Andrew Bagnell, and Anind K. Dey. Maximum entropy inverse
reinforcement learning. In AAAI, 2008.
Brian D Ziebart, J Andrew Bagnell, and Anind K Dey. Modeling interaction via the principle of
maximum causal entropy. In ICML, 2010.
13
A
Instantiated algorithm
Our high-probability analysis is based on the instantiation of algorithm 1 with sample splitting and
regression via non-parametric least squares. This is presented in algorithm 2.
Algorithm 2 Split–Classify–Regress IRL
1: Inputs: transitions {(si, ai, s′
i)}n
i=1; reference measure µ(· | s); discount factor γ; value class
V; steps K; a probabilistic multiclass classifier (outputs ˆπ(a | s)).
2: Split: partition the data into Dcls and Dreg (sizes n/2 each).
3: Classification on Dcls: train the classifier on {(si, ai) ∈Dcls} to obtain ˆπ(a | s); set ˆu(s, a) =
log ˆπ(a | s).
4: Folds for regression: split Dreg into K folds {Dk}K
k=1 of size
 n
2K

.
5: Initialize ˆv(0)(s, a) = 0.
6: for k = 1, . . . , K do
7:
Regression on fold k:
ˆv(k) ∈arg min
v∈V
X
(si,ai,s′
i)∈Dk
n
µ

γ ˆv(k−1) −ˆu

(s′
i)
|
{z
}
outcome yi
−v(si, ai)
o2
.
8:
Bellman update:
ˆck(s) ←µ

γ ˆv(k) −ˆu

(s).
9: end for
10: Return:
ˆr(s, a) = ˆu(s, a) + ˆcK(s) −γ ˆv(K)(s, a),
ˆv(s, a) = ˆv(K)(s, a).
B
Proofs
This appendix contains detailed proofs of all claims. Each proof is accompanied by comments on
intuition and scope.
B.1
Proof of Lemma 1
We start with feasibility: with u = log Pdata, for any s′ we have P
a′ exp{u(s′, a′)} = 1, so the RHS
of the soft Bellman equals 0 and v ≡0 is feasible. For optimality, write the per-sample log-likelihood
ℓ(r, v; s, a) = r(s, a) + γv(s, a) −log
X
a′
er(s,a′)+γv(s,a′).
The right-hand side is the log of a categorical distribution π(· | s) ∝exp{r + γv}, hence
E

ℓ(r, v; s, a)

= −EsKL
Pdata(· | s) ∥π(· | s)

+ EsH
Pdata(· | s)

,
maximized when π(· | s) = Pdata(· | s) for all s, which (u, 0) achieves.
B.2
Proof of Lemma 2
Let ˜r = r + c −γ
R
c(s′)P(ds′ | s, a) and ˜v = v +
R
c(s′)P(ds′ | s, a). For any s′,
X
a′
e˜r(s′,a′)+γ˜v(s′,a′) = ec(s′) X
a′
er(s′,a′)+γv(s′,a′).
14
Taking log and E[· | s, a] shows the soft Bellman is preserved. For the objective, observe
˜r(s, a) + γ˜v(s, a) = r(s, a) + γv(s, a) + c(s),
log
X
a′
e˜r+γ˜v = c(s) + log
X
a′
er+γv,
so the per-sample log-likelihood is unchanged.
B.3
Proof of Theorem 2
By Lemmas 1 and 2, all optima are (u, 0) shaped by a potential c. Enforcing the reward normalization
R
r dµ = 0 yields
0 = µu(s) + c(s) −γ
Z Z
c(s′)P(ds′ | s, a) µ(da | s) = µu(s) + c(s) −γ(Pµc)(s),
i.e., (I −γPµ)c = −µu. Since Pµ is a Markov operator with sup-norm ≤1 and γ < 1, I −γPµ is
invertible on bounded functions via the Neumann series, giving a unique c and hence unique (r⋆, v⋆).
B.4
Proofs for Section 5
Proof of Lemma 3. Recall λ is stationary for the state kernel Pµ(s, ds′) :=
R
P(ds′ | s, a) µ(da | s),
and let λ ⊗µ denote the induced stationary distribution on (s, a) (i.e., a ∼µ(· | s) when s ∼λ).
By definition, vu is the unique fixed point of Tu : v 7→Pµ(γv −u) on functions v : S × A →R,
so
vu = Pµ(γvu −u)
⇐⇒
(I −γPµ) vu = −(Pµ)u.
Hence
vu = −(I −γPµ)−1(Pµ)u,
vu′ = −(I −γPµ)−1(Pµ)u′.
Since γ < 1 and Pµ is a contraction on L2(λ ⊗µ) (by Jensen’s inequality and stationarity of
λ ⊗µ),
∥(I −γPµ)−1∥2,λ⊗µ ≤(1 −γ)−1.
Therefore,
∥vu −vu′∥2,λ⊗µ =

(I −γPµ)−1 (Pµ)(u′ −u)

2,λ⊗µ
≤∥(I −γPµ)−1∥2,λ⊗µ ∥(Pµ)(u′ −u)∥2,λ⊗µ
≤(1 −γ)−1 ∥u −u′∥2,λ⊗µ,
where the last inequality uses that ∥Pµ∥2,λ⊗µ ≤1. Thus u 7→vu is (1 −γ)−1-Lipschitz from
L2(λ ⊗µ) to L2(λ ⊗µ).
Proof of Lemma 4. Let ek := ∥vˆu −ˆv(k)∥2,λ⊗µ. Since vˆu = Tˆuvˆu and Tˆu is a γ-contraction in v,
ek ≤∥Tˆuvˆu −Tˆuˆv(k−1)∥2,λ⊗µ + ∥ˆv(k) −Tˆuˆv(k−1)∥2,λ⊗µ ≤γ ek−1 + ηk.
Unrolling this recursion gives
eK ≤γKe0 +
K
X
t=1
γK−tηt,
which is the desired result.
15
Proof of Theorem 3. By the triangle inequality,
∥ˆv(K) −v⋆∥2,λ⊗µ ≤∥ˆv(K) −vˆu∥2,λ⊗µ + ∥vˆu −v⋆∥2,λ⊗µ.
Apply Lemma 4 to the first term and Lemma 3 with u′ = u⋆to the second to obtain
∥ˆv(K) −v⋆∥2,λ⊗µ ≤γK∥vˆu −ˆv(0)∥2,λ⊗µ +
K
X
t=1
γK−tηt +
1
1 −γ ∥ˆu −u⋆∥2,λ⊗µ.
The specialization with ηt ≤¯η follows by summing the geometric series.
Proof of Lemma 6. Let u = log π, ˆu = log ˆπ, and define the likelihood ratio ϑ(a | s) := ˆπ(a |
s)/π(a | s). Since ˆu, u ∈U are uniformly bounded from below and above, the log–likelihood ratio is
also bounded | log ϑ| ≤L for some L = 2B. The function ϕ(t) := −log t−(t−1) is e−2L–strongly
convex on [e−L, eL], hence for each s,
KL
π(· | s) ∥ˆπ(· | s)

= Eπ[−log ϑ] = Eπ[ϕ(ϑ)] ≥
e−2L
2
Eπ

(ϑ −1)2
.
Moreover, by the mean value theorem on [e−L, eL], | log ϑ| ≤eL |ϑ −1|, so
Eπ

(ˆu −u)2
= Eπ

(log ϑ)2
≤e2L Eπ

(ϑ −1)2
≤2 e4L KL
π(· | s) ∥ˆπ(· | s)

.
Averaging over s and taking square roots yields
∥ˆu −u∥2 ≤
√
2 e2L
EsKL(π∥ˆπ)
1/2
.
By Assumption 1 (KL generalization), EsKL(π∥ˆπ) ≤ˆρ2
U(n/2, δ) with prob. ≥1 −δ, hence
∥ˆu −u∥2 ≤C(L) ˆρU
 n
2 , δ

,
C(L) :=
√
2 e2L.
We turn to the second bound. As shorthand, let
(Tˆuv)(s, a) := (Pµ)(γv −ˆu)(s, a) =
Z
γv(s′, a′) −ˆu(s′, a′)

µ(da′ | s′) P(ds′ | s, a).
By Assumption 2, Tˆuˆv(k−1) ∈V and is given by the population risk minimizer
Tˆuˆv(k−1) = arg min
v∈V E(s,a)∼Pdata
h
(Tˆuˆv(k−1))(s, a) −v(s, a)
i2
= arg min
v∈V E(s,a,s′)∼Pdata
" Z γˆv(k−1)(s′, a′) −ˆu(s′, a′)

µ(da′ | s′) −v(s, a)
#2
.
The estimator ˆv(k) is precisely the empirical risk minimizer over v ∈V for the empirical least-squares
analogue of the right-hand side. By Corollary 1 in Appendix C, for any δ ∈(0, 1), with probability at
least 1 −
δ
2K ,
∥ˆv(k) −Tˆuˆv(k−1)∥2 ≲ρV

mK,
δ
2K

,
where hidden constants only depend on the function-class bound B. We apply the corollary condi-
tional on the training data used to form the pseudo-outcome
Y (s, a, s′) =
Z γˆv(k−1)(s′, a′) −ˆu(s′, a′)

µ(da′ | s′).
16
By sample splitting, the data used to fit ˆv(k) are conditionally i.i.d., and the LS-ERM assumptions
(bounded outcomes, convex V) hold. Moreover, Y (s, a, s′) is uniformly bounded since ˆu and ˆv(k−1)
are uniformly bounded by assumption on the classes U and V.
The result now follows by collecting all bounds and noting that a union bound shows all events
occur with probability at least
1 −

δ
2 + K ·
δ
2K

= 1 −δ.
Proof of Theorem 4. Intersect the events in Lemma 6 to obtain, with probability at least 1 −δ, the
bounds ν := ∥ˆu −u⋆∥2 ≤ˆρU(n/2, δ/2) and ηk := ∥ˆv(k) −Aˆuˆv(k−1)∥2 ≤ˆρV(mK, δ/(2K)) for
all k. Plug these into Theorem 3 to get
∥ˆv(K) −c⋆∥2 ≤
1
1 −γ ˆρU
 n
2 , δ
2

+ γK∥vˆu −ˆv(0)∥2 +
K
X
t=1
γK−t ˆρV
mK,
δ
2K

,
and summing the geometric series yields the displayed inequality. The simplified bound for K ≃
c log n follows by noting that γK decays polynomially in n and is dominated by the statistical
terms.
Proof of Lemma 5.
ˆr(K)−r⋆= (ˆu−u) + µ

γ(ˆv(K)−v⋆)−(ˆu−u)

−γ (ˆv(K)−v⋆) = (µ−I)

γ(ˆv(K)−v⋆)−(ˆu−u)

.
Now add–subtract a Bellman step. Let
εK := ˆv(K) −Tˆuˆv(K−1),
Tˆuv := (Pµ)
γv −ˆu

.
Then
ˆr(K) −r⋆= (µ −I)

γ
Tˆuˆv(K−1) −v⋆
−(ˆu −u)

+ (µ −I)

γ εK

.
Thus
ˆr(K) −r⋆= (µ −I)
h
−(I −γPµ)(ˆu −u) + γ2(Pµ)(ˆv(K−1) −v⋆)
i
+ γ(µ −I) εK.
Assume the coverage condition
κ := ess sup
(s,a)
dP
d(λ ⊗µ)(s, a) < ∞
and stationarity λPµ = λ. Then, by Jensen and stationarity,
∥(Pµ)f∥2,λ⊗µ ≤∥f∥2,λ⊗µ,
and by change of measure,
∥(Pµ)f∥2 ≤κ1/2 ∥(Pµ)f∥2,λ⊗µ ≤κ1/2 ∥f∥2,λ⊗µ.
Hence,
∥ˆr(K) −r⋆∥2 ≲∥ˆu −u∥2 + ∥(Pµ)(ˆu −u)∥2 + ∥(Pµ)(ˆv(K−1) −v⋆)∥2 + ∥εK∥2,
and using the bounds above,
∥ˆr(K) −r⋆∥2 ≲∥ˆu −u∥2 + ∥ˆv(K−1) −v⋆∥2,λ⊗µ + ∥εK∥2,
with all hidden constants depending only on κ and γ.
17
C
High-probability bound for least squares
The following theorem is, up to notation, equivalent to Theorem 5.2. in Koltchinskii (2011)
Theorem 5 (Theorem 5.2. in Koltchinskii (2011)). Let G be a convex class of bounded functions and
let ˆg denote the least squares estimator of the regression function
ˆg := arg min
g∈G
1
n
n
X
j=1
(Yj −g(Xj))2,
where each Yj is almost surely uniformly bounded.
Then, there exist constants K > 0, C > 0 such that for all t > 0,
P

∥ˆg −g⋆∥2
2 ≥inf
g∈G ∥g −g⋆∥2
2 + K

ˆrG(n)2 + t
n

≤Ce−t,
where
ˆrG(n) := inf
n
r > 0 : ˆRn(G, r) ≲r2o
,
ˆRn(G, r) := Eε
"
sup
g,h∈G: ∥g−h∥2≤r
1
n
n
X
i=1
εi{g −h}(Xi)
#
.
We have the following corollary.
Corollary 1 (PAC form; well-specified). Under the conditions of Theorem 5 and assuming g⋆∈G,
for any δ ∈(0, 1), with probability at least 1 −δ,
∥ˆg −g⋆∥2
2 ≤K

ˆrG(n)2 + 1
n log 1
δ

.
Proof. By Theorem 5, for all t > 0,
Pr

∥ˆg −g⋆∥2
2 ≥inf
g∈G ∥g −g⋆∥2
2 + K
ˆrG(n)2 + t
n

≤Ce−t.
Under g⋆∈G, the infimum is 0. Set t = log(C/δ) so that Ce−t = δ. Then with probability at least
1 −δ,
∥ˆg −g⋆∥2
2 ≤K
ˆrG(n)2 + 1
n log C
δ

≤K′ˆrG(n)2 + 1
n log 1
δ

,
absorbing log C into K′.
18