text:
# Einstein-Podolsky-Rosen correlations of spin measurements in two moving inertial frames

## I Introduction

Contemporary considerations of the Einstein-Podolsky-Rosen (EPR) Einstein et al. (1935); Bohm (1951) correlations are restricted mostly to observers staying in a fixed inertial frame of reference (for the theoretical prescriptions and experimental results see, e.g., Refs. Aspect et al. (1981, 1982a, 1982b); Bernstein et al. (1993); Bouwmeester et al. (1999); Pan et al. (2001); Stefanov et al. (2002); Tittel et al. (1998, 1999); Weihs et al. (1998); Zbinden et al. (2001)). This is motivated not only by the experimental requirement but, first of all, because of very serious difficulties connected with description of EPR-like experiments in frames in a relative motion. There are two reasons of the troubles with understanding and calculating the EPR correlation function in this case. The first one is related to the relativity of the notion of simultaneity for moving observers versus instantaneous state reduction. The second problem is related to the nonexistence of a covariant notion of localization in the relativistic quantum mechanics Bacry (1988). The latter deficiency is especially serious because every realistic measurement involves localization in the detector area.

Proposed solutions to these problems strongly depend on the adopted interpretation of quantum mechanics. From an orthodox point of view, attribution of physical meaning to the final probabilities only does not lead to a serious tension between quantum mechanics (QM) and special relativity, so they can “peacefully coexist” Aharonov and Albert (1981); Peres (1995, 2000a, 2000b).

The second line of understanding of QM lies in attributing a physical meaning to the physical state, its time evolution, localization, etc. From this point of view there are serious problems on the border between quantum mechanics and special relativity Bell (1981); Hardy (1992); Stapp (1977); Suarez ; Scarani and Gisin (2002). The most important ones are: lack of the manifest Lorentz covariance of quantum mechanics with finite degrees of freedom and the above mentioned nonexistence of a covariant notion of localization. Troubles with a sharp localization in the relativistic QM arise also if we restrict ourselves to a fixed inertial frame (cf. Hegerfeld theorem Hegerfeld (1998, 2001)).

Following Bell Bell (1981), a consistent formulation of quantum mechanics requires a preferred frame (PF) at the fundamental level (it is interesting that also Einstein and Dirac had admitted such a “nonmechanical” notion of a preferred frame Einstein (1922); Dirac (1951)). Bell gives the very clear point of view on to this question in Ref. Davies and Brown (1986).

A conceptual difficulty related to the notion of the PF lies in an apparent contradiction with the Lorentz symmetry. In Refs. Caban and Rembieliński (1999); Rembieliński (1980, 1997) it was shown that this is not the case: it is possible to arrange Lorentz group transformations in such a way that the Lorentz covariance survives while the relativity principle (democracy between inertial frames) is broken on the quantum level. Moreover, such an approach is consistent with all the classical phenomena. The physical meaning of the new form of the Lorentz group transformations lies in new, absolute synchronization scheme for clocks different from Einstein’s scheme Rembieliński (1980, 1997); Reichenbach (1969); Jammer (1979); Will (1992, 1993); Mansouri and Sexl (1977); Anderson et al. (1998). Both synchronizations, the new and the standard one, are physically inequivalent on the classical level only for velocities greater than the velocity of light. Furthermore, the causality notion, which is implied by the nonstandard absolute synchronization, is more general than the Einstein one and thus it is applicable to nonlocal phenomena. A Lorentz-covariant formulation of QM based on the above mentioned absolute synchronization scheme was given in Caban and Rembieliński (1999). In such a formalism it is possible to define the Lorentz covariant notion of localization and spin, i.e., covariant localized states and a covariant position operator as well as the spin operator transforming properly under the action of the Lorentz group. Note, that exactly these notions are relevant to a correct discussion of (non) locality in QM.

A serious candidate for a PF is the cosmic background radiation frame (CBRF); this choice is connected with possible dynamical (cosmological) distinguishing of a local privileged frame. Most recent EPR experiments performed in Geneva Scarani et al. (2000) have been analyzed according to PF hypothesis Caban and Rembieliński (1999) and give a lower bound for the speed of “quantum information” in CBRF at $`2\times 10^4c`$. Moreover some attention was also devoted to PF as a consequence of a possible breaking of the Lorentz invariance Coleman and Glashow (1999); Colladay and Kostelecký (1998) in high-energy processes.

Since the covariant spin operator also exists in the formulation of QM based on the absolute synchronization scheme Caban and Rembieliński (1999), therefore we can calculate precisely the EPR correlation function for any spin. To our knowledge, our results are the first exact EPR correlation functions obtained for Lorentz-covariant quantum-mechanical systems in moving frames under physically acceptable conditions (some attempts were given in interesting papers by Czachor Czachor (1997a, b); see however Ref. 111In Refs. Czachor (1997a, b) an average value of the operator $`a_\mu W^\mu b_\nu W^\nu `$, where $`W^\mu `$ is Pauli–Lubanski four-vector, was calculated. However, in our opinion, this average cannot be treated as a correlation function of spin measurements in the relativistic case because the spatial part of $`W^\mu `$ is not the spin operator in QM ($`𝐖`$ belongs to the enveloping algebra of Lie algebra of Poincaré group, while the spin operator is a generator of the intrinsic rotations—see, e.g., Ref. Bogolubov et al. (1975)). Moreover, the derivation of correlation function in Refs. Czachor (1997a, b) does not involve localization of measured particles in detectors and is restricted to the measurements performed in the same inertial frames.). Because the resulting formula for the correlation function depends on the velocities of the preferred frame it can also help us to answer the old question concerning the existence of a PF by means of the quantum mechanical EPR experiment and possibly solve the dilemma posed by Bell.

## II Preliminaries

### II.1 Realizations of the Lorentz group in the absolute synchronization scheme

In this section we briefly describe main features of the absolute synchronization scheme mentioned above which is used in this work. The derivation of the presented results can be found in Refs. Caban and Rembieliński (1999); Rembieliński (1980, 1997). The main idea is based on a well-known fact that the definition of time coordinate depends on the procedure used to synchronize clocks Jammer (1979). If we restrict ourselves to the timelike or lightlike signal propagation, the choice of this procedure is a convention Jammer (1979); Mansouri and Sexl (1977); Reichenbach (1969); Will (1992, 1993); Anderson et al. (1998). Now, the form of Lorentz transformations depends on the synchronization scheme, and we can find a synchronization procedure which leads to the desired form of Lorentz transformation preserving instant time (i.e., $`x^0=\mathrm{const}`$) hyperplanes. To perform such a program one has to distinguish an inertial frame, called the preferred frame: Every absolute synchronization scheme (ASS) distinguishes formally such a priviledged inertial frame. We can go from one ASS to another by the action of the so-called synchronization group Caban and Rembieliński (1999); Rembieliński (1997). The classical relativity principle can be formulated in this language as the invariance of physical laws under the action of the synchronization group, or more simply, by the statement that each inertial frame can be chosen as the preferred frame, i.e., the choice of the preferred frame is physically irrelevant. The very serious advantage of ASS is the separation of the two fundamental notions of special relativity, namely, the relativity principle and the Lorentz covariance. In the absolute synchronization scheme, even in the case if the relativity principle is broken, the Lorentz symmetry survives.

Now, each inertial frame is determined by its four-velocity with respect to the preferred one. We shall denote the four-velocity of the preferred frame as seen by an observer at rest in an inertial frame by $`u=(u^0,𝐮)`$.

According to Refs. Rembieliński (1980, 1997) the transformation of the coordinates between inertial frames $`O_u`$ and $`O_u^{}`$ takes the following form:

$$x^{}(u^{})=D(\mathrm{\Lambda },u)x(u),$$
(1a)
where $`\mathrm{\Lambda }`$ is an element of the Lorentz group, $`u`$ is the four-velocity of the preferred frame with respect to $`O_u`$, and $`D(\mathrm{\Lambda },u)`$ is a $`4\times 4`$ matrix depending on $`\mathrm{\Lambda }`$ and $`u`$. This equation must be accompanied by the transformation law for the four-velocity of a preferred frame, which \[according to Eq. (1a)\] takes the form
$$u^{}=D(\mathrm{\Lambda },u)u.$$
(1b)
We point out that both Eqs. (1) are written for contravariant components of coordinate and four-velocity.

The explicit form of the matrix $`D(\mathrm{\Lambda },u)`$ is (see Refs. Caban and Rembieliński (1999); Rembieliński (1997)), for rotations

$$D(R,u)=\left(\begin{array}{cc}1& 0\\ & \\ 0& R\end{array}\right),$$
(2a)
where $`R\mathrm{SO}(3)`$ is a standard rotation matrix, and for boosts,
$$D(w,u)=\left(\begin{array}{cc}(w^0)^1& 0\\ & \\ 𝐰& I+\frac{𝐰𝐰^T}{1+\sqrt{1+|𝐰|^2}}u^0𝐰𝐮^T\end{array}\right),$$
(2b)
where $`w=(w^0,𝐰)`$ denotes a four-velocity of the frame $`O_u^{}`$ as seen by the observer in the frame $`O_u`$.

Hereafter we use the natural system of units with $`c=\mathrm{}=1`$.

Transformations (1) leave the line element $`ds^2=g_{\mu \nu }(u)dx^\mu dx^\nu `$ invariant, where

$$[g_{\mu \nu }(u)]=\left(\begin{array}{cc}1& u^0𝐮^T\\ & \\ u^0𝐮& I+(u^0)^2𝐮𝐮^T\end{array}\right).$$

Notice that $`g(u)`$ is constant (i.e., $`x`$ independent) in each inertial frame and is congruent to the Minkowskian metric $`\eta =\mathrm{diag}(+,,,)`$. It is easy to check that the space element is Euclidean, i.e., $`dl^2=d𝐱^2`$.

The four-velocities $`u`$, $`u^{}`$, and $`w`$ are related by

$$w^0=\frac{u^0}{u_{}^{}{}_{}{}^{0}},𝐰=\frac{(u^0+u_{}^{}{}_{}{}^{0})(𝐮𝐮^{})}{1+u^0u_{}^{}{}_{}{}^{0}(1+𝐮𝐮^{})}.$$
(3)
The relation between coordinates in the standard and the absolute synchronization is given by

$`x_E^0`$ $`=x^0+u^0𝐮𝐱,`$ $`𝐱_E`$ $`=𝐱,`$ (4a)
$`u_E^0`$ $`=(u^0)^1,`$ $`𝐮_E`$ $`=𝐮,`$ (4b)
where the subscript $`E`$ indicates coordinates in the standard (Einstein’s) synchronization, while the coordinates in the absolute synchronization are written without any subscript. We see that only the time coordinate changes. Note also that in the same point of space we have $`\mathrm{\Delta }x_E^0=\mathrm{\Delta }x^0`$, so the time lapse is the same in both synchronizations. The coordinates $`x_E`$ in the Einstein’s synchronization transform according to the standard law, i.e., $`x_{}^{}{}_{E}{}^{\mu }=\mathrm{\Lambda }^\mu {}_{\nu }{}^{}x_{E}^{\nu }`$.

It is important to stress that the transformations (1) form a realization of the Lorentz group which transforms linearly space-time coordinates according to Eq. (1a) and simultaneously, nonlinearly transforms the PF four-velocity according to (1b). The round-trip velocity of light is invariant under Eqs. (1). In particular, the Reichenbach synchronization coefficient Reichenbach (1969); Jammer (1979) is given by $`ϵ(𝐧,𝐮)=(1u^0𝐧𝐮)/2`$. Moreover, from Eqs. (4) we have the following relation between velocities in the absolute and the standard synchronizations:

$$𝐯=\frac{𝐯_E}{1{\displaystyle \frac{𝐯_E𝐮_E}{u_E^0}}},$$
(5a)
$$𝐯_E=\frac{𝐯}{1+u^0𝐯𝐮}.$$
(5b)
Notice that for $`|𝐯_E|>1`$ the above formulas have singularities, i.e., if a superluminal propagation (possibly related to the nonlocality of the theory) takes place then both descriptions are no longer equivalent and consequently an ASS is physically distinguished in such a case even on the classical level Rembieliński (1997). It is remarkable that the velocity manifold of spacelike particles is a proper carrier space for the Lorentz group only in an ASS Rembieliński (1997).

We point out that the triangular form (2b) of a boost matrix implies that under Lorentz transformations the time coordinate is only rescaled by a positive factor, i.e., $`x_{}^{}{}_{}{}^{0}=x^0/w^0`$, so the time ordering of events cannot be inverted by any Lorentz transformations between inertial frames, regardless of the space-time separation. This is important in the QM context because the transformations of time do not involve position operators.

### II.2 Lorentz covariant quantum mechanics

The Lorentz-covariant QM was discussed in the framework of an absolute synchronization scheme in Ref. Caban and Rembieliński (1999). We associate with each inertial observer in $`O_u`$ a Hilbert space $`_u`$, so we have a bundle of Hilbert spaces rather than a single Hilbert space of states. It has been shown in Ref. Caban and Rembieliński (1999) that one can introduce Hermitian momentum and coordinate four-vector operators satisfying

$`[\widehat{x}^\mu (u),\widehat{p}_\nu (u)]`$ $`=i\left({\displaystyle \frac{u_\nu \widehat{p}^\mu (u)}{u_\lambda \widehat{p}^\lambda (u)}}\delta _\nu ^\mu \right),`$ (6a)
$`[\widehat{p}_\mu (u),\widehat{p}_\nu (u)]`$ $`=0,`$ (6b)
$`[\widehat{x}^\mu (u),\widehat{x}^\nu (u)]`$ $`=0.`$ (6c)
We see that $`\widehat{x}^0`$ commutes with all the observables. This allows us to interpret $`\widehat{x}^0`$ as a parameter just like in the standard nonrelativistic quantum mechanics. Moreover, for $`\widehat{\stackrel{}{x}}`$ Eq. (6a) is equivalent to $`[\widehat{x}^i,\widehat{p}_k]=i\delta _k^i`$ and $`[\widehat{x}^i,\widehat{p}_0]=\widehat{p}^i/\widehat{p}^0`$ (notice that the covariant components $`u_i=0`$ in each frame), i.e., it has the standard form. We stress that the commutation relations (6) are *covariant* in the absolute synchronization. In fact, we have the following transformation law for four-vector operators

$`U(\mathrm{\Lambda })\widehat{x}^\mu (u)U^{}(\mathrm{\Lambda })`$ $`=[D^1(\mathrm{\Lambda },u)]^\mu {}_{\nu }{}^{}\widehat{x}_{}^{\nu }(u^{}),`$ (7a)
$`U(\mathrm{\Lambda })\widehat{p}_\mu (u)U^{}(\mathrm{\Lambda })`$ $`=[D^T(\mathrm{\Lambda },u)]_\mu {}_{}{}^{\nu }\widehat{p}_{\nu }^{}(u^{}),`$ (7b)
where $`u^{}=D(\mathrm{\Lambda },u)u`$ and $`D(\mathrm{\Lambda },u)`$ is given by Eqs. (2). Using Eqs. (7) we can transform Eqs. (6) to another reference frame. We point out once again that under transformations (7a) for the time component $`\widehat{x}^0`$ does not mix with spatial components $`\widehat{x}^k`$ ($`k=1,2,3`$). One can also check that

$$[\widehat{x}^\mu (u),\widehat{p}^2(u)]=[\widehat{p}_\mu (u),\widehat{p}^2(u)]=0,$$
(8)
which means that a localized state has a definite mass. It is important to stress that the unitary map which connects one choice of ASS to another choice of ASS and preserves Eqs. (6) and (7) does not exist (this means that the synchronization group Caban and Rembieliński (1999); Rembieliński (1997) cannot be unitarily realized in this case). For this reason QM distinguishes an ASS, i.e., a particular preferred frame—the quantum preferred frame. In Ref. Rembieliński (1997) it was shown that the choice of the quantum preferred frame can be done by the spontaneous breaking of the synchronization group. As it was mentioned earlier, a natural candidate for quantum preferred frame is the CBRF 222Another motivation for such a choice is the very recent cosmological hypothesis of the so-called rolling tachyon field Gibbons (2002); Mukohyama (2002); Sen (2002). In Ref. Rembieliński (1997) it was proved that quantization of the tachyonic field must be necessarily associated with the spontaneous breaking of the synchronization group and consequently with the distinguishing of a privileged frame..

Transformations of the Lorentz group induce an orbit in a bundle of Hilbert spaces $`_u`$. Unitary orbits are parameterized by mass and spin, similarly as for standard unitary representations of the Poincaré group.

An orbit induced by an action of the operator $`U(\mathrm{\Lambda })`$ in the bundle of Hilbert spaces under consideration is fixed by the following covariant conditions: (i) $`k^2=m^2`$, (ii) $`\mathrm{sgn}(k^0)`$ is invariant; for physical representations $`k^0>0`$, $`\mathrm{sgn}(k^0)=1`$. As a consequence there exists a positive defined Lorentz-invariant measure

$$d\mu (k,m)=d^4k\theta (k^0)\delta (k^2m^2).$$
(9)
Now, applying the Wigner method and using Eqs. (7) one can easily determine the action of the operator $`U(\mathrm{\Lambda })`$ on a basis of eigenvectors of the four-momentum operator Caban and Rembieliński (1999)

$$\widehat{p}_\mu (u)|k,u,m;s,\sigma =k_\mu |k,u,m;s,\sigma .$$
(10)
We find 333Here we have chosen slightly different convention that in Ref. Caban and Rembieliński (1999) in the definition of this unitary action.

$$U(\mathrm{\Lambda })|k,u,m;s,\sigma =𝒟^s(R_{(\mathrm{\Lambda },u)})_{\lambda \sigma }|k^{},u^{},m;s,\lambda ,$$
(11)
where the contravariant components $`u^\mu `$ and $`k^\mu `$ transform as follows

$$u^{}=D(\mathrm{\Lambda },u)u=D(L_u^{},\stackrel{~}{u})\stackrel{~}{u},$$
(12)
$$k^{}=D(\mathrm{\Lambda },u)k,$$
(13)
while

$$\begin{array}{cc}\hfill R_{(\mathrm{\Lambda },u)}& =D(R_{(\mathrm{\Lambda },u)},\stackrel{~}{u})\hfill \\ & =D^1(L_u^{},\stackrel{~}{u})D(\mathrm{\Lambda },u)D(L_u,\stackrel{~}{u})\mathrm{SO}(3).\hfill \end{array}$$
(14)
Here $`\stackrel{~}{u}=(1,\mathrm{𝟎})`$, $`u=D(L_u,\stackrel{~}{u})\stackrel{~}{u}`$ and $`𝒟^s`$ is the standard spin $`s`$ matrix representation of $`\mathrm{SU}(2)`$, $`s=0,\frac{1}{2},1,\mathrm{}`$; $`\sigma ,\lambda =s,s+1,\mathrm{},s1,s`$. $`R_{(\mathrm{\Lambda },u)}`$ is a Wigner rotation belonging to the little group of a vector $`\stackrel{~}{u}`$. It should be noted that in this approach, contrary to the standard one, representations of the Poincaré group are induced from the little group of the vector $`\stackrel{~}{u}`$, and not $`\stackrel{~}{k}=(m,0,0,0)`$. Finally, the normalization condition for the basis vectors takes the form

$$k,u,m;s,\lambda |k^{},u,m;s^{},\lambda ^{}=2k^0\delta ^3(\underset{¯}{𝐤}^{}\underset{¯}{𝐤})\delta _{s^{}s}\delta _{\lambda ^{}\lambda },$$
(15)
where $`\underset{¯}{𝐤}`$ denotes the vector formed from covariant components of the momentum, i.e., $`\underset{¯}{𝐤}=(k_1,k_2,k_3)`$.

### II.3 The localized states and spin

Following Ref. Caban and Rembieliński (1999) we construct the localized states (i.e., the eigenvectors of the position operator which coincides in PF with the Newton-Wigner one) and the covariant spin operator. Eigenstates of the position operator $`\widehat{\stackrel{}{x}}(u)`$ (locked up in the $`t_0=0`$) are of the form Caban and Rembieliński (1999)

$$|𝐱,u,m;s,\sigma =\frac{1}{(2\pi )^{3/2}}\frac{d^3\underset{¯}{𝐤}}{2\omega (\underset{¯}{𝐤})}\sqrt{u^\lambda k_\lambda }e^{i\underset{¯}{𝐤}𝐱}|k,u,m;s,\sigma ,$$
(16)
where $`\omega (\underset{¯}{𝐤})=k^0`$ is a positive solution of the dispersion relation $`g_{\mu \nu }(u)k^\mu k^\nu =m^2`$. In the Schrödinger picture, after time $`t=x^0`$ they develop as

$$\begin{array}{c}|x^0,𝐱,u,m;s,\sigma \hfill \\ \hfill =\frac{1}{(2\pi )^{3/2}}\frac{d^3\underset{¯}{𝐤}}{2\omega (\underset{¯}{𝐤})}\sqrt{u^\lambda k_\lambda }e^{ik_\mu x^\mu }|k,u,m;s,\sigma ,\end{array}$$
(17)
which is not an eigenvector of $`\widehat{x}(u)`$ except for $`x^0=0`$. These vectors transform under the action of the Lorentz group according to the following law:

$$U(\mathrm{\Lambda })|x^0,𝐱,u,m;s,\sigma =𝒟^s(R_{(\mathrm{\Lambda },u)})_{\lambda \sigma }|x_{}^{}{}_{}{}^{0},𝐱^{},u^{},m;s,\lambda ,$$
(18)
where $`x^{}`$ and $`u^{}`$ are given by Eqs. (1). Notice that for $`x^0=0`$ we have $`x_{}^{}{}_{}{}^{0}=0`$ and $`x_{}^{}{}_{}{}^{k}=D(\mathrm{\Lambda },u)^k{}_{i}{}^{}x_{}^{i}`$.

Now we define a spin operator 444The operator $`\widehat{𝐒}(u)`$ is connected with the operator $`\widehat{S}_{ij}(u)`$ introduced in Ref. Caban and Rembieliński (1999) by the formula $`\widehat{S}^i(u)=(1/2u^0)(\delta ^{ij}(u^0)^2/(1+u^0)u^iu^j)ϵ^{jkl}\widehat{S}_{kl}(u).`$ in absolute synchronization as follows:

$$\widehat{\stackrel{}{S}}(u)|k,u,m;s,\tau =𝚺_{\sigma \tau }^s|k,u,m;s,\sigma ,$$
(19a)
so
$$\widehat{\stackrel{}{S}}(u)|𝐱,u,m;s,\tau =𝚺_{\sigma \tau }^s|𝐱,u,m;s,\sigma ,$$
(19b)
where $`𝚺^s`$ are the standard generators of rotation in the representation $`𝒟^s`$. The transformation law (11) for states implies the following transformation law for the components of $`\widehat{\stackrel{}{S}}(u)`$:

$$U(\mathrm{\Lambda })\widehat{S}^i(u)U^{}(\mathrm{\Lambda })=R_{(\mathrm{\Lambda },u)}^T{}_{}{}^{i}{}_{j}{}^{}\widehat{S}_{}^{j}(u^{}),$$
(20)
where $`R_{(\mathrm{\Lambda },u)}`$ is a Wigner rotation as above.

Moreover, $`\widehat{S}^i(u)`$ fulfill the standard commutation relations such that

$$[\widehat{S}^i(u),\widehat{S}^j(u)]=iϵ^{ijk}\widehat{S}^k(u).$$
(21)
The invariant $`u`$-independent spin square operator $`\widehat{S}^2`$ can be written in terms of $`\widehat{\stackrel{}{S}}(u)`$ in the standard form

$$\widehat{S}^2=(\widehat{\stackrel{}{S}}(u))^2=s(s+1)I.$$
(22)
We stress that only in that formulation of QM it is possible to introduce the spin operator which transforms properly under the action of the Lorentz group \[see Eq. (20)\] and satisfies the standard commutation relations (21) 555Recently the problem of the spin operator in the standard relativistic QM has been discussed in Ref. Terno ..

To analyze the EPR-type experiments we define an observable $`𝐧\widehat{\stackrel{}{S}}(u)`$, where

$$𝐧=\left(\begin{array}{c}\mathrm{sin}\theta \mathrm{cos}\varphi \\ \mathrm{sin}\theta \mathrm{sin}\varphi \\ \mathrm{cos}\theta \end{array}\right),$$

which is the projection of operator $`\widehat{\stackrel{}{S}}(u)`$ on the direction of a unit vector $`𝐧`$ in the frame of reference $`𝒪_u`$. Since $`\widehat{𝐒}`$ and $`\widehat{𝐱}`$ commute, i.e., $`[\widehat{\stackrel{}{S}}(u),\widehat{\stackrel{}{x}}(u)]=0`$, we can introduce a set of common eigenvectors of $`\widehat{\stackrel{}{x}}(u)`$ and $`𝐧\widehat{\stackrel{}{S}}(u)`$. They are given by

$$\begin{array}{cc}\hfill |𝐱,𝐧,u,m;s,\lambda & =\sqrt{2u^0}\mathrm{exp}\left(i\theta 𝐞_𝐧\widehat{𝐒}(u)\right)|𝐱,u,m;s,\lambda \hfill \\ & =𝒟^s\left(e^{i\theta 𝐞_𝐧𝚺^s}\right)_{\sigma \lambda }\sqrt{2u^0}|𝐱,u,m;s,\sigma ,\hfill \end{array}$$
(23)
where

$$𝐞_𝐧=\left(\begin{array}{c}\mathrm{sin}\varphi \\ \mathrm{cos}\varphi \\ 0\end{array}\right).$$

Vectors (23) satisfy the following eigenequations:

$`\widehat{\stackrel{}{x}}(u)|𝐱,𝐧,u,m;s,\lambda `$ $`=𝐱|𝐱,𝐧,u,m;s,\lambda ,`$ (24a)
$`𝐧\widehat{\stackrel{}{S}}(u)|𝐱,𝐧,u,m;s,\lambda `$ $`=\lambda |𝐱,𝐧,u,m;s,\lambda ,`$ (24b)
with the normalization

$$\begin{array}{c}𝐱,𝐚,u,m;s,\lambda |𝐲,𝐛,u,m;s,\sigma \hfill \\ \hfill =\delta ^3(𝐱𝐲)𝒟^s\left(e^{i\theta _𝐚𝐞_𝐚𝚺}e^{i\theta _𝐛𝐞_𝐛𝚺}\right)_{\lambda \sigma }.\end{array}$$
(25)
Thus the projector corresponding to a region $`\mathrm{\Omega }`$ and to a value $`\lambda `$ of the spin component in the $`𝐧`$ direction in the frame $`𝒪_u`$ is of the form

$$P_{\mathrm{\Omega },𝐧}^\lambda (u)=_\mathrm{\Omega }d^3𝐱|𝐱,𝐧,u,m;s,\lambda 𝐱,𝐧_,u,m;s,\lambda |.$$
(26)
Now, in the Schrödinger picture projectors $`P_{\mathrm{\Omega },𝐧}^\lambda (u)`$, locked up in $`x^0=0`$, are time independent and transform under Lorentz group transformations by means of Eqs. (5) and (23) as follows:

$$U(\mathrm{\Lambda })P_{\mathrm{\Omega },𝐧}^\lambda (u)U^{}(\mathrm{\Lambda })=P_{\mathrm{\Omega }^{},𝐧^{}}^\lambda (u^{}),$$
(27)
here $`𝐧^{}=R_{(\mathrm{\Lambda },u)}𝐧`$ and the region $`\mathrm{\Omega }^{}`$ is obtained from the region $`\mathrm{\Omega }`$ by $`x_{}^{}{}_{}{}^{k}=D(\mathrm{\Lambda },u)^k{}_{i}{}^{}x_{}^{i}`$. We stress that there is no analog of Eq. (27) in the standard formulation of relativistic QM.

## III EPR Correlations

In this section we employ the formalism introduced above to the calculation of the correlation function of the EPR-type experiment. We consider distinguishable particles (the case of identical particles is quite analogous). In this case vectors describing pure states belong to $`_\alpha ^{s_\alpha }(u)_\beta ^{s_\beta }(u)`$, where indices $`\alpha `$ and $`\beta `$ denote particles. We associate with the observers $`𝒜`$ and $``$ the two frames $`𝒜_{u_A}`$ and $`_{u_B}`$, the preferred frame four-velocities with respect to $`𝒜_{u_A}`$ and $`_{u_B}`$ are $`u_A`$ and $`u_B`$, respectively. These observers measure the spin component in the $`𝐚`$ and $`𝐛`$ directions, respectively ($`|𝐚|=|𝐛|=1`$), in the space regions $`A`$ and $`B`$, respectively. Let us denote their observables as $`M_{A,𝐚}`$ and $`M_{B,𝐛}`$, respectively. If we assume that the observer $`𝒜`$ registers the particle $`\alpha `$ and the observer $``$ registers the particle $`\beta `$, then

$`M_{A,𝐚}(u_A)`$ $`={\displaystyle \underset{\mu _\alpha =s_\alpha }{\overset{s_\alpha }{}}}\mu _\alpha P_{A,𝐚}^{\mu _\alpha }(u_A)I{\displaystyle \underset{\mu _\alpha =s_\alpha }{\overset{s_\alpha }{}}}\mu _\alpha \mathrm{\Pi }_{A,𝐚}^{\mu _\alpha },`$ (28a)
$`M_{B,𝐛}(u_B)`$ $`=I{\displaystyle \underset{\mu _\beta =s_\beta }{\overset{s_\beta }{}}}\mu _\beta P_{B,𝐛}^{\mu _\beta }(u_B){\displaystyle \underset{\mu _\beta =s_\beta }{\overset{s_\beta }{}}}\mu _\beta \mathrm{\Pi }_{B,𝐛}^{\mu _\beta },`$ (28b)
where $`P_{A,𝐚}^{\mu _\alpha }`$ and $`P_{B,𝐛}^{\mu _\beta }`$ are given by Eq. (26).

A state of the system under consideration in frame $`O_u`$ at a time $`t`$ is denoted by $`\rho (u,t)`$. Now we write down the sequence of events describing the development of the state $`\rho (u,t)`$.

1. The observer $`𝒜`$ performs measurement with selection of the spin component $`\mu _\alpha `$ in the direction $`𝐚`$, localizing the particle in the space region $`A`$ at a time $`t_A^1`$. This causes the following state reduction

$$\rho (u_A,t_A^1)\frac{\mathrm{\Pi }_{A,𝐚}^{\mu _a}\rho (u_A,t_A^1)\mathrm{\Pi }_{A,𝐚}^{\mu _a}}{\mathrm{Tr}[\rho (u_A,t_A^1)\mathrm{\Pi }_{A,𝐚}^{\mu _a}]}\rho _A(u_A,t_A^1;\mu _a).$$
2. The observer $``$ sees the state $`\rho _A(u_A,t_A^1;\mu _a)`$ at a time $`t_B^1`$ as

$$\rho _A(u_B,t_B^1;\mu _a)=U(\mathrm{\Lambda })\rho _A(u_A,t_A^1;\mu _a)U^{}(\mathrm{\Lambda }),$$

where $`x_B=D(\mathrm{\Lambda },u_A)x_A`$, $`u_B=D(\mathrm{\Lambda },u_A)u_A`$; so $`t_B^1=D(\mathrm{\Lambda },u_A)^0{}_{0}{}^{}t_{A}^{1}`$.
3. The state evolves freely in time from $`t_B^1`$ to $`t_B^2`$ to

$$\rho _A(u_B,t_B^2;\mu _a)=U(t_B^2t_B^1)\rho _A(u_B,t_B^1;\mu _a)U^{}(t_B^2t_B^1).$$
4. The observer $``$ performs measurement with selection of the observable $`M_{B,𝐛}`$ at a time $`t_B^2`$

$$\begin{array}{cc}\hfill \rho _A(u_B,t_B^2;\mu _a)& \frac{\mathrm{\Pi }_{B,𝐛}^{\mu _b}\rho _A(u_B,t_B^2;\mu _a)\mathrm{\Pi }_{B,𝐛}^{\mu _b}}{\mathrm{Tr}[\rho _A(u_B,t_B^2;\mu _a)\mathrm{\Pi }_{B,𝐛}^{\mu _b}]}\hfill \\ & \rho _{AB}(u_B,t_B^2;\mu _b|\mu _a).\hfill \end{array}$$

Recall that in the absolute synchronization $`t_B^2t_B^1=D(\mathrm{\Lambda },u_A)^0{}_{0}{}^{}(t_A^2t_A^1)`$ and $`D(\mathrm{\Lambda },u_A)^0{}_{0}{}^{}>0`$, so the causal relationship between measurements in $`𝒜`$ and $``$ is well established (contrary to the Einstein’s synchronization).

Therefore the probability $`p(\mu _a)`$ that the observer $`𝒜`$ has measured value $`\mu _a`$ and the probability $`p(\mu _b|\mu _a)`$ that the observer $``$ has measured the value $`\mu _b`$ if the observer $`𝒜`$ had measured $`\mu _a`$ are

$$p(\mu _a)=\mathrm{Tr}[\rho (u_A,t_A^1)\mathrm{\Pi }_{A,𝐚}^{\mu _a}],$$

$`p(\mu _b|\mu _a)`$ $`=\mathrm{Tr}[\rho _A(u_B,t_B^2;\mu _a)\mathrm{\Pi }_{B,𝐛}^{\mu _b}]`$
$`={\displaystyle \frac{\mathrm{Tr}\left[\rho (u_A,t_A^1)\mathrm{\Pi }_{A,𝐚}^{\mu _a}U^{}(\mathrm{\Lambda })U^{}(t_B^2t_B^1)\mathrm{\Pi }_{B,𝐛}^{\mu _b}U(t_B^2t_B^1)U(\mathrm{\Lambda })\mathrm{\Pi }_{A,𝐚}^{\mu _a}\right]}{p(\mu _a)}},`$
thus

$$p(\mu _a)p(\mu _b|\mu _a)=\mathrm{Tr}\left[\rho (u_A,t_A^1)\mathrm{\Pi }_{A,𝐚}^{\mu _a}U^{}(\mathrm{\Lambda })U^{}(t_B^2t_B^1)\mathrm{\Pi }_{B,𝐛}^{\mu _b}U(t_B^2t_B^1)U(\mathrm{\Lambda })\mathrm{\Pi }_{A,𝐚}^{\mu _a}\right].$$

Therefore the correlation function reads

$$\begin{array}{cc}\hfill C(𝐚,𝐛)& =\underset{\mu _a,\mu _b}{}\mu _a\mu _bp(\mu _a)p(\mu _b|\mu _a)\hfill \\ & =\underset{\mu _a}{}\mu _a\mathrm{Tr}[\rho (u_A,t_A^1)\mathrm{\Pi }_{A,𝐚}^{\mu _a}U^{}(\mathrm{\Lambda })U^{}(t_B^2t_B^1)\hfill \\ & \times M_{B,𝐛}U(t_B^2t_B^1)U(\mathrm{\Lambda })\mathrm{\Pi }_{A,𝐚}^{\mu _a}].\hfill \end{array}$$
(29)
Recall that in $`_\alpha _\beta `$, $`U(\mathrm{\Lambda })=U(\mathrm{\Lambda })_\alpha U(\mathrm{\Lambda })_\beta `$ and for the free evolution $`U(t)=U(t)_\alpha U(t)_\beta `$.

## IV Correlation Function—a Particular Case

In this section we discuss the case when the measurements in $`𝒜`$ and $``$ are simultaneous. So we assume that $`t_B^2=t_B^1t_B`$ \[i.e., there is no free evolution of a state between measurements 1 and 2, so $`U(t_B^2t_B^1)=I`$\]. Moreover, we assume that the regions $`A`$ and $`B`$ are disjoint. Therefore in Eq. (29) $`M_{B,𝐛}`$ commutes with $`U(\mathrm{\Lambda })\mathrm{\Pi }_{A,𝐚}^{\mu _a}U^{}(\mathrm{\Lambda })`$ and in this case we have ($`t_A^1t_A`$)

$$C(𝐚,𝐛)=\mathrm{Tr}\left[\rho (u_A,t_A)M_{A,𝐚}U^{}(\mathrm{\Lambda })M_{B,𝐛}U(\mathrm{\Lambda })\right].$$
(30)
Assume that the initial state is a pure state $`|\mathrm{\Psi }_\alpha (u_A)_\beta (u_B)`$, thus $`\rho (u_A,t_A)=|\mathrm{\Psi }\mathrm{\Psi }|`$, $`\mathrm{\Psi }|\mathrm{\Psi }=1`$. Since in this case

$$C(𝐚,𝐛)=\mathrm{\Psi }|M_{A,𝐚}(u_A)U^{}(\mathrm{\Lambda })M_{B,𝐛}(u_B)U(\mathrm{\Lambda })|\mathrm{\Psi },$$

therefore using $`U(\mathrm{\Lambda })=U(\mathrm{\Lambda })_\alpha U(\mathrm{\Lambda })_\beta `$, we find

$$\begin{array}{c}C(𝐚,𝐛)=\underset{\mu _\alpha ,\mu _\beta }{}\mu _\alpha \mu _\beta \mathrm{\Psi }|(P_{A,𝐚}^{\mu _\alpha }(u_A)\hfill \\ \hfill U(\mathrm{\Lambda }^1)P_{B,𝐛}^{\mu _\beta }(u_B)U^{}(\mathrm{\Lambda }^1))|\mathrm{\Psi }.\end{array}$$
(31)
Hence, taking into account Eq. (27) we obtain

$$C(𝐚,𝐛)=\underset{\mu _\alpha ,\mu _\beta }{}\mu _\alpha \mu _\beta \mathrm{\Psi }|P_{A,𝐚}^{\mu _\alpha }(u_A)P_{B_𝒜,𝐛^{}}^{\mu _\beta }(u_A)|\mathrm{\Psi },$$
(32)
where $`B_𝒜`$ is obtained from the region $`B`$ by transformation $`x_{}^{}{}_{}{}^{i}=D^1(\mathrm{\Lambda },u_A)^i{}_{j}{}^{}x_{}^{j}`$ and $`𝐛^{}=R_{(\mathrm{\Lambda },u_A)}^T𝐛`$. Now, using the expansion of the vector $`|\mathrm{\Psi }`$ such that

$$\begin{array}{c}|\mathrm{\Psi }=\underset{\lambda _\alpha ,\lambda _\beta }{}d^3𝐱d^3𝐲\mathrm{\hspace{0.17em}2}u^0\psi _{\lambda _\alpha \lambda _\beta }(𝐱,𝐲,u_A)\hfill \\ \hfill \times |𝐱,u_A,m_\alpha ;s_\alpha ,\lambda _\alpha |𝐲,u_A,m_\beta ;s_\beta ,\lambda _\beta ,\end{array}$$
(33)
where

$$\mathrm{\Psi }|\mathrm{\Psi }=d^3𝐱d^3𝐲\mathrm{Tr}\left[\psi ^{}(𝐱,𝐲,u_A)\psi (𝐱,𝐲,u_A)\right]=1,$$
(34)
(hereafter $`\psi `$ denotes the matrix $`\psi =[\psi _{\lambda _\alpha \lambda _\beta }]`$) and using Eqs. (30) and (31) we get, after some calculations, the following formula:

$$\begin{array}{c}C(𝐚,𝐛)=_Ad^3𝐱_{B_𝒜}d^3𝐲\mathrm{Tr}[\psi ^{}(𝐱,𝐲,u_A)𝐚𝚺^{s_\alpha }\psi (𝐱,𝐲,u_A)\hfill \\ \hfill \times (R_{(\mathrm{\Lambda },u_A)}^T𝐛)𝚺_{}^{s_\beta }{}_{}{}^{T}].\end{array}$$
(35)
Consider now the case of the spin $`s_\alpha =s_\beta =1/2`$. We can write then $`\psi (𝐱,𝐲,u_A)=(i/\sqrt{2})\chi (𝐱,𝐲,u_A)\sigma ^2`$ and $`𝚺=\frac{1}{2}𝝈`$, where $`\sigma ^i`$ ($`i=1,2,3`$) are the Pauli matrices. Thus

$$C(𝐚,𝐛)=\frac{1}{4}_Ad^3𝐱_{B_𝒜}d^3𝐲|\chi (𝐱,𝐲,u_A)|^2(𝐚R_{(\mathrm{\Lambda },u_A)}^T𝐛),$$

i.e., up to a factor

$$C(𝐚,𝐛)𝐚R_{(\mathrm{\Lambda },u_A)}^T𝐛.$$
(36)
If the orientation of axes in the frames $`𝒜_{u_A}`$ and $`_{u_B}`$ is the same, we need to deal with boosts $`\mathrm{\Lambda }(𝐰)`$ only and

$$C(𝐚,𝐛)𝐚R_{(\mathrm{\Lambda }(𝐰),u_A)}^T𝐛,$$
(36a)
where $`w^\mu `$ are components of four-velocity of the frame $`_{u_B}`$ with respect to $`𝒜_{u_A}`$.

From Eq. (14) we can calculate the explicit form of the Wigner matrix $`R_{(\mathrm{\Lambda }(𝐰),u_A)}`$,

$$R_{(\mathrm{\Lambda }(𝐰),u_A)}=B^1(u_B)\mathrm{\Omega }(𝐰,u_A)B(u_A),$$
(37)
where

$$B(u)=I+\frac{u^0}{1+u^0}𝐮𝐮^T,$$
(38a)
$$\mathrm{\Omega }(𝐰,u)=I+\frac{1}{1+\sqrt{1+𝐰^2}}𝐰𝐰^Tu^0𝐰𝐮^T,$$
(38b)
and by means of Eq. (3)

$$u_B^0=\frac{u_A^0}{w^0},𝐮_B=𝐮_A\frac{𝐰}{u_A^0}\frac{1+w^0}{1+\sqrt{1+𝐰^2}}.$$
(39)
The corresponding velocities of the preferred frame with respect to frames $`𝒜`$ and $``$ are $`𝝈_A=𝐮_A/u_A^0`$ and $`𝝈_B=𝐮_B/u_B^0`$, respectively, while the velocity of the frame $``$ with respect to $`𝒜`$ is $`𝐕=𝐰/w^0`$ (see Fig. 1).

We remark that it is possible to express $`R_{(\mathrm{\Lambda }(𝐰),u_A)}`$, given by Eq. (37), by these velocities as well as by the corresponding velocities in the Einstein’s synchronization with the help of Eqs. (5) because it is only a reparametrization on the level of *classical* parameters, so it cannot affect the quantum correlations: They are still dependent on the corresponding velocities of PF with respect to the observers. Indeed, as it was discussed in Sec. II.2, the QMs built up on the different PFs are not unitary equivalent. Thus the dependence of quantum correlation functions on the velocities of PF is unremovable because it is a pure quantum phenomenon.

Now, the correlation function (36a) is

$$C(𝐚,𝐛)𝐚B(u_A)\mathrm{\Omega }^T(𝐰,u_A)B^1(u_B)𝐛,$$
(40)
where $`B`$ and $`\mathrm{\Omega }`$ are given by the formulas (38). Note, that the correlation function given by Eq. (40) depends on the choice of PF, i.e., the two correlation functions, say $`C(𝐚,𝐛)`$ obtained for PF with the four-velocities $`u_A`$ and $`u_B`$ with respect to the observers and $`\stackrel{~}{C}(𝐚,𝐛)`$ obtained under *another choice* of PF with the four-velocities $`\stackrel{~}{u}_A`$ and $`\stackrel{~}{u}_B`$ with respect to the observers, *do not turn into themselves* when expressing $`u_A`$ and $`u_B`$ by $`\stackrel{~}{u}_A`$ and $`\stackrel{~}{u}_B`$ or vice versa. This property, related to the above mentioned nonequivalence of QMs built on different PFs, can be used to set up the experiments testing the existence and/or identification of the quantum preferred frame.

Let us discuss some special cases of Eq. (40).

1. $`𝐰=\mathrm{𝟎}`$ (i.e. $`𝐕=\mathrm{𝟎}`$). In this case both measurements are performed in the same inertial frame. It follows from Eq. (40), that the correlation function has the standard nonrelativistic form in this case,

$$C(𝐚,𝐛)𝐚𝐛=\mathrm{cos}\theta _{\mathrm{𝐚𝐛}},$$
(41)
as it should be expected. We would like to point out that the correlation function for relativistic EPR particles calculated in Refs. Czachor (1997a, b) contains corrections of the order $`(\text{particle velocity}/c)^2`$ to Eq. (41). It would be interesting to verify both the predictions experimentally.
2. $`𝐮_A=\mathrm{𝟎}`$ or $`𝐮_B=\mathrm{𝟎}`$. In this case one of the observers performs his/her measurement in the preferred frame. With the help of Eqs. (40) and (39) we find that

$$C(𝐚,𝐛)𝐚𝐛,$$
(42)
that is, we get the standard nonrelativistic formula.
3. Let us assume that the velocities $`𝝈_A`$ and $`𝝈_B`$ are small, i.e. $`|𝝈_A|1`$ and $`|𝝈_B|1`$. Such a situation occurs if the quantum-mechanical preferred frame coincides with the CBRF and the observers’ velocities are similar to the velocity of the solar system (i.e., $`|𝝈_A|`$ and $`|𝝈_B|10^3`$). In this case

$$\begin{array}{cc}\hfill R_{(\mathrm{\Lambda }(𝐰),u_A)}& I+\frac{𝐕𝝈_A^T𝝈_A𝐕^T}{2}\hfill \\ & =I+\frac{𝝈_A𝝈_B^T𝝈_B𝝈_A^T}{2},\hfill \end{array}$$

so

$$C(𝐚,𝐛)𝐚𝐛+\frac{(𝐚\times 𝐛)(𝝈_A\times 𝝈_B)}{2}.$$
(43)
Here $`𝝈_A`$, $`𝝈_B`$ and $`𝐕`$ are related by the approximate formula $`𝝈_B𝝈_A𝐕`$. In the formula (43) the velocities are given in the absolute synchronization scheme but up to the fourth-order corrections they have the same form in terms of velocities $`𝐕_E,𝝈_{A}^{}{}_{E}{}^{},𝝈_{B}^{}{}_{E}{}^{}`$ defined in the Einstein’s synchronization scheme (as it was mentioned above the reparametrization of the *classical* velocities cannot affect the distinguishing of the quantum preferred frame, i.e., the quantum correlation function is still dependent on the corresponding velocities of PF with respect to the observers). The deviation from the standard formula when $`𝐚`$ and $`𝐛`$ are perpendicular is shown in the Fig. 2.

Note that it follows from Eq. (43) that the corrections to the standard formula are of the order 2 in velocities. With the identification of the preferred frame with the CBRF and $`𝒜`$ and $``$ with the solar system these corrections are of the order $`10^6`$. Therefore, we can imagine an experiment testing this identification based on the measurement of the quantum correlations under the condition that the vectors $`𝐚`$ and $`𝐛`$ are perpendicular. In this case the standard part of the correlation function vanishes and only the effect caused by the existence of the quantum preferred frame remains \[see Eq. (43) and Fig. 2\]. Now, unlike in the standard EPR experiments, we should not measure the dependence of the correlation function on the angle between the vectors $`𝐚`$ and $`𝐛`$, but rather its dependence on the change of the velocities of PF, $`𝝈_A`$ and $`𝝈_B`$, caused by the movement of the Earth.
4. Finally we consider the case when velocities of the preferred frame are high. Denoting $`𝐮_A/|𝐮_A|=𝐧_A`$, $`𝐮_B/|𝐮_B|=𝐧_B`$ we obtain in this case

$$\begin{array}{c}R_{(\mathrm{\Lambda }(𝐰),u_A)}I(𝐧_A𝐧_A^T+𝐧_B𝐧_B^T+𝐧_A𝐧_B^T\hfill \\ \hfill (1+2𝐧_A𝐧_B)𝐧_B𝐧_A^T\left)\right(1+𝐧_A𝐧_B)^1,\end{array}$$

hence,

$$\begin{array}{c}C(𝐚,𝐛)𝐚𝐛\frac{1}{1+𝐧_A𝐧_B}[(𝐚𝐧_A)(𝐛𝐧_A)+(𝐚𝐧_B)(𝐛𝐧_B)\hfill \\ \hfill +(𝐚𝐧_B)(𝐛𝐧_A)(1+2𝐧_A𝐧_B)(𝐚𝐧_A)(𝐛𝐧_B)].\end{array}$$
(44)

We point out that the simultaneity of the measurements ($`t_A=t_B`$) is defined in the *corresponding absolute synchronization scheme* related to the choice of the PF 666If the two measurements are performed simultaneously (in absolute synchronization) in the places at the distance $`l`$, the difference of their time in Einstein’s synchronization is $`|\mathrm{\Delta }\tau |\sigma _El/c^2`$ \[cf. Eqs. (4)\], i.e., if PF is CBRF then for $`l1\mathrm{km}`$ we have $`\mathrm{\Delta }\tau 1\mathrm{ns}`$..

## V Conclusions

In the framework of the Lorentz-covariant quantum mechanics with the preferred frame one can build the formalism that allows to calculate correlation function in the EPR-type experiments \[see Eqs. (29) and (30)\] performed in moving inertial frames. We would like to point out that our results are the exact EPR correlation functions obtained for Lorentz-covariant quantum mechanical systems in moving frames under physically acceptable conditions, i.e., taking into account the localization of the particles during the detection and using the spin operator with proper transformation properties under the action of the Lorentz group.

We applied the general result to the case of simultaneous measurements of the spin component for bipartite spin-1/2 system done by the spatially bounded detectors. The resulting correlation function is proportional to $`𝐚R_{(\mathrm{\Lambda },u_A)}^T𝐛`$, where $`𝐚`$ and $`𝐛`$ are the direction vectors and $`R_{(\mathrm{\Lambda },u_A)}`$ is the Wigner rotation matrix associated with the Lorentz transformation $`\mathrm{\Lambda }`$ connecting the frames of the detectors. Next we have studied the limiting cases of this particular correlation function and have shown that in the case when both measurements are performed in the same inertial frame we obtain the standard nonrelativistic result that the correlation function is proportional to the scalar product of the direction vectors. This result also holds if one of the measurements is performed in the preferred frame. We have also found the limit of the correlation function for small velocities and shown that it leads to the correction of the second order in velocities to the standard $`𝐚𝐛`$ relation. On the other hand, the correlation function for the very high velocities of the PF with respect to the observers depends only on the directions of movement of the PF.

It is important to stress that the exact EPR correlation function (29) depends on the PF velocity in an essential way, i.e., this dependence cannot be removed by expressing the correlation function by classical quantities (velocities) given in the Einstein’s synchronization scheme. This means that *the Lorentz-covariant quantum mechanics must distinguish a preferred frame*. The above results can be used to propose a realistic experiment which can answer the question of the existence of quantum-mechanical preferred frame (and its possible identification with the CBRF). A more exhaustive discussion of this problem as well as an analysis of the subtle question concerning the synchronization of clocks in the experimental setup will be given in the forthcoming paper.

###### Acknowledgements.

One of the authors (JR) is grateful to Marek Czachor, Ryszard Horodecki, Valerio Scarani, and Anton Zeilinger for discussions during the second EFS QIT conference in Gdańsk as well as Harvey R. Brown for discussing the conventionality of the synchronization in the special relativity. This work was supported by Lodz University Grant No. 267.


format: text

text:
# Technical Report IDSIA-04-02 - 17 April 2002 - Proceedings of COLT-2002 Self-Optimizing and Pareto-Optimal Policies in General Environments based on Bayes-Mixtures

## 1 Introduction

### Reinforcement learning:

There exists a well developed theory for reinforcement learning agents in known probabilistic environments (like Blackjack) called sequential decision theory \[Bel57, Ber95\]. The optimal agent is the one which maximizes the future expected reward sum. This setup also includes deterministic environments (like static mazes). Even adversarial environments (like Chess or Backgammon) may be seen as special cases in some sense \[Hut00, ch.6\] (the reverse is also true \[BT00\]). Sequential decision theory deals with a wide range of problems, and provides a general formal solution in the sense that it is mathematically rigorous and (uniquely) specifies the optimal solution (leaving aside computational issues). The theory breaks down when the environment is unknown (like when driving a car in the real world). Reinforcement learning algorithms exist for unknown Markov decision processes (mdp$`s`$) with small state space, and for other restricted classes \[KLM96, SB98, Ber95, KV86\], but even in these cases their learning rate is usually far from optimum.

### Performance measures:

In this work we are interested in general (probabilistic) environmental classes $``$. We assume $``$ is given, and that the true environment $`\mu `$ is in $``$, but is otherwise unknown. The expected reward sum (value) $`V_\mu ^p`$ when following policy $`p`$ is of central interest. We are interested in policies $`\stackrel{~}{p}`$ which perform well (have high value) independent of what the true environment $`\mu `$ is. A natural demand from an optimal policy is that there is no other policy yielding higher or equal value in all environments $`\nu `$ and a strictly higher value in one $`\nu `$. We call such a property Pareto-optimality. The other quantity of interest is how close $`V_\mu ^{\stackrel{~}{p}}`$ is to the value $`V_\mu ^{}`$ of the optimal (but infeasible) policy $`p^\mu `$ which knows $`\mu `$ in advance. We call a policy whose average value converges asymptotically for all $`\mu `$ to the optimal value $`V_\mu ^{}`$ if $`\mu `$ is the true environment, self-optimizing.

### Main new results for Bayes-mixtures:

We define the Bayes-mixture $`\xi `$ as a weighted average of the environments $`\nu `$ and analyze the properties of the Bayes-optimal policy $`p^\xi `$ which maximizes the mixture value $`V_\xi `$. One can show that not all environmental classes $``$ admit self-optimizing policies. One way to proceed is to search for and prove weaker properties than self-optimizingness \[Hut00\]. Here we follow a different approach: Obviously, the least we must demand from $``$ to have a chance of finding a self-optimizing policy is that there exists some self-optimizing policy $`\stackrel{~}{p}`$ at all. The main new result of this work is that this necessary condition is also sufficient for $`p^\xi `$ to be self-optimizing. No other properties need to be imposed on $``$. The other new result is that $`p^\xi `$ is always Pareto-optimal, with no conditions at all imposed on $``$.

### Contents:

Section 2 defines the model of agents acting in general probabilistic environments and defines the finite horizon value of a policy and the optimal value-maximizing policy. Furthermore, the mixture-distribution is introduced and the fundamental linearity and convexity properties of the mixture-values is stated. Section 3 defines and proves Pareto-optimality of $`p^\xi `$. The concept is refined to balanced Pareto-optimality, showing that a small increase of the value for some environments only leaves room for a small decrease in others. Section 4 shows that $`p^\xi `$ is self-optimizing if $``$ admits self-optimizing policies, and also gives the speed of convergence in the case of finite $``$. The finite horizon model has several disadvantages. For this reason Section 5 defines the discounted (infinite horizon) future value function, and the corresponding optimal value-maximizing policy. Pareto-optimality and self-optimizingness of $`p^\xi `$ are shown shown for this model. As an application we show in Section 6 that the class of ergodic mdp$`s`$ admits self-optimizing policies w.r.t. the undiscounted model and w.r.t. the discounted model if the effective horizon tends to infinity. Together with the results from the previous sections this shows that $`p^\xi `$ is self-optimizing for erdodic mdp$`s`$. Conclusions and outlook can be found in Section 7.

## 2 Rational Agents in Probabilistic Environments

### The agent model:

A very general framework for intelligent systems is that of rational agents \[RN95\]. In cycle $`k`$, an agent performs action $`y_k𝒴`$ (output) which results in a perception or observation $`x_k𝒳`$ (input), followed by cycle $`k+1`$ and so on. We assume that the action and perception spaces $`𝒳`$ and $`𝒴`$ are finite. We write $`p(x_{<k})=y_{1:k}`$ to denote the output $`y_{1:k}y_1\mathrm{}y_k`$ of the agents policy $`p`$ on input $`x_{<k}x_1\mathrm{}x_{k1}`$ and similarly $`q(y_{1:k})=x_{1:k}`$ for the environment $`q`$ in the case of deterministic environments. We call policy $`p`$ and environment $`q`$ behaving in this way chronological. Note that policy and environment are allowed to depend on the complete history. We do not make any mdp or pomdp assumption here, and we don’t talk about states of the environment, only about observations. In the more general case of a probabilistic environment, given the history $`yx_{<k}y_kyx_1\mathrm{}yx_{k1}y_ky_1x_1\mathrm{}y_{k1}x_{k1}y_k`$, the probability that the environment leads to perception $`x_k`$ in cycle $`k`$ is (by definition) $`\rho (yx_{<k}y\underset{¯}{x}_k)`$. The underlined argument $`\underset{¯}{x}_k`$ in $`\rho `$ is a random variable and the other non-underlined arguments $`yx_{<k}y_k`$ represent conditions.222The standard notation $`\rho (x_k|yx_{<k}y_k)`$ for conditional probabilities destroys the chronological order and would become quite confusing in later expressions. We call probability distributions like $`\rho `$ chronological. Since value optimizing policies can always be chosen deterministic, there is no real need to generalize the setting to probabilistic policies. Arbitrarily we formalize Sections 3 and 4 in terms of deterministic policies and Section 5 in terms of probabilistic policies.

### Value functions and optimal policies:

The goal of the agent is to maximize future rewards, which are provided by the environment through the inputs $`x_k`$. The inputs $`x_kx_k^{}r_k`$ are divided into a regular part $`x_k^{}`$ and some (possibly empty or delayed) reward $`r_k[0,r_{max}]`$.333In the reinforcement learning literature when dealing with (po)mdp$`s`$ the reward is usually considered to be a function of the environmental state. The zero-assumption analogue here is that the reward $`r_k`$ is some probabilistic function $`\rho ^{}`$ depending on the complete history. It is very convenient to integrate $`r_k`$ into $`x_k`$ and $`\rho ^{}`$ into $`\rho `$. We use the abbreviation

$`\rho (yx_{<k}y\underset{¯}{x}_{k:m})=\rho (yx_{<k}y\underset{¯}{x}_k)\rho (yx_{1:k}y\underset{¯}{x}_{k+1})\mathrm{}\rho (yx_{<m}y\underset{¯}{x}_m),`$ (1)
which is essentially Bayes rules, and $`\epsilon =yx_{<1}`$ for the empty string. The $`\rho `$-expected reward sum (value) of future cycles $`k`$ to $`m`$ with outputs $`y_{k:m}`$ generated by the agent’s policy $`p`$, the optimal policy $`p^\rho `$ which maximizes the value, its action $`y_k`$ and the corresponding value can formally be defined as follows.

###### Definition 1 (Value function and optimal policy)

We define the value of policy $`p`$ in environment $`\rho `$ given history $`yx_{<k}`$, or shorter, the $`\rho `$-value of $`p`$ given $`yx_{<k}`$, as

$$V_{km}^{p\rho }(yx_{<k}):=\underset{x_{k:m}}{}(r_k+\mathrm{}+r_m)\rho (yx_{<k}y\underset{¯}{x}_{k:m})_{|y_{1:m}=p(x_{<m})}.$$
(2)
$`m`$ is the lifespan or initial horizon of the agent. The $`\rho `$-optimal policy $`p^\rho `$ which maximizes the (total) value $`V_\rho ^p:=V_{1m}^{p\rho }(\epsilon )`$ is

$$p^\rho :=\mathrm{arg}\underset{p}{\mathrm{max}}V_\rho ^p,V_{km}^\rho (yx_{<k}):=V_{km}^{p^\rho \rho }(yx_{<k}).$$
(3)
Explicit expressions for the action $`y_k`$ in cycle $`k`$ of the $`\rho `$-optimal policy $`p^\rho `$ and their value $`V_{km}^\rho (yx_{<k})`$ are

$$y_k=\mathrm{arg}\underset{y_k}{\mathrm{max}}\underset{x_k}{}\underset{y_{k+1}}{\mathrm{max}}\underset{x_{k+1}}{}\mathrm{}\underset{y_m}{\mathrm{max}}\underset{x_m}{}(r_k+\mathrm{}+r_m)\rho (yx_{<k}y\underset{¯}{x}_{k:m}),$$
(4)
$$V_{km}^\rho (yx_{<k})=\underset{y_k}{\mathrm{max}}\underset{x_k}{}\underset{y_{k+1}}{\mathrm{max}}\underset{x_{k+1}}{}\mathrm{}\underset{y_m}{\mathrm{max}}\underset{x_m}{}(r_k+\mathrm{}+r_m)\rho (yx_{<k}y\underset{¯}{x}_{k:m}).$$
(5)
where $`yx_{<k}`$ is the actual history.

One can show \[Hut00\] that these definitions are consistent and correctly capture our intention. For instance, consider the expectimax expression (5): The best expected reward is obtained by averaging over possible perceptions $`x_i`$ and by maximizing over the possible actions $`y_i`$. This has to be done in chronological order $`y_kx_k\mathrm{}y_mx_m`$ to correctly incorporate the dependency of $`x_i`$ and $`y_i`$ on the history. Obviously

$$V_{km}^\rho (yx_{<k})V_{km}^{p\rho }(yx_{<k})p,\text{especially}V_\rho ^{}V_\rho ^pp.$$
(6)
### Known environment $`\mu `$:

Let us now make a change in conventions and assume that $`\mu `$ is the true environment in which the agent operates and that we know $`\mu `$ (like in Blackjack).444If the existence of true objective probabilities violates the philosophical attitude of the reader he may assume a deterministic environment $`\mu `$. Then, policy $`p^\mu `$ is optimal in the sense that no other policy for an agent leads to higher $`\mu `$-expected reward. This setting includes as special cases deterministic environments, Markov decision processes (mdp$`s`$), and even adversarial environments for special choices of $`\mu `$ \[Hut00\]. There is no principle problem in determining the optimal action $`y_k`$ as long as $`\mu `$ is known and computable and $`𝒳`$, $`𝒴`$ and $`m`$ are finite.

### The mixture distribution $`\xi `$:

Things drastically change if $`\mu `$ is unknown. For (parameterized) mdp$`s`$ with small state (parameter) space, suboptimal reinforcement learning algorithms may be used to learn the unknown $`\mu `$ \[KLM96, SB98, Ber95, KV86\]. In the Bayesian approach the true probability distribution $`\mu `$ is not learned directly, but is replaced by a Bayes-mixture $`\xi `$. Let us assume that we know that the true environment $`\mu `$ is contained in some known set $``$ of environments. For convenience we assume that $``$ is finite or countable. The Bayes-mixture $`\xi `$ is defined as

$$\xi (y\underset{¯}{x}_{1:m})=\underset{\nu }{}w_\nu \nu (y\underset{¯}{x}_{1:m})\text{with}\underset{\nu }{}w_\nu =1,w_\nu >0\nu $$
(7)
The weights $`w_\nu `$ may be interpreted as the prior degree of belief that the true environment is $`\nu `$. Then $`\xi (y\underset{¯}{x}_{1:m})`$ could be interpreted as the prior subjective belief probability in observing $`x_{1:m}`$, given actions $`y_{1:m}`$. It is, hence, natural to follow the policy $`p^\xi `$ which maximizes $`V_\xi ^p`$. If $`\mu `$ is the true environment the expected reward when following policy $`p^\xi `$ will be $`V_\mu ^{p^\xi }`$. The optimal (but infeasible) policy $`p^\mu `$ yields reward $`V_\mu ^{p^\mu }V_\mu ^{}`$. It is now of interest $`(a)`$ whether there are policies with uniformly larger value than $`V_\mu ^{p^\xi }`$ and $`(b)`$ how close $`V_\mu ^{p^\xi }`$ is to $`V_\mu ^{}`$. These are the main issues of the remainder of this work.

### A universal choice of $`\xi `$ and $``$:

One may also ask what the most general class $``$ and weights $`w_\nu `$ could be. Without any prior knowledge we should include all environments in $``$. In this generality this approach leads at best to negative results. More useful is the assumption that the environment possesses some structure, we just don’t know which. From a computational point of view we can only unravel effective structures which are describable by (semi)computable probability distributions. So we may include all (semi)computable (semi)distributions in $``$. Occam’s razor tells us to assign high prior belief to simple environments. Using Kolmogorov’s universal complexity measure $`K(\nu )`$ for environments $`\nu `$ one should set $`w_\nu 2^{K(\nu )}`$, where $`K(\nu )`$ is the length of the shortest program on a universal Turing machine computing $`\nu `$. The resulting policy $`p^\xi `$ has been developed and intensively discussed in \[Hut00\]. It is a unification of sequential decision theory \[Bel57, Ber95\] and Solomonoff’s celebrated universal induction scheme \[Sol78, LV97\]. In the following we consider generic $``$ and $`w_\nu `$. The following property of $`V_\rho `$ is crucial.

###### Theorem 1 (Linearity and convexity of $`V_\rho `$ in $`\rho `$)

$`V_\rho ^p`$ is a linear function in $`\rho `$ and $`V_\rho ^{}`$ is a convex function in $`\rho `$ in the sense that

$$V_\xi ^p=\underset{\nu }{}w_\nu V_\nu ^p\text{and}V_\xi ^{}\underset{\nu }{}w_\nu V_\nu ^{}\text{where}\xi (y\underset{¯}{x}_{1:m})=\underset{\nu }{}w_\nu \nu (y\underset{¯}{x}_{1:m})$$

Proof: Linearity is obvious from the definition of $`V_\rho ^p`$. Convexity follows from $`V_\xi ^{}V_\xi ^{p^\xi }=_\nu w_\nu V_\nu ^{p^\xi }_\nu w_\nu V_\nu ^{}`$, where the identity is definition (3), the equality uses linearity of $`V_\rho ^{p^\xi }`$ just proven, and the last inequality follows from the dominance (6) and non-negativity of the weights $`w_\nu `$. $``$

One loose interpretation of the convexity is that a mixture can never increase performance. In the remainder of this work $`\mu `$ denotes the true environment, $`\rho `$ any distribution, and $`\xi `$ the Bayes-mixture of distributions $`\nu `$.

## 3 Pareto Optimality of policy $`p^\xi `$

The total $`\mu `$-expected reward $`V_\mu ^{p^\xi }`$ of policy $`p^\xi `$ is of central interest in judging the performance of policy $`p^\xi `$. We know that there are policies (e.g. $`p^\mu `$) with higher $`\mu `$-value ($`V_\mu ^{}V_\mu ^{p^\xi }`$). In general, every policy based on an estimate $`\rho `$ of $`\mu `$ which is closer to $`\mu `$ than $`\xi `$ is, outperforms $`p^\xi `$ in environment $`\mu `$, simply because it is more taylored toward $`\mu `$. On the other hand, such a system probably performs worse than $`p^\xi `$ in other environments. Since we do not know $`\mu `$ in advance we may ask whether there exists a policy $`p`$ with better or equal performance than $`p^\xi `$ in all environments $`\nu `$ and a strictly better performance for one $`\nu `$. This would clearly render $`p^\xi `$ suboptimal. We show that there is no such $`p`$.

###### Theorem 2 (Pareto optimality)

Policy $`p^\xi `$ is Pareto-optimal in the sense that there is no other policy $`p`$ with $`V_\nu ^pV_\nu ^{p^\xi }`$ for all $`\nu `$ and strict inequality for at least one $`\nu `$.

Proof: We want to arrive at a contradiction by assuming that $`p^\xi `$ is not Pareto-optimal, i.e. by assuming the existence of a policy $`p`$ with $`V_\nu ^pV_\nu ^{p^\xi }`$ for all $`\nu `$ and strict inequality for at least one $`\nu `$:

$$V_\xi ^p=\underset{\nu }{}w_\nu V_\nu ^p>\underset{\nu }{}w_\nu V_\nu ^{p^\xi }=V_\xi ^{p^\xi }V_\xi ^{}V_\xi ^p$$

The two equalities follow from linearity of $`V_\rho `$ (Theorem 1). The strict inequality follows from the assumption and from $`w_\nu >0`$. The identity is just Definition 1(3). The last inequality follows from the fact that $`p^\xi `$ maximizes by definition the universal value (6). The contradiction $`V_\xi ^p>V_\xi ^p`$ proves Pareto-optimality of policy $`p^\xi `$.$``$

Pareto-optimality should be regarded as a necessary condition for an agent aiming to be optimal. From a practical point of view a significant increase of $`V`$ for many environments $`\nu `$ may be desirable even if this causes a small decrease of $`V`$ for a few other $`\nu `$. The impossibility of such a “balanced” improvement is a more demanding condition on $`p^\xi `$ than pure Pareto-optimality. The next theorem shows that $`p^\xi `$ is also balanced-Pareto-optimal in the following sense:

###### Theorem 3 (Balanced Pareto optimality)

$$\mathrm{\Delta }_\nu :=V_\nu ^{p^\xi }V_\nu ^{\stackrel{~}{p}},\mathrm{\Delta }:=\underset{\nu }{}w_\nu \mathrm{\Delta }_\nu \mathrm{\Delta }0.$$

This implies the following: Assume $`\stackrel{~}{p}`$ has lower value than $`p^\xi `$ on environments $``$ by a total weighted amount of $`\mathrm{\Delta }_{}:=_\lambda w_\lambda \mathrm{\Delta }_\lambda `$. Then $`\stackrel{~}{p}`$ can have higher value on $`\eta :=`$, but the improvement is bounded by $`\mathrm{\Delta }_{}:=|_\eta w_\eta \mathrm{\Delta }_\eta |\mathrm{\Delta }_{}`$. Especially $`|\mathrm{\Delta }_\eta |w_\eta ^1\mathrm{max}_\lambda \mathrm{\Delta }_\lambda `$.

This means that a weighted value increase $`\mathrm{\Delta }_{}`$ by using $`\stackrel{~}{p}`$ instead of $`p^\xi `$ is compensated by an at least as large weighted decrease $`\mathrm{\Delta }_{}`$ on other environments. If the decrease is small, the increase can also only be small. In the special case of only a single environment with decreased value $`\mathrm{\Delta }_\lambda `$, the increase is bound by $`\mathrm{\Delta }_\eta \frac{w_\lambda }{w_\eta }|\mathrm{\Delta }_\lambda |`$, i.e. a decrease by an amount $`\mathrm{\Delta }_\lambda `$ can only cause an increase by at most the same amount times a factor $`\frac{w_\lambda }{w_\eta }`$. For the choice of the weights $`w_\nu 2^{K(\nu )}`$, a decrease can only cause a smaller increase in simpler environments, but a scaled increase in more complex environments. Finally note that pure Pareto-optimality (Theorem 2) follows from balanced Pareto-optimality in the special case of no decrease $`\mathrm{\Delta }_{}0`$.

Proof: $`\mathrm{\Delta }0`$ follows from $`\mathrm{\Delta }=_\nu w_\nu [V_\nu ^{p^\xi }V_\nu ^{\stackrel{~}{p}}]=V_\xi ^{p^\xi }V_\xi ^{\stackrel{~}{p}}0`$, where we have used linearity of $`V_\rho `$ (Theorem 1) and dominance $`V_\xi ^{p^\xi }V_\xi ^p`$ (6). The remainder of Theorem 3 is obvious from $`0\mathrm{\Delta }=\mathrm{\Delta }_{}\mathrm{\Delta }_{}`$ and by bounding the weighted average $`\mathrm{\Delta }_\eta `$ by its maximum.$``$

## 4 Self-optimizing Policy $`p^\xi `$ w.r.t. Average Value

In the following we study under which circumstances555Here and elsewhere we interpret $`a_mb_m`$ as an abbreviation for $`a_mb_m0`$. $`lim_m\mathrm{}b_m`$ may not exist.

$$\frac{1}{m}V_{1m}^{p^\xi \nu }\frac{1}{m}V_{1m}^\nu \text{for}m\mathrm{}\text{for }\text{all}\nu .$$
(8)
The least we must demand from $``$ to have a chance that (8) is true is that there exists some policy $`\stackrel{~}{p}`$ at all with this property, i.e.

$$\stackrel{~}{p}:\frac{1}{m}V_{1m}^{\stackrel{~}{p}\nu }\frac{1}{m}V_{1m}^\nu \text{for}m\mathrm{}\text{for }\text{all}\nu .$$
(9)
Luckily, this necessary condition will also be sufficient. This is another (asymptotic) optimality property of policy $`p^\xi `$. If universal convergence in the sense of (9) is possible at all in a class of environments $``$, then policy $`p^\xi `$ converges in the sense of (8). We will call policies $`\stackrel{~}{p}`$ with a property like (9) self-optimizing \[KV86\]. The following two Lemmas pave the way for proving the convergence Theorem.

###### Lemma 1 (Value difference relation)

$$0V_\nu ^{}V_\nu ^{\stackrel{~}{p}}=:\mathrm{\Delta }_\nu 0V_\nu ^{}V_\nu ^{p^\xi }\frac{1}{w_\nu }\mathrm{\Delta }\text{with}\mathrm{\Delta }:=\underset{\nu }{}w_\nu \mathrm{\Delta }_\nu $$

Proof: The following sequence of inequalities proves the lemma:

$$0w_\nu [V_\nu ^{}V_\nu ^{p^\xi }]_\nu w_\nu [V_\nu ^{}V_\nu ^{p^\xi }]_\nu w_\nu [V_\nu ^{}V_\nu ^{\stackrel{~}{p}}]=_\nu w_\nu \mathrm{\Delta }_\nu \mathrm{\Delta }$$

In the first and second inequality we used $`w_\nu 0`$ and $`V_\nu ^{}V_\nu ^{p^\xi }0`$. The last inequality follows from $`_\nu w_\nu V_\nu ^{p^\xi }=V_\xi ^{p^\xi }V_\xi ^{}V_\xi ^{\stackrel{~}{p}}=_\nu w_\nu V_\nu ^{\stackrel{~}{p}}`$. $``$

We also need some results for averages of functions $`\delta _\nu (m)0`$ converging to zero.

###### Lemma 2 (Convergence of averages)

For $`\delta (m):=_\nu w_\nu \delta _\nu (m)`$ the following holds (we only need $`_\nu w_\nu 1`$):

$$\begin{array}{ccccccc}\hfill i)& \delta _\nu (m)f(m)\hfill & \nu \hfill & \text{implies}\hfill & \delta (m)f(m).\hfill & & \\ \hfill ii)& \delta _\nu (m)\stackrel{m\mathrm{}}{}0\hfill & \nu \hfill & \text{implies}\hfill & \delta (m)\stackrel{m\mathrm{}}{}0\hfill & \text{if}\hfill & 0\delta _\nu (m)c.\hfill \end{array}$$

Proof: $`(i)`$ immediately follows from $`\delta (m)=_\nu w_\nu \delta _\nu (m)_\nu w_\nu f(m)f(m)`$. For $`(ii)`$ we choose some order on $``$ and some $`\nu _0`$ large enough such that $`_{\nu \nu _0}w_\nu \frac{\epsilon }{c}`$. Using $`\delta _\nu (m)c`$ this implies

$$\underset{\nu \nu _0}{}w_\nu \delta _\nu (m)\underset{\nu \nu _0}{}w_\nu c\epsilon .$$

Furthermore, the assumption $`\delta _\nu (m)0`$ means that there is an $`m_{\nu \epsilon }`$ depending on $`\nu `$ and $`\epsilon `$ such that $`\delta _\nu (m)\epsilon `$ for all $`mm_{\nu \epsilon }`$. This implies

$$\underset{\nu \nu _0}{}w_\nu \delta _\nu (m)\underset{\nu \nu _0}{}w_\nu \epsilon \epsilon \text{for all}m\underset{\nu \nu _0}{\mathrm{max}}\{m_{\nu \epsilon }\}=:m_\epsilon .$$

$`m_\epsilon <\mathrm{}`$, since the maximum is over a finite set. Together we have

$$\delta (m)\underset{\nu }{}w_\nu \delta _\nu (m)2\epsilon \text{for}mm_\epsilon \delta (m)0\text{for}m\mathrm{}$$

since $`\epsilon `$ was arbitrary and $`\delta (m)0`$. $``$

###### Theorem 4 (Self-optimizing policy $`p^\xi `$ w.r.t. average value)

There exists a sequence of policies $`\stackrel{~}{p}_m`$, $`m=1,2,3,\mathrm{}`$ with value within $`\mathrm{\Delta }(m)`$ to optimum for all environments $`\nu `$, then, save for a constant factor, this also holds for the sequence of universal policies $`p_m^\xi `$, i.e.

$$i)\text{If}\stackrel{~}{p}_m\nu :V_{1m}^\nu V_{1m}^{\stackrel{~}{p}_m\nu }\mathrm{\Delta }(m)V_{1m}^\mu V_{1m}^{p_m^\xi \mu }\frac{1}{w_\mu }\mathrm{\Delta }(m).$$

If there exists a sequence of self-optimizing policies $`\stackrel{~}{p}_m`$ in the sense that their expected average reward $`\frac{1}{m}V_{1m}^{\stackrel{~}{p}_m\nu }`$ converges to the optimal average $`\frac{1}{m}V_{1m}^\nu `$ for all environments $`\nu `$, then this also holds for the sequence of universal policies $`p_m^\xi `$, i.e.

$$ii)\text{If}\stackrel{~}{p}_m\nu :\frac{1}{m}V_{1m}^{\stackrel{~}{p}_m\nu }\stackrel{m\mathrm{}}{}\frac{1}{m}V_{1m}^\nu \frac{1}{m}V_{1m}^{p_m^\xi \mu }\stackrel{m\mathrm{}}{}\frac{1}{m}V_{1m}^\mu .\text{ }$$

The beauty of this theorem is that if universal convergence in the sense of (9) is possible at all in a class of environments $``$, then policy $`p^\xi `$ converges (in the sense of (8)). The necessary condition of convergence is also sufficient. The unattractive point is that this is not an asymptotic convergence statement for $`V_{km}^{p^\xi \mu }`$ of a single policy $`p^\xi `$ for $`k\mathrm{}`$ for some fixed $`m`$, and in fact no such theorem could be true, since always $`km`$. The theorem merely says that under the stated conditions the average value of $`p_m^\xi `$ can be arbitrarily close to optimum for sufficiently large (pre-chosen) horizon $`m`$. This weakness will be resolved in the next subsection.

Proof: $`(i)`$ $`\mathrm{\Delta }_\nu (m)=f(m)`$ implies $`\mathrm{\Delta }(m)=f(m)`$ by Lemma 2$`(i)`$. Inserting this in Lemma 1 proves Theorem 4$`(i)`$ (recovering the $`m`$ dependence and finally renaming $`f\mathrm{\Delta }`$).

$`(ii)`$ We define $`\delta _\nu (m):=\frac{1}{m}\mathrm{\Delta }_\nu (m)=\frac{1}{m}[V_\nu ^{}V_\nu ^{\stackrel{~}{p}}]`$. Since we assumed bounded rewards $`0rr_{max}`$ we have

$$V_\nu ^{}mr_{max}\text{and}V_\nu ^{\stackrel{~}{p}}0\mathrm{\Delta }_\nu mr_{max}0\delta _\nu (m)c:=r_{max}.$$

The premise in Theorem 4$`(ii)`$ is that $`\delta _\nu (m)=\frac{1}{m}[V_{1m}^\nu V_{1m}^{\stackrel{~}{p}\nu }]0`$ which implies

$$0\frac{1}{m}[V_{1m}^\nu V_{1m}^{p^\xi \nu }]\frac{1}{w_\nu }\frac{\mathrm{\Delta }(m)}{m}=\frac{1}{w_\nu }\delta (m)0.$$

The inequalities follow from Lemma 1 and convergence to zero from Lemma 2$`(ii)`$. This proves Theorem 4$`(ii)`$. $``$.

In Section 6 we show that a converging $`\stackrel{~}{p}`$ exists for ergodic mdp$`s`$, and hence $`p^\xi `$ converges in this environmental class too (in the sense of Theorem 4).

## 5 Discounted Future Value Function

We now shift our focus from the total value $`V_{1m}`$, $`m\mathrm{}`$ to the future value (value-to-go) $`V_{k\mathrm{?}}`$, $`k\mathrm{}`$. The main reason is that we want to get rid of the horizon parameter $`m`$. In the last subsection we have shown a convergence theorem for $`m\mathrm{}`$, but a specific policy $`p^\xi `$ is defined for all times relative to a fixed horizon $`m`$. Current time $`k`$ is moving, but $`m`$ is fixed666A dynamic horizon like $`mm_k=k^2`$ can lead to policies with very poor performance \[Hut00, Ch.4\].. Actually, to use $`k\mathrm{}`$ arguments we have to get rid of $`m`$, since $`km`$. This is the reason for the question mark in $`V_{k\mathrm{?}}`$ above.

We eliminate the horizon by discounting the rewards $`r_k\gamma _kr_k`$ with $`_{i=1}^{\mathrm{}}\gamma _i<\mathrm{}`$ and letting $`m\mathrm{}`$. The analogue of $`m`$ is now an effective horizon $`h_k^{eff}`$ which may be defined by $`_{i=k}^{k+h_k^{eff}}\gamma _k_{i=k+h_k^{eff}}^{\mathrm{}}\gamma _k`$. See \[Hut00, Ch.4\] for a detailed discussion of the horizon problem. Furthermore, we renormalize $`V_k\mathrm{}`$ by $`_{i=k}^{\mathrm{}}\gamma _i`$ and denote it by $`V_{k\gamma }`$. It can be interpreted as a future expected weighted-average reward. Furthermore we extend the definition to probabilistic policies $`\pi `$.

###### Definition 2 (Discounted value function and optimal policy)

We define the $`\gamma `$ discounted weighted-average future value of (probabilistic) policy $`\pi `$ in environment $`\rho `$ given history $`yx_{<k}`$, or shorter, the $`\rho `$-value of $`\pi `$ given $`yx_{<k}`$, as

$$V_{k\gamma }^{\pi \rho }(yx_{<k}):=\frac{1}{\mathrm{\Gamma }_k}\underset{m\mathrm{}}{lim}\underset{yx_{k:m}}{}(\gamma _kr_k+\mathrm{}+\gamma _mr_m)\rho (yx_{<k}y\underset{¯}{x}_{k:m})\pi (yx_{<k}\underset{¯}{y}x_{k:m})$$

with $`\mathrm{\Gamma }_k:=_{i=k}^{\mathrm{}}\gamma _i`$. The policy $`p^\rho `$ is defined as to maximize the future value $`V_{k\gamma }^{\pi \rho }`$:

$$p^\rho :=\mathrm{arg}\underset{\pi }{\mathrm{max}}V_{k\gamma }^{\pi \rho },V_{k\gamma }^\rho :=V_{k\gamma }^{p^\rho \rho }=\underset{\pi }{\mathrm{max}}V_{k\gamma }^{\pi \rho }V_{k\gamma }^{\pi \rho }\pi .$$

Remarks:

* $`\pi (yx_{<k}\underset{¯}{y}x_{k:m})`$ is actually independent of $`x_m`$, since $`\pi `$ is chronological.
* Normalization of $`V_{k\gamma }`$ by $`\mathrm{\Gamma }_k`$ does not affect the policy $`p^\rho `$.
* The definition of $`p^\rho `$ is independent of $`k`$.
* Without normalization by $`\mathrm{\Gamma }_k`$ the future values would converge to zero for $`k\mathrm{}`$ in every environment for every policy.
* For an mdp environment, a stationary policy, and geometric discounting $`\gamma _k\gamma ^k`$, the future value is independent of $`k`$ and reduces to the well-known mdp value function.
* There is always a deterministic optimizing policy $`p^\rho `$ (which we use).
* For a deterministic policy there is exactly one $`y_{k:m}`$ for each $`x_{k:m}`$ with $`\pi 0`$. The sum over $`y_{k:m}`$ drops in this case.
* An iterative representation as in Definition 1 is possible.
* Setting $`\gamma _k=1`$ for $`km`$ and $`\gamma _k=0`$ for $`k>m`$ gives back the undiscounted model (1) with $`V_{1\gamma }^{p\rho }=\frac{1}{m}V_{1m}^{p\rho }`$.
* $`V_{k\gamma }`$ (and $`w_k^\nu `$ defined below) depend on the realized history $`yx_{<k}`$.

Similarly to the previous sections one can prove the following properties:

###### Theorem 5 (Linearity and convexity of $`V_\rho `$ in $`\rho `$)

$`V_{k\gamma }^{\pi \rho }`$ is a linear function in $`\rho `$ and $`V_{k\gamma }^\rho `$ is a convex function in $`\rho `$ in the sense that

$$V_{k\gamma }^{\pi \xi }=\underset{\nu }{}w_k^\nu V_{k\gamma }^{\pi \nu }\text{and}V_{k\gamma }^\xi \underset{\nu }{}w_k^\nu V_{k\gamma }^\nu $$

$$\text{where}\xi (yx_{<k}y\underset{¯}{x}_{k:m})=\underset{\nu }{}w_k^\nu \nu (yx_{<k}y\underset{¯}{x}_{k:m})\text{with}w_k^\nu :=w_\nu \frac{\nu (y\underset{¯}{x}_{<k})}{\xi (y\underset{¯}{x}_{<k})}$$

The conditional representation of $`\xi `$ can be proven by dividing the definition (7) of $`\xi (y\underset{¯}{x}_{1:m})`$ by $`\xi (y\underset{¯}{x}_{<k})`$ and by using Bayes rules (1). The posterior weight $`w_k^\nu `$ may be interpreted as the posterior belief in $`\nu `$ and is related to learning aspects of policy $`p^\xi `$.

###### Theorem 6 (Pareto optimality)

For every $`k`$ and history $`yx_{<k}`$ the following holds: $`p^\xi `$ is Pareto-optimal in the sense that there is no other policy $`\pi `$ with $`V_{k\gamma }^{\pi \nu }V_{k\gamma }^{p^\xi \nu }`$ for all $`\nu `$ and strict inequality for at least one $`\nu `$.

###### Lemma 3 (Value difference relation)

$$0V_{k\gamma }^\nu V_{k\gamma }^{\stackrel{~}{\pi }_k\nu }=:\mathrm{\Delta }_k^\nu 0V_{k\gamma }^\nu V_{k\gamma }^{p^\xi \nu }\frac{1}{w_k^\nu }\mathrm{\Delta }_k\text{with}\mathrm{\Delta }_k:=\underset{\nu }{}w_k^\nu \mathrm{\Delta }_k^\nu $$

The proof of Theorem 6 and Lemma 3 follows the same steps as for Theorem 2 and Lemma 1 with appropriate replacements. The proof of the analogue of the convergence Theorem 4 involves one additional step. We abbreviate “with $`\mu `$ probability 1” by w.$`\mu `$.p.1.

###### Theorem 7 (Self-optimizing policy $`p^\xi `$ w.r.t. discounted value)

For any $``$, if there exists a sequence of self-optimizing policies $`\stackrel{~}{\pi }_k`$ $`k=1,2,3,\mathrm{}`$ in the sense that their expected weighted-average reward $`V_{k\gamma }^{\stackrel{~}{\pi }_k\nu }`$ converges for $`k\mathrm{}`$ with $`\nu `$-probability one to the optimal value $`V_{k\gamma }^\nu `$ for all environments $`\nu `$, then this also holds for the universal policy $`p^\xi `$ in the true $`\mu `$-environment, i.e.

$$\text{If }\stackrel{~}{\pi }_k\nu :V_{k\gamma }^{\stackrel{~}{\pi }_k\nu }\stackrel{k\mathrm{}}{}V_{k\gamma }^\nu \text{w.}\nu \text{.p.1}V_{k\gamma }^{p^\xi \mu }\stackrel{k\mathrm{}}{}V_{k\gamma }^\mu \text{w.}\mu \text{.p.1}.$$

The probability qualifier refers to the historic perceptions $`x_{<k}`$. The historic actions $`y_{<k}`$ are arbitrary.

The conclusion is valid for action histories $`y_{<k}`$ if the condition is satisfied for this action history. Since we usually need the conclusion for the $`p^\xi `$-action history, which is hard to characterize, we usually need to prove the condition for all action histories. Theorem 7 is a powerful result: An (inconsistent) sequence of probabilistic policies $`\stackrel{~}{\pi }_k`$ suffices to prove the existence of a (consistent) deterministic policy $`p^\xi `$. A result similar to Theorem 4$`(i)`$ also holds for the discounted case, roughly saying that $`V^{\stackrel{~}{\pi }}V^{}=O(\mathrm{\Delta }(k))`$ implies $`V^{p^\xi }V^{}=\frac{1}{\epsilon }O(\mathrm{\Delta }(k))`$ with $`\mu `$ probability $`1\epsilon `$ for finite $``$.

Proof: We define $`\delta _\nu (k):=\mathrm{\Delta }_k^\nu =V_{k\gamma }^\nu V_{k\gamma }^{\stackrel{~}{\pi }\nu }`$. Since we assumed bounded rewards $`0rr_{max}`$ and $`V_{k\gamma }^\nu `$ is a weighted average of rewards we have

$$V_{k\gamma }^\mu r_{max}\text{and}V_{k\gamma }^{\stackrel{~}{\pi }\mu }00\delta _\nu (k)=\mathrm{\Delta }_k^\nu c:=r_{max}.$$

The following inequalities follow from Lemma 3:

$$0V_{k\gamma }^\mu V_{k\gamma }^{p^\xi \mu }\frac{1}{w_k^\mu }\mathrm{\Delta }_k=\frac{1}{w_k^\mu }\delta (k)\stackrel{\mathrm{?}}{}0$$
(10)
The premise in Theorem 7 is that $`\delta _\nu (k)=V_{k\gamma }^\nu V_{k\gamma }^{\stackrel{~}{\pi }\nu }0`$ for $`k\mathrm{}`$ which implies $`\delta (k)0`$ (w.$`\mu `$.p.1) by Lemma 2$`(ii)`$. What is new and what remains to be shown is that $`w_k^\mu `$ is bounded from below in order to have convergence of (10) to zero. We show that $`z_{k1}:=\frac{w_\mu }{w_k^\mu }=\frac{\xi (y\underset{¯}{x}_{<k})}{\mu (y\underset{¯}{x}_{<k})}0`$ converges to a finite value, which completes the proof. Let $`𝐄`$ denote the $`\mu `$ expectation. Then

$$𝐄[z_k|x_{<k}]=\underset{x_k}{\overset{}{}}\mu (yx_{<k}y\underset{¯}{x}_k)\frac{\xi (y\underset{¯}{x}_{1:k})}{\mu (y\underset{¯}{x}_{1:k})}=\frac{\underset{x_k}{\overset{}{}}\xi (yx_{<k}y\underset{¯}{x}_k)\xi (y\underset{¯}{x}_{<k})}{\mu (y\underset{¯}{x}_{<k})}\frac{\xi (y\underset{¯}{x}_{<k})}{\mu (y\underset{¯}{x}_{<k})}=z_{k1}$$

$`_{x_k}^{}`$ runs over all $`x_k`$ with $`\mu (y\underset{¯}{x}_{1:k})0`$. The first equality holds w.$`\mu `$.p.1. In the second equality we have used Bayes rule twice. $`𝐄[z_k|x_{<k}]z_{k1}`$ shows that $`z_k`$ is a semi-martingale. Since $`z_k`$ is non-positive, \[Doo53, Th.$`4.1s(i)`$,p324\] implies that $`z_k`$ converges for $`k\mathrm{}`$ to a finite value w.$`\mu `$.p.1. $``$

## 6 Markov Decision Processes

From all possible environments, Markov (decision) processes are probably the most intensively studied ones. To give an example, we apply Theorems 4 and 7 to ergodic Markov decision processes, but we will be very brief.

###### Definition 3 (Ergodic Markov Decision Processes)

We call $`\mu `$ a (stationary) Markov Decision Process (mdp) if the probability of observing $`x_k𝒳`$, given history $`yx_{<k}y_k`$ does only depend on the last action $`y_k𝒴`$ and the last observation $`x_{k1}`$, i.e. if $`\mu (yx_{<k}y_k\underset{¯}{x}_k)=\mu (yx_{k1}\underset{¯}{x}_k)`$. In this case $`x_k`$ is called a state, $`𝒳`$ the state space, and $`\mu (yx_{k1}\underset{¯}{x}_k)`$ the transition matrix. An mdp $`\mu `$ is called ergodic if there exists a policy under which every state is visited infinitely often with probability 1. Let $`_{MDP}`$ be the set of mdp$`s`$ and $`_{MDP1}`$ be the set of ergodic mdp$`s`$. If an mdp $`\mu (yx_{k1}\underset{¯}{x}_k)`$ is independent of the action $`y_{k1}`$ it is a Markov process, if it is independent of the last observation $`x_{k1}`$ it is an i.i.d. process.

Stationary mdp$`s`$ $`\mu `$ have stationary optimal policies $`p^\mu `$ mapping the same state / observation $`x_t`$ always to the same action $`y_t`$. On the other hand a mixture $`\xi `$ of mdp$`s`$ is itself not an mdp, i.e. $`\xi _{MDP}`$, which implies that $`p^\xi `$ is, in general, not a stationary policy. The definition of ergodicity given here is least demanding, since it only demands on the existence of a single policy under which the Markov process is ergodic. Often, stronger assumptions, e.g. that every policy is ergodic or that a stationary distribution exists, are made. We now show that there are self-optimizing policies for the class of ergodic mdp$`s`$ in the following sense.

###### Theorem 8 (Self-optimizing policies for ergodic mdp$`s`$)

There exist self-optimizing policies $`\stackrel{~}{p}_m`$ for the class of ergodic mdp$`s`$ in the sense that

$$i)\stackrel{~}{p}_m\nu _{MDP1}:\frac{1}{m}V_{1m}^\nu \frac{1}{m}V_{1m}^{\stackrel{~}{p}_m\nu }c_\nu m^{1/3}\stackrel{m\mathrm{}}{}0,\text{ }$$

where $`c_\nu `$ are some constants. In the discounted case, if the discount sequence $`\gamma _k`$ has unbounded effective horizon $`h_k^{eff}\stackrel{k\mathrm{}}{}\mathrm{}`$, then there exist self-optimizing policies $`\stackrel{~}{\pi }_k`$ for the class of ergodic mdp$`s`$ in the sense that

$$ii)\stackrel{~}{\pi }_k\nu _{MDP1}:V_{k\gamma }^{\stackrel{~}{\pi }_k\nu }\stackrel{k\mathrm{}}{}V_{k\gamma }^\nu \text{if }\frac{\gamma _{k+1}}{\gamma _k}1.\text{ }$$

There is much literature on constructing and analyzing self-optimizing learning algorithms in mdp environments. The assumptions on the structure of the mdp$`s`$ vary, all include some form of ergodicity, often stronger than Definition 3, demanding that the Markov process is ergodic under every policy. See, for instance, \[KV86, Ber95\]. We will only briefly outline one algorithm satisfying Theorem 8 without trying to optimize performance.

Proof idea: For $`(i)`$ one can choose a policy $`\stackrel{~}{p}_m`$ which performs (uniformly) random actions in cycles $`1\mathrm{}k_01`$ with $`1k_0m`$ and which follows thereafter the optimal policy based on an estimate of the transition matrix $`T_{ss^{}}^a\nu (as\underset{¯}{s}^{})`$ from the initial $`k_01`$ cycles. The existence of an ergodic policy implies that for every pair of states $`s_{start},s𝒳`$ there is a sequence of actions and transitions of length at most $`|𝒳|1`$ such that state $`s`$ is reached from state $`s_{start}`$. The probability that the “right” transition occurs is at least $`T_{min}`$ with $`T_{min}`$ being the smallest non-zero transition probability in $`T`$. The probability that a random action is the “right” action is at least $`|𝒴|^1`$. So the probability of reaching a state $`s`$ in $`|𝒳|1`$ cycles via a random policy is at least $`(T_{min}/|𝒴|)^{|𝒳|1}`$. In state $`s`$ action $`a`$ is taken with probability $`|𝒴|^1`$ and leads to state $`s^{}`$ with probability $`T_{ss^{}}^aT_{min}`$. Hence, the expected number of transitions $`s\stackrel{a}{}s^{}`$ to occur in the first $`k_0`$ cycles is $`\frac{k_0}{|𝒳|}(T_{min}/|𝒴|)^{|𝒳|}k_0`$.777For $`T_{ss^{}}^a=0`$ the estimate $`\widehat{T}_{ss^{}}^a=0`$ is exact. The accuracy of the frequency estimate $`\widehat{T}_{ss^{}}^a`$ of $`T_{ss^{}}^a`$ hence is $`k_0^{1/2}`$. Similar mdp$`s`$ lead to “similar” optimal policies, which lead to similar values. More precisely, one can show that $`\widehat{T}Tk_0^{1/2}`$ implies the same accuracy in the average value, i.e. $`|\frac{1}{m}V_{k_0m}^{\stackrel{~}{p}_m\nu }\frac{1}{m}V_{k_0m}^\nu |k_0^{1/2}`$, where $`\stackrel{~}{p}_m`$ is the optimal policy based on $`\widehat{T}`$ and $``$ is the optimal policy based on $`T(=\nu )`$. Since $`\frac{1}{m}V_{1k_0}\frac{k_0}{m}`$, $`(i)`$ follows (with probability 1) by setting $`k_0m^{2/3}`$. The policy $`\stackrel{~}{p}_m`$ can be derandomized, showing $`(i)`$ for sure.

The discounted case $`(ii)`$ can be proven similarly. The history $`yx_{<k}`$ is simply ignored and the analogue to $`m\mathrm{}`$ is $`h_k^{eff}\mathrm{}`$ for $`k\mathrm{}`$, which is ensured by $`\frac{\gamma _{k+1}}{\gamma _k}\mathrm{}`$. Let $`\stackrel{~}{\pi }_k`$ be the policy which performs (uniformly) random actions in cycles $`k\mathrm{}k_01`$ with $`kk_0h_k^{eff}`$ and which follows thereafter the optimal policy888For non-geometric discounts as here, optimal policies are, in general, not stationary. based on an estimate $`\widehat{T}`$ of the transition matrix $`T`$ from cycles $`k\mathrm{}k_01`$. The existence of an ergodic policy, again, ensures that the expected number of transitions $`s\stackrel{a}{}s^{}`$ occurring in cycles $`k\mathrm{}k_01`$ is proportional to $`\mathrm{\Delta }:=k_0k`$. The accuracy of the frequency estimate $`\widehat{T}`$ of $`T`$ is $`\mathrm{\Delta }^{1/2}`$ which implies

$$V_{k_0\gamma }^{\stackrel{~}{\pi }_k\nu }V_{k_0\gamma }^\nu \text{for}\mathrm{\Delta }=k_0k\mathrm{},$$
(11)
where $`\stackrel{~}{\pi }_k`$ is the optimal policy based on $`\widehat{T}`$ and $``$ is the optimal policy based on $`T(=\nu )`$. It remains to show that the achieved reward in the random phase $`k\mathrm{}k_01`$ gives a negligible contribution to $`V_{k\gamma }`$. The following implications for $`k\mathrm{}`$ are easy to show:

$$\frac{\gamma _{k+1}}{\gamma _k}1\frac{\gamma _{k+\mathrm{\Delta }}}{\gamma _k}1\frac{\mathrm{\Gamma }_{k+\mathrm{\Delta }}}{\mathrm{\Gamma }_k}1\frac{1}{\mathrm{\Gamma }_k}\underset{i=k}{\overset{k_01}{}}\gamma _ir_i\frac{r_{max}}{\mathrm{\Gamma }_k}[\mathrm{\Gamma }_{k+\mathrm{\Delta }}\mathrm{\Gamma }_k]0.$$

Since convergence to zero is true for all fixed finite $`\mathrm{\Delta }`$ it is also true for sufficiently slowly increasing $`\mathrm{\Delta }(k)\mathrm{}`$. This shows that the contribution of the first $`\mathrm{\Delta }`$ rewards $`r_k+\mathrm{}+r_{k_01}`$ to $`V_{k\gamma }`$ is negligible. Together with (11) this shows $`V_{k\gamma }^{\stackrel{~}{\pi }_k\nu }V_{k\gamma }^\nu `$ for $`k_0:=k+\mathrm{\Delta }(k)`$.$``$

The conditions $`\mathrm{\Gamma }_k<\mathrm{}`$ and $`\frac{\gamma _{k+1}}{\gamma _k}1`$ on the discount sequence are, for instance, satisfied for $`\gamma _k=1/k^2`$, so the Theorem is not vacuous. The popular geometric discount $`\gamma _k=\gamma ^k`$ fails the latter condition; it has finite effective horizon. \[Hut00\] gives a detailed account on discount and horizon issues, and motivates $`h_k^{eff}\mathrm{}`$ philosophically.

Together with Theorems 4 and 7, Theorem 8 immediately implies that policy $`p^\xi `$ is self-optimizing for the class of to ergodic mdp$`s`$.

###### Corollary 1 (Policy $`p^\xi `$ is self-optimizing for ergodic mdp$`s`$)

If $``$ is a finite or countable class of ergodic mdp$`s`$, and $`\xi ():=_\nu w_\nu \nu ()`$, then policies $`p_m^\xi `$ maximizing $`V_{1m}^{p\xi }`$ and $`p^\xi `$ maximizing $`V_{k\gamma }^{\pi \xi }`$ are self-optimizing in the sense that

$$\nu :\frac{1}{m}V_{1m}^{p_m^\xi \nu }\stackrel{m\mathrm{}}{}\frac{1}{m}V_{1m}^\nu \text{and}V_{k\gamma }^{p^\xi \nu }\stackrel{k\mathrm{}}{}V_{k\gamma }^\nu \text{if }\frac{\gamma _{k+1}}{\gamma _k}1.$$

If $``$ is finite, then the speed of the first convergence is at least $`O(m^{1/3})`$.

## 7 Conclusions

### Summary:

We studied agents acting in general probabilistic environments with reinforcement feedback. We only assumed that the true environment $`\mu `$ belongs to a known class of environments $``$, but is otherwise unknown. We showed that the Bayes-optimal policy $`p^\xi `$ based on the Bayes-mixture $`\xi =_\nu w_\nu \nu `$ is Pareto-optimal and self-optimizing if $``$ admits self-optimizing policies. The class of ergodic mdp$`s`$ admitted self-optimizing policies w.r.t. the average value and w.r.t. the discounted value if the effective horizon grew indefinitely.

### Continuous classes $``$:

There are uncountably many (ergodic) mdp$`s`$. Since we have restricted our development to countable classes $``$ we had to give the Corollary for a countable subset of $`_{MDP1}`$. We may choose $``$ as the set of all ergodic mdp$`s`$ with rational (or computable) transition probabilities. In this case $``$ is a dense subset of $`_{MDP1}`$ which is, from a practical point of view, sufficiently rich. On the other hand, it is possible to extend the theory to continuously parameterized families of environments $`\mu _\theta `$ and $`\xi =w_\theta \mu _\theta 𝑑\theta `$. Under some mild (differentiability and existence) conditions, most results of this work remain valid in some form, especially Corollary 1 for all ergodic mdp$`s`$.

### Bayesian self-optimizing policy:

Policy $`p^\xi `$ with unbounded effective horizon for ergodic mdp$`s`$ is the first purely Bayesian self-optimizing consistent policy for ergodic mdp$`s`$. The policies of all previous approaches were either hand crafted, like the ones in the proof of Theorem 8, or were Bayesian with a pre-chosen horizon $`m`$, or with geometric discounting $`\gamma `$ with finite effective horizon (which does not allow self-optimizing policies) \[KV86, Ber95\]. The combined conditions $`\mathrm{\Gamma }_k<\mathrm{}`$ and $`\frac{\gamma _{k+1}}{\gamma _k}1`$ allow a consistent self-optimizing Bayes-optimal policy based on mixtures.

### Bandits:

Bandits are a special subclass of ergodic mdp$`s`$. In a two-armed bandit problem you pull repeatedly one of two levers resulting in a gain of A$1 with probability $`p_i`$ for arm number $`i`$. The game can be described as an mdp with parameters $`p_i`$. If the $`p_i`$ are unknown, Corollary 1 shows that policy $`p^\xi `$ yields asymptotically optimal payoff. The discounted unbounded horizon approach and result is, to the best of our knowledge, even new when restricted to Bandits.

### Other environmental classes:

Bandits, i.i.d. processes, classification tasks, and many more are all special (degenerate) cases of ergodic mdp$`s`$, for which Corollary 1 shows that $`p^\xi `$ is self-optimizing. But the existence of self-optimizing policies is not limited to (subclasses of ergodic) mdp$`s`$. Certain classes of pomdp$`s`$, $`k^{th}`$ order ergodic mdp$`s`$, factorizable environments, repeated games, and prediction problems are not mdp$`s`$, but nevertheless admit self-optimizing policies (to be shown elsewhere), and hence the corresponding Bayes-optimal mixture policy $`p^\xi `$ is self-optimizing by Theorems 4 and 7.

### Outlook:

Future research could be the derivation of non-asymptotic bounds, possibly along the lines of \[Hut01\]. To get good bounds one may have to exploit extra properties of the environments, like the mixing rate of mdp$`s`$ \[KS98\]. Another possibility is to search for other performance criteria along the lines of \[Hut00, Ch.6\], especially for the universal prior \[Sol78\] and for the Speed prior \[Sch02\]. Finally, instead of convergence of the expected reward sum, studying convergence with high probability of the actual reward sum would be interesting.