text:
# Non-Gaussian equilibrium in a long-range Hamiltonian system

## Abstract

We study the dynamics of a system of N classical spins with infinite-range interaction. We show that, if the thermodynamic limit is taken before the infinite-time limit, the system does not relax to the Boltzmann-Gibbs equilibrium, but exhibits different equilibrium properties, characterized by stable non-Gaussian velocity distributions, Lévy walks and dynamical correlation in phase-space.

PACS numbers: 05.50.+q, 05.70.Fh, 64.60.Fr

Though not always clearly stated, standard equilibrium thermodynamics  is valid only for sufficiently short-range interactions. This is not the case, for example, for gravitational or unscreened Coulombian fields, or for systems with long-range microscopic memory and fractal structures in phase space. The increasing experimental evidence of dynamics and thermodynamics anomalies in turbulent plasmas  and fluids  , astrophysical systems , nuclei  and atomic clusters, granular media, glasses and complex systems found in the last years, provide further motivation for a generalization of thermodynamics.

In this paper we consider a simple model of classical spins with infinite range interactions, and we show that, if the thermodynamic limit is performed before the infinite time limit, the system does not relax to the Boltzmann-Gibbs (BG) equilibrium, but exhibits different equilibrium properties characterized by non-Gaussian velocity distributions, Lévy walks and dynamical correlation in phase-space, and the validity of the zeroth principle of thermodynamics. Our results show some consistency with the predictions of a generalized non-extensive thermodynamics recently proposed. The Hamiltonian Mean Field (HMF) model describes a system of N planar classical spins interacting through an infinite-range potential. The Hamiltonian can be written as:

$$H=K+V=\underset{i=1}{\overset{N}{}}\frac{p_{i}^{}{}_{}{}^{2}}{2}+\frac{1}{2N}\underset{i,j=1}{\overset{N}{}}[1cos(\theta _i\theta _j)],$$
(1)
where $`\theta _i`$ is the $`ith`$ angle and $`p_i`$ the conjugate variable representing the angular momentum (or the rotational velocity since unit mass is assumed). The interaction is the same as in the ferromagnetic X-Y model , though the summation is extended to all couples of spins and not restricted to first neighbors. Following tradition, the coupling constant in the potential is divided by N. This makes H only formally extensive ($`VN`$ when $`N\mathrm{}`$), since the energy remains non-additive, i.e. the system cannot be trivially divided in two independent sub-systems. The canonical analytical solution of the model predicts a second-order phase transition from a low-energy ferromagnetic phase with magnetization $`M1`$ (M is the modulus of $`𝐌=\frac{1}{N}_{i=1}^N𝐦_i`$ , where $`𝐦_i=[cos(\theta _i),sin(\theta _i)]`$, to a high-energy one where the spins are homogeneously oriented on the unit circle and $`M0`$. The caloric curve, i.e. the dependence of the energy density $`U=E/N`$ on the temperature $`T`$, is given by $`U=\frac{T}{2}+\frac{1}{2}(1M^2)`$ and shown in Fig.1(a). The critical point is at energy density $`U_c=0.75`$ corresponding to a critical temperature $`T_c=0.5`$ . The dynamical behavior of HMF can be investigated in the microcanonical ensemble by starting the system with water bag initial conditions (WBIC), i.e. $`\theta _i=0`$ for all $`i`$ ($`M=1`$) and velocities uniformly distributed, and integrating numerically the equations of motion . As shown in Fig.1(a), microcanonical simulations are in general in good agreement with the canonical ensemble, except for a region below $`U_c`$, where it has also been found a dynamics characterized by Lévy walks, anomalous diffusion  and a negative specific heat. Ensemble inequivalence and negative specific heat have also been found in self-gravitating systems , nuclei and atomic clusters , though in the present model such anomalies emerge as dynamical features . In order to understand better this disagreement we focus on a particular energy value, namely $`U=0.69`$, and we follow the time evolution of temperature, magnetization, and velocity distributions.

In Fig.1(b) we report the time evolution of $`2<K>/N`$, a quantity that, evaluated at equilibrium, is expected to coincide with the temperature ($`<>`$ denotes time averages). The system is started with WBIC and rapidly reaches a metastable or quasi-stationary state (QSS) which does not coincide with the canonical prediction. In fact, after a short transient time, $`2<K>/N`$ shows a plateau corresponding to a N-dependent temperature $`T_{QSS}(N)`$ (and $`M_{QSS}0`$) lower than the canonical temperature. This metastable state needs a long time to relax to the canonical equilibrium state with temperature $`T_{can}=0.476`$ and magnetization $`M_{can}=0.307`$. The duration of the plateau increases with the size of the system: in particular we have checked that the lifetime of QSS has a linear dependence on N, see Fig.1(c). Therefore the two limits $`t\mathrm{}`$ and $`N\mathrm{}`$ do not commute and if the thermodynamic limit is performed before the infinite time limit, the system does not relax to the BG equilibrium. This has been conjectured to be an ubiquitous feature in non-extensive systems, but it has also been found for spin glasses. When N increases $`T_{QSS}(N)`$ tends to $`T_{\mathrm{}}=0.380`$, a value obtained analytically as the metastable prolongation (at energies below $`U_c=0.75`$) of the high-energy solution ($`M=0`$). We have also found that $`[T_{QSS}(N)T_{\mathrm{}}]N^{1/3}`$ and $`M_{QSS}N^{1/6}`$, see Fig. 1(d). At the same time we have checked that increasing the size, the largest Lyapunov exponent for the QSS tends to zero. In this sense mixing is negligible and one expects anomalies in the relaxation process . The fact that $`T_{QSS}`$ converges to a nonzero value of temperature for $`N\mathrm{}`$ means that, when N is macroscopically large, systems can share the same temperature, though this equilibrium is not the familiar one. All this amounts to say that the zeroth principle of thermodynamics is stronger than what one might think through BG statistical mechanics, since it is true even when the system is not at the usual BG equilibrium. We have checked the robustness of the above results by changing the level of accuracy of the numerical integration and by adding small perturbations. We also verified that the QSS has a finite basin of attraction, by adopting different initial conditions, as for example double water bag (DWBIC). In Fig.2 we focus on the velocity probability distribution functions (pdfs). The initial velocity pdfs (WBIC or DWBIC) , reported in Fig. 2(a) , quickly acquire and maintain during the entire duration of the metastable state a non-Gaussian shape , see Figs.2(b) and 2(c). The velocity pdf of the QSS is wider than a Gaussian for small velocities, but shows a faster decrease for $`p>1.2`$. The enhancement for velocities around $`p1`$ is consistent with the anomalous diffusion and the Lévy walks (with average velocity $`p1`$) observed in the QSS regime . The following rapid decrease for $`p>1.2`$ is due to conservation of total energy . The stability of the QSS velocity pdf can be explained by the fact that, for $`N\mathrm{}`$, $`M_{QSS}0`$ and thus the force on the spins tends to zero with N, being $`F_i=M_xsin\theta _i+M_ycos\theta _i`$. Of course, for finite N, we have always a small random force, which makes the system eventually evolve into the usual Maxwell-Boltzmann distribution after some time. We show this for small systems (N=500,1000) at time t=500000 in Fig.2(e). When this happens, Lévy walks disappear and anomalous diffusion leaves place to Brownian diffusion . A possible frame to reproduce the non-Gaussian pdf in Fig.2 (b) could be the non-extensive statistical mechanics recently proposed with the entropic index $`q1`$. This formalism provides, for the canonical ensemble, a q-dependent power-law distribution in the variables $`p_i`$ , $`\theta _i`$ . This distribution has to be integrated over all $`\theta _i`$ and all but one $`p_i`$ in order to obtain the one-momentum pdf, $`P_q(p)`$, to be compared with the numerical one, $`P_{num}(p)`$, obtained by considering, within the present molecular dynamical frame, increasingly large N-sized subsystems of an increasingly large M-system. Within the $`M>>N>>1`$ numerical limit, we expect to go from the microcanonical ensemble to the canonical one (the cut-off is then expected to gradually disappear as indeed occurs in the usual short-range Hamiltonians), thus justifying the comparison between $`P_q(p)`$ and $`P_{num}(p)`$. The enormous complexity of this procedure made us to turn instead onto a naive, but tractable, comparison, namely that of our present numerical results with the following one-free-particle pdf  $`P(p)=[1(\frac{1}{2T})(1q)p^2]^{1/(1q)}`$ , which recovers the Maxwell-Boltzmann distribution for $`q=1`$. This formula has been recently used to describe successfully turbulent Couette-Taylor flow  and non-Gaussian pdfs related to anomalous diffusion of Hydra cells in cellular aggregates . In our case, the best fit is obtained by a curve with $`q=7`$, $`T=0.38`$ as shown in Figs. 2 (b) and 2 (c). The agreement between numerical results and theoretical curve improves with the size of the system. A finite-size scaling confirming the validity of the fit is reported in panel (d), where $`\mathrm{\Delta }=P_{th}P_{num}`$, the difference between the numerical results and the theoretical curve for $`q=7`$, is shown to go to zero as a power of N (for four values of p). Since $`q>3`$, the theoretical curve does not have a finite integral and therefore it needs to be truncated with a sharp cut-off to make the total probability equal to one. It is however clear that, the fitting value $`q=7`$ is only an effective non-extensive entropic index. Similar non-Gaussian pdfs have also been found in turbulence and granular matter experiments , though this is the first evidence in a Hamiltonian system. In Fig.3 we verify, through the calculation of the fractal dimension $`D_2`$ , that a dynamical correlation emerges in the $`\mu `$-space before the final arrival to a quasi-uniform distribution. During intermediate times some filamentary structures appear, a similar feature has recently been found also in self-gravitating systems, which might be closely related to the plateaux observed in Fig.1(b). We learn from the curves in Fig.3(c) that, since they do not sensibly depend on N, the possible connection does not concern the entire $`\mu `$-space, but perhaps only the small sticky regions between the ”chaotic sea” and the quasi-orbits.

Metastable states are ubiquitous in nature. Their full understanding is, however, far from trivial. They basically correspond to local, instead of global, minima of the relevant thermodynamic energy. The two types of minima are separated by activation barriers which, at the thermodynamic limit, can be low, high or infinite, all of them presumably occurring in nature. The last case yields of course to quite drastic consequences. Moreover, the local minimum can either make the system to live in a smooth part of the a priori accessible phase space, or it can force it to live in a geometrically more complex (e.g., multifractal) part of the phase space. The richness of such situation is what makes interesting the study of glasses, nuclei, atomic clusters, self-gravitating and other complex systems. It is natural to expect for such systems that the infinite size and infinite time limits are not interchangeable. What has emerged quite clearly here is that thermodynamically large systems with long-range interactions belong to this very rich class. We have verified that the usual attributes of thermal equilibrium: zeroth principle at finite temperatures, robustness associated with a finite basin of attraction in the space of the initial conditions, stable distribution of velocities, are satisfied, but they systematically differ from what BG statistical mechanics has make familiar to us along the last 130 years. Our findings indicate some consistency with the predictions of non-extensive statistical mechanics , though a firm and unambiguous connection remains a challenge for future studies. In particular we believe all these features not to be exclusive of the present HMF model. Similar scenarios are expected for systems with say two-body interactions decaying like $`r^\alpha `$ for $`0\alpha \alpha _c`$ , where $`\alpha _c`$ is equal, for classical systems, to the space dimension .

We thank M. Antoni, F. Baldovin, M. Baranger, E.P. Borges, E.G.D. Cohen, X. Campi, H. Krivine, M. Mézard, S. Ruffo and A. Torcini for stimulating discussions.

E-mail: vito.latora@ct.infn.it

E-mail: andrea.rapisarda@ct.infn.it

E-mail: tsallis@cbpf.br


format: text

text:
# CE-315 : A New Interacting Double-degenerate Binary Star.Based on observations obtained at La Silla (ESO), Chile. Project 64.H-0318.

## 1 Introduction

During the spectroscopic follow-up of proper motion stars in the Calán-ESO Catalog (Ruiz et al. 2001), the star CE-315 ($`\mu =0.34^{\prime \prime }yr^1,\theta =264^{}`$) was found to have a spectrum consisting of a blue continuum with strong emission lines of He I and He II (weaker). The line profiles were clearly variable in time scales shorter than 5 minutes, with multiple components evident even in a low resolution discovery spectrum. The spectrum of CE-315 is strikingly similar to that of the unique object GP Com (= G61-29), discovered by Burbidge & Strittmatter (1971).

Nather, Robinson & Stover (1981) found that GP Com has an orbital period of 46.5 minutes, and suggested that it is a double degenerate system in which the lower mass star transfers He through an accretion disk onto the higher mass component. Evidence for CNO processed material in the accretion disk of GP Com was found by Marsh et al. (1991), who detected emission in lines of N I, N II, O I, Ne I and possibly Mg II. The relative strength of these lines indicate that nitrogen is highly over-abundant with respect to carbon and oxygen compared to solar values, thus providing evidence for the action of CNO cycle in GP Com. The enhanced abundance of C, N, O, Mg, may have its origin in the thermal pulse stage of the primary star which later on transferred this processed material to the secondary during a common envelope phase. More recently Marsh (1999) published a detailed study of the kinematics of GP Com based on time-resolved spectroscopy. He found a radial velocity of the narrow component at the center of the triple peak He lines of $`10.8\pm 1.6`$ km s-1 in phase relative to the “S”-wave (Nather, Robinson & Stover 1981), consistent with an accreting white dwarf if the mass ratio of the system $`q(=M_2/M_1)`$ is $`0.02`$.

The understanding of the evolutionary path of a binary system leading to a double-degenerate (DD) system, such as GP Com, is a matter of considerable theoretical interest (Iben & Tutukov 1984; Mochkovitch & Livio 1989; Branch et al. 1995). However, in spite of extensive search efforts (Robinson & Shafter 1987; Bragaglia et al. 1990), there are only a handful of interacting binary white dwarfs known so far, and thus little is known observationally about this class of objects. Here we report the discovery of an accreting DD system with a period of 65.1 minutes, which makes it the longest period member of the AM CVn group.

## 2 Observations

CE-315 was discovered during the spectroscopic follow-up of high proper motion stars from the Calán-ESO Catalog (CE). The discovery spectrum was obtained March 22, 1998, using the ESO 3.6m telescope at La Silla equipped with EFOSC2 and the B300b grism. This combination produces spectra with a $`16`$ Å resolution, covering the spectral range from 3860 Å to 8070 Å. A first inspection of the spectrum, taken with 1800s integration time, showed no sign of any other lines, except for the HeI and HeII (weak) emission lines.

Shortly after the discovery, during April 3 and 4 of 1998, a series of spectra were obtained with the CTIO Blanco 4m telescope with the RC Spectrograph, the Loral 3K CCD and grating KPGL #2. The nights were not photometric. However, what became clear from this series of observations was the variable nature of the spectrum of CE-315, with line profile changing in shape and strength in time scales of less than 300 s.

During the nights of March 19 and 20, 1999, spectra of CE-315 were obtained, under photometric conditions, with the Blanco 4m telescope at CTIO. In this opportunity we used the RC Spectrograph with the Loral 3K CCD detector and grating KPGL #3, covering the spectral range between 3750 Å and 7000 Å with a resolution of $`4`$ Å. The slit was 1.5″ wide oriented according to the parallactic angle. The seeing during the first night was $``$ 1.3″ while during the second night was 1.5″. During the first night (3/19/99), 25 consecutive spectra of 360 s integration time each were obtained. Taking into account the reading time, the time difference between each spectrum is about 510 s. Similarly, during the second night (3/20/99), 30 consecutive spectra of 360 s integration time were obtained.

During March 23, 1999, using the ESO 3.6m telescope at La Silla equipped with EFOSC2 and grism #16, we obtained a sequence of 27 spectra (360 s integration each) with a 60 s R image taken between each spectrum. The spectral coverage was from 4700 Å to 6755 Å at a resolution of 13 Å. The slit was 1″ wide oriented according to the parallactic angle.

In all spectroscopic runs described above, we observed spectrophotometric standards and obtained He-Ar frames in order to flux and wavelength calibrate the data using IRAF packages.

Photometry of CE-315 was obtained in April 12, 1999, at the CTIO 0.9m telescope using a CCD (Tek 2k #3), under photometric conditions. The following magnitudes were obtained : B=17.23, V=17.67, R=17.47, and I=17.27 (Kron-Coussins). The series of R images taken with the 3.6m telescope, described above, clearly showed that the object is photometrically variable in time scales of minutes, therefore, given that the integration times used for obtaining the photometry with the 0.9m were 1200 s in B, 900 s in V, and 660 s in R and I, the above magnitudes are only an average. The observed variations in R occurred in time scales of two to three minutes and with amplitudes up to 0.2 mag.

In March 21, 1999 using ESO 3.6m telescope and EFOSC2 a direct He I (6678 Å) image of CE-315 was obtained, with an integration time of 1200 s. The PSF shape and size of CE-315 was similar to the rest of the stars in the frame indicating that, as in GP Com (Stover, 1983), there is no resolved He emission surrounding CE-315 that could be attributed to mass ejections during previous evolutionary phases.

## 3 Results

### 3.1 Average Spectra

Figure 1 presents the average of all the individual spectra taken in a given observing run. The upper panels show the average spectrum obtained from the CTIO observations made during April 4, 1998, covering the spectral range from 3750 Å to 8830 Å. The lower panels show the average spectra obtained from the observations made in March 1999 at CTIO and ESO, respectively. Most of the detected lines can be identified with either He I or He II lines in emission, with a handful of faint lines of N I. Notable is the complete absence of hydrogen emission lines.

### 3.2 Line Profiles

He I lines from CE-315 show triple-peaked profiles, although in a few cases the emission from the central component almost completely dominates the spectrum. This is illustrated in Figure 2, which shows the profile of the $`\lambda `$4471.5, $`\lambda `$5875.6, and $`\lambda `$6678.1 lines of He I. The relative intensities of the outer and central components differ from line to line, implying that the line profiles are made up of emission that originates from two independent sources. We suggest that the emission from the outer components of the line profiles arises from an accreting disk (double-peaked profiles are characteristics of disk emission), whereas the central component has a different origin.

The profiles of the He I emission are remarkably broad, covering a velocity range of about 2000 km s-1 (FWZP). FWHM line widths, determined by simultaneously fitting 3 Gaussian components to the observed profiles, are typically 500 km s-1 for the blue disk feature, 580 km s-1 for the red disk feature, and 345 km s-1 for the central feature.

The above characteristics suggest that CE-315 is a binary system with the lower mass star transferring mass by way of an accretion disk. In what follows we present the determination of observed parameters of this system.

### 3.3 Orbital Period

For the determination of the orbital period of the system we followed the method devised by Nather, Robinson & Stover (1981), in which the profile of each line is divided in a blue wing region and a red wing region, and the difference between the integrated intensity in the two bands is analyzed for periodicity.

Figure 3 shows the difference in red and blue integrated intensities (R-B) versus time for three of the most intense lines within the observed spectral range and for three different nights. These non-uniformly spaced time series were investigated using the periodogram analysis for unevenly spaced data devised by Scargle (1982), and developed in Press et al. (1992). The resulting power spectrum are also shown in Figure 3. The periods determined using the data taken on March 19, March 20, and March 24 of 1999 are, respectively, $`65.2\pm 0.9`$ minutes, $`64.4\pm 1.1`$ minutes, and $`65.7\pm 1.2`$ minutes. In what follows we adopt a period for the system of 65.1 minutes.

### 3.4 The “S” Wave

An inspection of the line profiles shows that the dominant form of the modulation is due to changes in intensity of the line profiles. To study this modulation in more detail we subtracted from each individual spectrum an average spectrum and analyzed the resulting spectra as a function of phase. Figure 4 presents trailed spectra, using the data obtained during March of 1999 at CTIO, showing that most of the modulation is due to an emission feature that wanders in velocity between the blue and red velocity of the disk component. Due to its appearance on single trailed spectra, this feature is usually labeled as the “S” wave. The “S” feature is thought to mark a bright region of enhanced emission where the gas stream from the donor hits the accretion disk. In the reference frame rotating with the binary system the bright spot is fixed in position, but the velocity of the gas in the spot is similar to the velocity of the disk at that position. The velocity variation shown in Figure 4 is well fitted by a sinusoidal function, with a semiamplitude of $`630\pm 15`$ km s-1. This suggests that the hot spot is located at a radius of about 0.7 times the radius of the outer edge of the disk and that it is undergoing circular orbital motions. Similar to what is observed in GP Com, in CE-315 the intensity of the “S” wave is noticeably weaker when moving from blueshifted to redshifted velocities, which is probably due to effects of the inclination of the system.

### 3.5 Disk velocity and Orbital Velocity of the Primary Component

The projected rotational disk velocity, $`V_dsini`$, can be determined from the half-separation of the outer peaks. From the profiles in the $`\lambda `$5875.6 and $`\lambda `$6678.1 lines observed with the CTIO telescope during March 1999, which are the lines with the largest velocity resolution, we derive a half-separation of $`510\pm 10`$ km s-1.

Assuming a simple model of a Keplerian disk rotating around the primary component of a binary system, the radial velocities of the red and blue components of the disk profiles, $`V_r`$ and $`V_b`$, respectively, should vary with phase, $`\varphi `$, as (e.g., Smak 1976)

$$V_{r,b}=\gamma \pm V_dsini+K_1sin\varphi ,$$
(1)
where $`\gamma `$ is the systemic velocity and $`K_1`$ is the amplitude of the radial velocity of the primary component. For each line we computed $`V_{r,b}`$ from the observed profile using the expression

$$V_{r,b}=\frac{_{r_1,b_1}^{r_2,b_2}I(v)v𝑑v}{_{r_1,b_1}^{r_2,b_2}I(v)𝑑v},$$
(2)
where $`I(v)`$ is the observed intensity at velocity $`v`$, and $`r_1,b_1`$ and $`r_2,b_2`$ are the initial and final velocities of the range of red and blue disk velocities, respectively. Figure 5 shows the dependence of the blue disk velocity with phase derived using the data taken during the night of March 20 of 1999. The blue disk velocity corresponds to an average of the individual $`V_b`$ determined for the $`\lambda `$5875.6 and $`\lambda `$6678.1 lines. A least squares fit to the observed trend gives $`\gamma V_dsini=490`$ km s-1 and $`K_1=8\pm 1`$ km s-1. The first order velocity moment used by us as representative of the left hand side of eqn.(1), should be less sensitive than other estimates to the presence of perturbations such as the “S” wave.

## 4 Discussion

### 4.1 The Binary System

Given the short period of CE-315 (65.1 minutes), two candidates for the donor star are possible (see Faulkner et al. 1972) : i) a helium ZAMS of $`1.7`$$`M_{\mathrm{}}`$, and ii) a degenerate helium low-mass white dwarf with M2 $`0.017`$ $`M_{\mathrm{}}`$ (Warner 1995). The first alternative can be easily dismissed, since in that case the light of the ZAMS star would dominate the spectrum. We therefore conclude that the donor is a low-mass helium degenerate. In addition, the large rotational velocities of the accretion disk, of the order of 510 km s-1, require that the accreting star be also a collapsed star. Since CE-315 is not known to be a strong X-ray source, the accreting star is unlikely to be a neutron star or a black hole, and therefore we conclude it must be a white dwarf.

From the observed projected rotational velocity of the disk and the orbital velocity of the primary component, it is possible to compute the mass ratio of the binary system, $`q=\frac{m_2}{m_1}`$, using the expression (Smak 1976)

$$\frac{q}{(1+q)^{1/2}}=\left(\frac{K_1}{V_dsini}\right)\left(\frac{a}{r_d}\right)^{1/2},$$
(3)
where $`a`$ is the separation of the system and $`r_d`$ the disk radius. Using $`V_dsini`$= 510 km s-1, $`K_1`$=8 km s-1, and assuming $`\frac{r_d}{a}=0.5`$, we obtain $`q=0.022`$. The last assumption is based on the computations of Lin & Pringle (1976), who found that the accretion disk is comparable in size to the Roche lobe of the primary. Assuming M2 = 0.017 $`M_{\mathrm{}}`$, the derived mass ratio implies that the accretor mass is $`0.77`$ $`M_{\mathrm{}}`$.

### 4.2 Origin of the Central Component

The emission from the central component of CE-315 is barely resolved, exhibiting a line width of $``$350 km s-1, broader than that of the central feature in GP Com which is $``$ 120 km s-1 (Marsh 1999). The ratio of intensities of the central to the disk component differ from line to line, increasing for higher excitation lines. In particular, in the He II 4686 line the emission from the central component completely dominates over that of the disk (see Table 1). These results suggest that the emission from the central component arises from a region with a temperature higher than that in the disk, high enough to ionize helium.

Our direct He I ($`\lambda 6678`$) image of CE-315 shows that the helium emission is unresolved, thus it is unlikely that the emission from the central component arises in a nebular shell, produced by mass ejections during a previous evolutionary phase. Further, the small radial velocity variation of the central feature makes it unlikely that it arises in the mass donor. Thus, we suggest that the central emission originates near the accreting white dwarf. Possible sites are: i) the inner boundary layer produced by accretion onto the white dwarf, and ii) the region of inflow from the Alfven surface to the star for a magnetized white dwarf (see Frank, King, & Raine 1992). In the first case the temperature of the boundary layer is about 6 times larger than that of the disk, and implies that CE-315 should be a soft X-ray source. An interesting difference between CE-315 and GP Com is that in CE-315 the radial velocity of the central component is similar to the mean radial velocity of the double peaks, whereas in GP Com the central component is significantly redshifted with respect to the later.

### 4.3 Evolutionary Stage of CE-315

Following the same line of argument that lead Nather, Robinson & Stover (1981) to conclude that GP Com is an evolutionary descendant of a system like AM CVn, we propose that CE-315, having an almost pure He emission line spectrum and a longer period, is also a descendant of an AM CVn system but at a slightly more evolved stage than GP Com.

The number of double degenerate (DD) systems detected in different systematic searches (see Robinson & Shafter 1987, Foss et al. 1991, Bragaglia et al. 1990) have turned out to be very small. In an effort to explain this observational fact, Hernanz, Isern, & Salaris (1997) estimated the expected population of DD systems in the solar neighborhood taking into account the scale height of these objects and the white dwarf’s cooling time. As a result they find that the predictions from the models are in agreement with the observed scarcity of DD systems. In particular, they predict a number density of DD systems with a period of 65 minutes (like CE-315) between $`10^5`$ and $`1.4\times 10^5`$ pc-3 depending on whether the stellar formation rate is compatible with the models of chemical evolution of the Galaxy or with the white dwarf’s luminosity function. Unfortunately the distance to CE-315 is not known, which prevents us from deriving a density to compare with model predictions. An accurate determination of the distance to CE-315 (and GP Com) through trigonometric parallaxes, would allow us to calculate the number density of DD systems in the CE survey, with well defined limits in $`m_R19.5`$ and $`\mu =0.2\mathrm{}`$ yr-1, which would be crucial to test model predictions, thus making an important contribution to a better understanding of the evolutionary phase of this class of objects.

M.T.R. and P.R. received partial support from FONDECYT grant 1980659 and Catedra Presidencial en Ciencias. G.G. acknowledges support from a Chilean Presidential Science Fellowship and FONDECYT project 1980660.


text:
# Learning Policies with External Memory

## 1 Introduction

A reinforcement-learning agent must learn a mapping from a stream of observations of the world to a stream of actions. In completely observable domains, it is sufficient to look only at the last observation, so the agent can learn a “memoryless” mapping from observations to actions . In general, however, the agent’s actions may have to depend on the history of previous observations.

##### Previous Work

There have been many approaches to learning to behave in partially observable domains. They fall roughly into three classes: optimal memoryless, finite memory, and model-based.

The first strategy is to search for the best possible memoryless policy. In many partially observable domains, memoryless policies can actually perform fairly well. Basic reinforcement-learning techniques, such as Q-learning , often perform poorly in partially observable domains, due to a very strong Markov assumption. Littman showed  that finding the optimal memoryless policy is NP-Hard. However, Loch and Singh  effectively demonstrated that techniques, such as sarsa$`(\lambda )`$, that are more oriented toward optimizing total reward, rather than Bellman residual, often perform very well. In addition, Jaakkola, Jordan, and Singh  have developed an algorithm for finding stochastic memoryless policies, which can perform significantly better than deterministic ones .

One class of finite memory methods are the finite-horizon memory methods, which can choose actions based on a finite window of previous observations. For many problems this can be quite effective . More generally, we may use a finite-size memory, which can possibly be infinite-horizon (the systems remembers only a finite number of events, but these events can be arbitrarily far in the past). Wiering and Schmidhuber  proposed such an approach, learning a policy that is a finite sequence of memoryless policies.

Another class of approaches assumes complete knowledge of the underlying process, modeled as a partially observable Markov decision process (pomdp). Given a model, it is possible to attempt optimal solution , or to search for approximations in a variety of ways . These methods can, in principle, be coupled with techniques, such as variations of the Baum-Welch algorithm , for learning the model to yield model-based reinforcement-learning systems.

##### Stigmergy

In this paper, we pursue an approach based on stigmergy. The term is defined in the Oxford English Dictionary  as “The process by which the results of an insect’s activity act as a stimulus to further activity,” and is used in the mobile robotics literature  to describe activity in which an agent’s changes to the world affect its future behavior, usually in a useful way.

One form of stigmergy is the use of external memory devices. We are all familiar with practices such as making grocery lists, tying a string around a finger, or putting a book by the door at home so you will remember to take it to work. In each case, an agent needs to remember something about the past and does so by modifying its external perceptions in such a way that a memoryless policy will perform well.

We can apply this approach to the general problem of learning to behave in partially observable environments. Figure 1 shows the architectural idea. We think of the agent as having two components: one is a set of memory bits; the other is a reinforcement-learning agent. The reinforcement-learning agent has as input the observation that comes from the environment, augmented by the memory bits. Its output consists of the original actions in the environment, augmented by actions that change the state of the memory. If there are sufficient memory bits, then the optimal memoryless policy for the internal agent will cause the entire agent to behave optimally in its partially observable domain.

Consider, for instance, the load-unload problem represented in Figure 2. In this problem, the agent is a cart that must drive from an Unload location to a load location, and then back to unload. This problem is a simple MDP with a one-bit hidden variable that makes it non-Markov (the agent cannot see whether it is loaded or not). It can be solved using a one-bit external memory: we set the bit when we make the Unload observation, and we go right as long as it is set to this value and we do not make the Load observation. When we do make the Load observation, we clear the bit and we go left as long as it stays at value 0, until we reach state 9, getting a reward.

There are two alternatives for designing an architecture with external memory:

* Either we augment the action space with actions that change the content of one of the memory bits (adds $`L`$ new actions if there are L memory bits); changing the state of the memory may require multiple steps.
* Or we compose the action space with the set of all possible values for the memory (the size of the action space is then multiplied by $`2^L`$, if there are $`L`$ bits of memory). In this case, changing the external memory is an instantaneous action that can be done at each time step in parallel with a primitive action, and hence we can reproduce the optimal policy of the load-unload problem, without taking additional steps.

Complexity considerations usually lead us to take the first option. It introduces a bias, since we have to lose at least one time-step each time we want to change the content of the memory. However, it can be fixed in most algorithms by not discounting memory-setting actions.

The external-memory architecture has been pursued in the context of classifier systems  and in the context of reinforcement learning by Littman  and by Martín . Littman’s work was model-based; it assumed that the model was completely known and did a branch-and-bound search in policy space. Martín worked in the model-free reinforcement-learning domain; his algorithms were very successful at finding good policies for very complex domains, including some simulated visual search and block-stacking tasks. However, he made a number of strong assumptions and restrictions: task domains are strictly goal-oriented; it is assumed that there is a deterministic policy that achieves the goal within some specified number of steps from every initial state; and there is no desire for optimality in path length.

##### This work

We were inspired by the success of Martín’s algorithm on a set of difficult problems, but concerned about its restrictions and a number of details of the algorithm that seemed relatively ad hoc. At the same time, Baird and Moore’s work on vaps , a general method for gradient descent in reinforcement learning, appealed to us on theoretical grounds. This paper is the result of attempting to apply vaps algorithms to stigmergic policies, and understanding how it relates to Martín’s algorithm. In this process, we have derived a much simpler version of vaps for the case of highly non-Markovian domains: we calculate the same gradient as vaps, but with much less computational effort.

In the next section, we present the relevant learning algorithms. Then we describe a set of experimental domains and discuss the relative performance of the algorithms.

## 2 Algorithms

We begin by describing the most familiar of the algorithms, sarsa$`(\lambda )`$. We then describe the vaps algorithm in some detail, followed by our simplified version.

### 2.1 sarsa$`(\lambda )`$

sarsa is an on-policy temporal-difference control learning algorithm . Given an experience in the world, characterized by starting state $`x`$, action $`a`$, reward $`r`$, resulting state $`x^{}`$ and next action $`a^{}`$, the update rule for sarsa(0) is

$$Q(x,a)Q(x,a)+\alpha [r+\gamma Q(x^{},a^{})Q(x,a)].$$
(1)
It differs from the classical Q-learning algorithm  in that, rather than using the maximum $`Q`$-value from the resulting state as an estimate of that state’s value, it uses the $`Q`$-value of the resulting state and the action that was actually chosen in that state. Thus, the values learned are sensitive to the policy being executed.

In truly Markov domains, Q-learning is usually the algorithm of choice; policy-sensitivity is often seen as a liability, because it makes issues of exploration more complicated. However, in non-Markov domains, policy-sensitivity is actually an asset. Because observations do not uniquely correspond to underlying states, the value of a policy depends on the distribution of underlying states given a particular observation. But this distribution generally depends on the policy. So, the value of a state, given a policy, can only be evaluated while executing that policy. In fact, Q-learning can be shown to fail to converge on very simple non-Markov domains . Note that, when sarsa is used in a non-Markovian environment, the symbols $`x`$ and $`x^{}`$ in equation (1) represent observations, which usually can correspond to several states.

The sarsa algorithm can be augmented with an eligibility trace, to yield the sarsa$`(\lambda )`$ algorithm (a detailed exposition is given by Sutton and Barto .) With the parameter $`\lambda `$ set to 0, sarsa$`(\lambda )`$ is just sarsa. With $`\lambda `$ set to 1, it is a pure Monte Carlo method, in which, at the end of every trial, each state-action pair is adjusted toward the cumulative reward received on this trial after the state-action pair occurred. Pure Monte-Carlo algorithms make no attempt at satisfying Bellman equations relating the values of subsequent states; in partially observable domains, it is often impossible to satisfy the Bellman equation, making Monte-Carlo a reasonable choice. sarsa$`(\lambda )`$ describes a useful class of algorithms, then, with appropriate choice of $`\lambda `$ depending on the problem. Thus, sarsa$`(\lambda )`$ with a large value of $`\lambda `$ seems like the most appropriate of the conventional reinforcement-learning algorithms for solving partially-observable problems.

### 2.2 vaps

Baird and Moore have derived, from first principles, a class of stochastic gradient-descent algorithms for reinforcement learning. At the most abstract level, we seek to minimize some measure of the expected cost of our policy; we can describe this high-level criterion as

$$B=\underset{T=0}{\overset{\mathrm{}}{}}\underset{\stackrel{~}{s}\stackrel{~}{S}_T}{}\mathrm{Pr}(\stackrel{~}{s})\epsilon (\stackrel{~}{s}),$$

where $`\stackrel{~}{S}_T`$ is the set of all possible experience sequences that terminate at time $`T`$. That is,

$$\stackrel{~}{s}=x_0,u_0,r_0,\mathrm{},x_t,u_t,r_t,\mathrm{},x_T,u_T,r_T,$$

where $`x_t`$, $`u_t`$, and $`r_t`$ are the observation, action, and reward at step $`t`$ of the sequence, and $`x_T`$ is an observation associated with a terminal state. The loss incurred by a sequence $`\stackrel{~}{s}`$ is $`\epsilon (\stackrel{~}{s})`$. We restrict our attention to time-separable loss functions, which can be written as

$$\epsilon (\stackrel{~}{s})=\underset{t=0}{\overset{T}{}}e(\mathrm{trunc}(\stackrel{~}{s},t)),\mathrm{for}\mathrm{all}\stackrel{~}{s}\stackrel{~}{S}_T,$$

where $`e(s)`$ is an instantaneous error function associated with each (finite) sequence prefix $`s=x_0,u_0,r_0,\mathrm{},x_t,u_t,r_t`$ ($`x_t`$ being any observation, not necessarily a terminal one), and $`\mathrm{trunc}(\stackrel{~}{s},t)`$ representing the sequence $`\stackrel{~}{s}`$ truncated after time $`t`$. For instance, an error measure closely related to Q-learning is the squared Bellman residual:

$$\begin{array}{cc}\hfill e_{\mathrm{ql}}(s)=& \frac{1}{2}\underset{x}{}\mathrm{Pr}(x_t=xx_{t1},u_{t1})\hfill \\ & [r_{t1}+\underset{u}{\mathrm{max}}\gamma Q(x,u)Q(x_{t1},u_{t1})]^2.\hfill \end{array}$$

The sarsa version of the algorithm uses the following error measure:

$$\begin{array}{cc}& e_{\mathrm{sarsa}}(s)=\hfill \\ & \frac{1}{2}\underset{x}{}\mathrm{Pr}(x_t=xx_{t1},u_{t1})\underset{u}{}\mathrm{Pr}(u_t=ux_t)\hfill \\ & [r_{t1}+\gamma Q(x,u)Q(x_{t1},u_{t1})]^2.\hfill \end{array}$$

Note that we average over all possible actions $`u_t`$ according to their probability of being chosen by the policy instead of picking the one that maximizes Q-values as in $`e_{\mathrm{ql}}`$. Baird and Moore also consider a kind of policy search, which is analogous to reinforce :

$$e_{\mathrm{policy}}(s)=b\gamma ^tr_t,$$

where $`b`$ is any constant. This immediate error is summed over all time $`t`$, leading to a summation of all discounted immediate rewards $`\gamma ^tr_t`$. In order to obtain the good properties of both criteria, they construct a final criterion that is a linear combination of the previous two:

$$e=(1\beta )e_{\mathrm{sarsa}}+\beta e_{\mathrm{policy}}.$$

This criterion combines Value And Policy Search and is, hence, called vaps. We will refer to it as vaps$`(\beta )`$, for different values of $`\beta `$.

Baird and Moore show that the gradient of the global error with respect to weight $`k`$ can be written as:

$$\begin{array}{cc}& \frac{}{w_k}B=\underset{t=0}{\overset{\mathrm{}}{}}\underset{sS_t}{}\mathrm{Pr}(s)\hfill \\ & \left[\frac{}{w_k}e(s)+e(s)\underset{j=1}{\overset{t}{}}\frac{}{w_k}\mathrm{ln}\mathrm{Pr}(u_{j1}|x_{j1})\right],\hfill \end{array}$$

where $`S_t`$ is the set of all experience prefixes of length $`t`$. Technically, it is necessary that $`\mathrm{Pr}(u^t=ux_t=x)>0`$ for all $`(x,u)`$ (otherwise, some zero probability trajectories may have a non-zero contribution to the gradient of $`B`$ ). In this work we use the Boltzmann law for picking actions, which guarantees this property (see section 2.3).

It is possible to perform stochastic gradient descent of the “error” $`B`$, by repeating several trials of interaction with the process. Each experimental trial of length $`T`$ provides one sample of $`sS_t`$ for each $`tT`$. Of course, these samples are not independent, but it does not matter since we are summing them and not multiplying them. We are thus using stochastic approximation to estimate the expectation over $`sS_t`$ in the above equation. During each trial, the weights are kept constant and the approximate gradients of the error at each time $`t`$,

$$\frac{}{w_k}e(s)+e(s)\underset{j=1}{\overset{t}{}}\frac{}{w_k}\mathrm{ln}\mathrm{Pr}(u_{j1}|x_{j1}),$$

are accumulated. Weights are updated at the end of each trial, using the sum of these immediate gradients. An incremental implementation of the algorithm can be obtained by using, at every step $`t`$, the following update rules:

$`\mathrm{\Delta }T_{k,t}`$ $`=`$ $`{\displaystyle \frac{}{w_k}}\mathrm{ln}\mathrm{Pr}(u_{t1}|x_{t1}),`$
$`\mathrm{\Delta }w_k`$ $`=`$ $`\alpha \left[{\displaystyle \frac{}{w_k}}e(\stackrel{~}{s}_t)+e(\stackrel{~}{s}_t)T_{k,t}\right],`$
where $`s_t`$ represents the experience prefix
$`x_0,u_0,r_0,\mathrm{},x_t,u_t,r_t`$, i.e., the history at time $`t`$. Note that the “exploration trace” $`T_{k,t}`$ is independent of the immediate error $`e`$ used. It only depends on the way the output $`\mathrm{Pr}(u_t=u|x_t)`$ varies with the weights $`w_k`$, i.e., on the representation chosen for the policy.

The gradient of the immediate error $`e`$ with respect to the weight $`w_k`$ is easy to calculate. For instance, in the case of the sarsa variant of the algorithm we have:

$$\begin{array}{cc}& \frac{}{w_k}e_{\mathrm{sarsa}}(s)=\hfill \\ & \underset{x}{}\mathrm{Pr}(x_t=xx_{t1},u_{t1})\underset{u}{}\mathrm{Pr}(u_t=ux_t)\hfill \\ & \left[r_{t1}+\gamma Q(x,u)Q(x_{t1},u_{t1})\right]\hfill \\ & \left[\gamma \frac{}{w_k}Q(x,u)\frac{}{w_k}Q(x_{t1},u_{t1})\right].\hfill \end{array}$$

Once more, we descend this gradient by stochastic approximation: the averaging over $`x_t`$ and $`u_t`$ is replaced by a sampling of these quantities. However, since these variables appear twice in the equation and they are not just added, we have to sample both $`x_t`$ and $`u_t`$ independently two times in order to avoid any bias in the estimation of the gradient. It is not realistic to satisfy this requirement in a truly on-line situation, since the only way to get a new observation is by actually performing the action. Note that for the case $`\beta =1`$ we do not need the second sample, so the vaps(1) algorithm is effective in the on-line case.

In the case of policy search we have: $`\frac{}{w_k}e_{\mathrm{policy}}(s)=0`$,
for all $`w_k`$. This may seem strange; but for policy search, the important thing is the state occupations, which enter into the weight updates through the trace.

### 2.3 vaps(1)

In this section, we explore a special case of vaps, in which the $`Q`$-values are stored in a look-up table. That is, there is one weight $`w_k=Q(x,u)`$ for each state-action pair. Note that it is not necessary to use the vaps sequence-based gradient in a look-up table implementation of ql or sarsa, as long as it is confined to a Markovian environment. However, it makes sense to use it in the context of pomdps. Under this hypothesis, the exploration trace $`T_{k,t}`$ associated with each parameter $`Q(x,u)`$ will be written $`T_{x,u,t}`$.

We will also focus on a very popular rule for randomly selecting actions as a function of their $`Q`$-value, namely the Boltzmann law:

$$Pr(u_t=u|x_t=x)=\frac{e^{Q(x,u)/c}}{_u^{}e^{Q(x,u^{})/c}},$$

where $`c`$ is a temperature parameter.111Note that Baird and Moore use an unusual version of the Boltzmann law, with $`1+e^x`$ in place of $`e^x`$ in both the numerator and the denominator. We have found that it complicates the mathematics and worsens the performance, so we will use the standard Boltzmann law throughout. Under this rule we get:

$$\begin{array}{cc}& \frac{\mathrm{ln}\mathrm{Pr}(u_t=u|x_t=x)}{Q(x^{},u^{})}=\hfill \\ & \{\begin{array}{cc}0\hfill & \mathrm{if}x^{}x,\hfill \\ \mathrm{Pr}(u_t=u^{}|x_t=x)/c\hfill & \mathrm{if}x^{}=x\mathrm{and}u^{}u,\hfill \\ \left[1\mathrm{Pr}(u_t=u|x_t=x)\right]/c\hfill & \mathrm{if}x^{}=x\mathrm{and}u^{}=u.\hfill \end{array}\hfill \end{array}$$

In this case, and if we add the hypothesis that the problem is an achievement task, i.e., the reward is always 0 except when we reach an absorbing goal state, the exploration trace $`T_{x,u,t}`$ takes a very simple form:

$$\begin{array}{ccc}T_{x,u,t}\hfill & =\hfill & \frac{1}{c}\left[N_{x,u}^tN_x^t\mathrm{Pr}\left(u_t=u|x_t=x\right)\right]\hfill \\ & =\hfill & \frac{1}{c}\left[N_{x,u}^tE[N_{x,u}^t]\right],\hfill \end{array}$$
(2)
where $`N_{x,u}^t`$ is the number of times that action $`u`$ has been executed in state $`x`$ at time $`t`$, $`N_x^t`$ is the number of times that state $`x`$ has been visited at time $`t`$, and $`E[N_{x,u}^t]`$ represents the expected number of times we should have performed action $`u`$ in state $`x`$, knowing our exploration policy and our previous history.

As a result of equation (2), vaps using $`e_{policy}`$ as immediate error, look-up tables and Boltzmann exploration reduces to a very simple algorithm. At each time-step where the current trial does not complete, we just increment the counter $`N_{x,u}^t`$ of the current state-action pair. When the trial completes, this trace is used to update all the $`Q`$-values, as described above.

It is interesting to try to understand the properties and implications of this simple rule. First, a direct consequence is that when something surprising happens, the algorithm adjusts the unlikely actions more than the likely ones. In other words, this simple procedure is very intuitive, since it assigns credit to state-action pairs proportional to the deviation from the expected behaviour. Note that sarsa$`(\lambda )`$ is not capable of such a discrimination. This difference in behaviour is illustrated in the simulation results.

A second interesting property is that the $`Q`$-value updates tend to $`0`$ as the length of the trial tends to infinity. This also makes sense, since the longer the trial, the less the final information received (the final reward) is relevant in evaluating each particular action. Alternatively, we could say that when too many actions have been performed, there is no reason to attribute the final result more to one of them than to others. Finally, unlike with Baird and Moore’s version of the Boltzmann law, the sum of the updates to the $`Q`$-values on every step is 0. This makes it more likely that the weights will stay bounded.

## 3 Experiments

##### Domains

We have experimented with sarsa and vaps on five simple problems. Two are illustrative problems previously used in the reinforcement-learning literature; two others are instances of load-unload with different parameters; and the fifth is a variant of load-unload designed by us in an attempt to demonstrate a situation in which vaps might outperform sarsa. The five problems are :

* Baird and Moore’s problem , designed to illustrate the behavior of vaps,
* McCallum’s 11-state maze , which has only 6 observations.
* The load-unload problem, as described above, in which there are three locations (the loading location, the unloading location, and one intermediate one),
* A five-location load-unload problem (fig. 2), and
* A variant of the load-unload problem where a second loading location has been added, and the agent is punished instead of rewarded if it gets loaded at the wrong location. The state space is shown in figure 3; states contained in a box are observationally indistinguishable to the agent. The idea here is that there is a single action that, if chosen, ruins the agent’s long-term prospects. If this action is chosen due to exploration, then sarsa$`(\lambda )`$ will punish all of the action choices along the chain but vaps will punish only that action.

All these domains have a single starting state, except McCallum’s problem, where the starting state is chosen uniformly at random.

##### Algorithmic Details

For each problem, we ran two algorithms: vaps(1) and sarsa(1). The optimal policy for Baird’s problem is memoryless, so the algorithms were applied directly in that case. For the other problems, we augmented the input space with an additional memory bit, and added two actions: one for setting the bit and one for clearing it.

The $`Q`$-functions were represented in a table, with one weight for each observation-action pair. The learning rate is determined by a parameter, $`\alpha _0`$; the actual learning rate has an added factor that decays to 0 over time: $`\alpha =\alpha _0+\frac{1}{10N}`$, where $`N`$ is trial number. The temperature was also decayed in an ad hoc way, from $`c_{\mathrm{max}}`$ down to $`c_{\mathrm{min}}`$ with an increment of

$$\delta c=\left(\frac{c_{min}}{c_{max}}\right)^{1/(N1)}$$

on each trial. In order to guarantee convergence of sarsa in mdps, it is necessary to decay the temperature in a way that is dependent on the $`Q`$-values themselves ; in the pomdp setting it is much less clear what the correct decay strategy is. In any case, we have found that the empirical performance of the algorithm is not particularly sensitive to the temperature. The parameter $`b`$ in the immidiate error $`e_{\mathrm{policy}}`$ of vaps was always set to $`0`$.

##### Experimental Protocol

Each learning algorithm was executed for $`K`$ runs; each run consisted of $`N`$ trials, which began at the start state and executed until a terminal state was reached or $`M`$ steps were taken. If the run was terminated at $`M`$ steps, it was given a terminal reward of -1; $`M`$ was chosen, in each case, to be 4 times the length of the optimal solution. At the beginning of each run, the weights were randomly reinitialized to small values.

##### Results

It was easy to make both algorithms work well on the first three problems: Baird’s, McCallum’s and small load-unload. The algorithms typically converged in fewer than 100 runs to an optimal policy. One thing to note here is that our version of vaps, using the true Boltzmann exploration distribution rather than the one described by Baird and Moore, seems to perform significantly better than the original, according to results in their paper.

Things were somewhat more complex with the last two problems (5 location load-unload with one or two loading locations). We experimented with parameters over a broad range and determined the following:

* vaps requires a value of $`\beta `$ equal or very nearly equal to 1; these problems are highly non-Markovian, so the Bellman error is not at all useful as a criterion.
* For similar reasons, $`\lambda =1`$ is best for sarsa$`(\lambda )`$.
* Exploration was simplified by setting $`ϵ`$ to 0; empirically, $`c_{\mathrm{𝑚𝑎𝑥}}=1.0`$ and $`c_{\mathrm{𝑚𝑖𝑛}}=0.2`$ worked well for vaps in both problems, and $`c_{\mathrm{𝑚𝑎𝑥}}=0.2`$, $`c_{\mathrm{𝑚𝑖𝑛}}=0.1`$ worked well for sarsa$`(\lambda )`$.
* A base learning rate of $`\alpha _0=0.5`$ worked well for both algorithms in both domains.

Figures 4 and 5 show learning curves for both algorithms, averaged over 50 runs, on the load-unload problem with one or two loading locations. Each run consisted of 1,000 trials. The vertical axis shows the number of steps required to reach the goal, with the terminated trials considered to have taken $`M`$ steps.

On the original load-unload problem, the algorithms perform essentially equivalently. Most runs of the algorithm converge to the optimal trial length of 9 and stay there; occasionally, however it reaches 9 and then diverges. This can probably be avoided by decreasing the learning rate more steeply. When we add the second loading location, however, there is a significant difference. vaps(1) consistently converges to a near-optimal policy, but sarsa(1) does not. The idea is that sometimes, even when the policy is pretty good, the agent is going to pick up the wrong load due to exploration and get punished for it. sarsa will punish all the state-action pairs equally; vaps(1) will punish the bad state-action pair more due to the different principle of credit assignment.

## 4 Conclusions

As Martín and Littman showed, small pomdps can be solved effectively using stigmergic policies. Learning reactive policies in highly non-Markovian domains is not yet well-understood. We have seen that the vaps algorithm, somewhat modified, can solve a collection of small pomdps, and that although sarsa$`(\lambda )`$ performs well on some pomdps, it is possible to construct cases on which it fails. In a generalization of this work, we applied the vaps algorithm to the problem of learning general finite-state controllers (which encompass external-memory policies) for pomdp.

#### Acknowledgments

This work was supported in part by DARPA/Rome Labs Planning Initiative grant F30602-95-1-0020.


text:
# 1 Introduction and summary

## 1 Introduction and summary

Physics with extra dimensions has attracted considerable interest in the last few years. This recent activity is mainly related to the idea that large (or even infinite) extra dimensions are possible if matter fields  and gravity  are localized on a three-brane embedded in the bulk space of higher dimension (see,  for earlier papers where the idea of large extra dimensions was discussed). Hopefully, this approach may provide new insights into the cosmological constant  and hierarchy  problems. Another advantage of the scenario with infinite extra dimensions is that it provides a framework for consistent treatment of those effects which are difficult to incorporate into conventional four-dimensional field theories. Examples of such effects are, e.g., modification of Newton’s law at ultra-large scales , electric charge non-conservation  and mass generation for gauge bosons without the Higgs particle .

One more effect, which is hard to incorporate in conventional quantum field theories, is violation of Lorentz invariance at high energies.Certainly, Lorentz invariance is broken in our world by the cosmological expansion of the Universe. However, this effect is related to the dynamics at low energies and reveals itself only at ultra-large distances of the order of the Hubble scale. On the other hand, one may imagine that Lorentz invariance is broken by high-energy effects in complete theory incorporating quantum gravity. This idea dates back at least to early 80’s  and since that time both laboratory (see, e.g.,  for recent review) and astroparticle (see, e.g.,  for recent discussion) consequences of high-energy violation of Lorentz invariance we elaborated.

The resulting phenomenology can be addressed by introducing about 50 Lorentz-violating parameters in the Lagrangian of the Standard Model , if one restricts oneself to renormalizable CPT even interactions only. The typical experimental limit on the dimensionless parameters in this approach is at the level of $`10^{20}`$ while some Lorentz violating parameters are limited at the level of $`10^{30}`$ .

Consequently, it is of interest to construct a consistent framework capable of predicting possible relations between various Lorentz-violating parameters and explaining their smallness. Scenarios with extra dimensions are natural candidates for such a framework. Another motivation to consider violation of Lorentz invariance in the context of models with extra dimensions comes from cosmology of these models. Namely, the standard assumption is that Lorentz invariance of low-energy effective theory requires bulk space with $`SO(1,3)`$ isometry. This gives rise to an analogue of the flatness problem in multidimensional cosmology, which may be even more severe than the flatness problem in the conventional cosmology . The results of this paper indicate that $`SO(1,3)`$ isometry of the bulk space may be not necessary.

A mechanism to obtain a low-energy effective theory with approximate Lorentz invariance from extra dimensions has been recently suggested in Ref.  (see Refs.  for earlier works in this direction) where it has been proposed that this scenario may help to overcome difficulties of adjustment mechanisms aimed to solve the cosmological constant problem. The basic idea is to consider an asymmetric generalization of the Randall–Sundrum metric, with different warp factors for time and space coordinates,

$$ds^2=a^2(z)dt^2b^2(z)d𝐱_i^2dz^2.$$
(1)
Explicit constructions of this metric in Ref.  involve (charged) black hole solutions in the AdS5 space. It has been assumed that all matter and gauge fields reside on the brane located at $`z=0`$, and, as a result, the only violation of Lorentz invariance comes from the gravitational sector. Approximate four-dimensional gravity emerges at low energies due to the presence of the localized graviton zero mode in the background (1). This mode has been found perturbatively, in the case of small difference between $`a(z)`$ and $`b(z)`$. The complete structure of the low-energy effective gravitational action remains an open issue.

In this paper we concentrate on a different scenario, namely that matter and gauge fields, as well as gravitons, are localized modes of the bulk quantum fields in the background metric of the type (1). We show that violation of Lorentz invariance may reveal itself through a rather peculiar effect at low energies — metastability of all particles. This effect takes place if the extra coordinate $`z`$ is non-compact and if the ratio of warp factors for time and space coordinates $`a(z)/b(z)`$ tends to zero as $`z`$ tends to infinity111Note that both of these properties are absent in the setup of Ref. .. The reason for this effect is that in this case there are no truly localized modes. Instead, four-dimensional particles are described by narrow resonances in the continuum spectrum of the bulk modes. These quasilocalized modes are metastable states that decay into the continuum modes. The rate of the decay depends on the energy of the particle and is naturally suppressed at low energies.

From the point of view of a four-dimensional observer, this type of decays shows up as literal disappearance of the particles. This situation is similar to the metastability of massive particles in the conventional Randall–Sundrum background studied in Ref. . We will closely follow Ref.  both in spirit and in some of the technicalities.

This Lorentz-violating effect is related to the presence of the continuum of bulk modes in models with infinite extra dimensions. It is difficult to incorporate it in the conventional approach  based on the notion of the low-energy effective action. A natural interpretation may be given in the holographic language, where disappearance of a particle may be interpreted as a decay into conformal matter, due to direct (Lorentz violating in our case) couplings between observed fields and conformal sector, which corresponds to the bulk modes in the AdS/CFT correspondence.

The structure of this paper is as follows. In Section 2 we describe a particular class of metrics of the type (1) with $`b(z)=const`$. To construct these metrics as solutions to the Einstein equations we invoke an antisymmetric two-form field interacting with gravity. These solutions may involve an arbitrary number (including zero) of extra compact warped dimensions. These extra dimensions are needed for the localization of bosons in the background metric with $`b(z)=const`$, their presence is not a necessary feature of more general Lorentz violating models with quasistable particles.

In Section 3 we describe tunneling into extra dimensions in the setup of Section 2, by using, as a specific example, fermions localized on the brane by Yukawa-type interactions. We first find the relation between energy and spatial momentum of the quasilocalized fermion mode. This dispersion relation is the same as in special relativity, but with the speed of massless fermions depending on the value of the Yukawa coupling between fermions and scalar field responsible for localization. We then calculate the life-time of the quasilocalized fermion and show that it is large at low energies. Both the speed of the quasilocalized fermion and its life-time do not depend on the number of the extra compact dimensions. In particular, the effect of quasilocalization exists in the absence of extra compact dimensions.

In the end of Section 3 we present an argument showing that only quasilocalized modes can be present in the background of the general type (1) with the described above property ($`a(z)/b(z)0`$ as $`z\mathrm{}`$). Thus, with this type of background metric, tunneling into extra dimensions is a general property of all particles having bulk modes.

In Section 4 we consider quasilocalized scalar, vector and transverse graviton field, calculate their dispersion relations and widths, using solutions constructed in Section 2 as concrete examples of Lorentz violating metrics. We discuss the fine-tuning conditions needed to make the maximum attainable speed of fermions being equal to the speed of photons.

In Section 5 we briefly review phenomenological constraints on our scenario following from cosmic ray physics and searches for forbidden decays, and present our conclusions.

## 2 Example of the Lorentz violating metrics

In our explicit calculations which follow, we use asymmetrically warped metric of the form

$$ds^2=\mathrm{e}^{2k|z|}\left(dt^2\underset{a=1}{\overset{n}{}}(d\theta ^a)^2\right)d𝐱_i^2dz^2$$
(2)
Here $`t`$ and $`𝐱_i`$ ($`i=1,2,3`$) are the usual time and space coordinates and $`z`$ is a coordinate along an infinite extra dimension. The coordinates $`\theta ^a[0,2\pi R_a]`$ parameterize internal compact space which we take in the form of $`n`$-dimensional torus for the sake of simplicity (more generally, it can be an arbitrary compact Ricci-flat manifold). In what follows we assume that the radii of compact extra dimensions $`R_a`$ are smaller than all distance scales of interest and take all fields to be constant along the $`\theta ^a`$-directions.

Let us show how this metric can be obtained as a solution of field equations. Consider the following action involving $`(5+n)`$-dimensional metric $`g_{AB}`$ and antisymmetric two-form field $`B_{AB}`$,

$$S=d^{5+n}x\sqrt{|g|}\left(\frac{M^{n+3}}{2}R\mathrm{\Lambda }+\frac{1}{4}H_{ABC}H^{ABC}\right).$$
(3)
Here $`M`$ and $`\mathrm{\Lambda }`$ are bulk Planck mass and cosmological constant and $`H_{ABC}`$ is the field strength tensor for two-form field $`B_{AB}`$,

$$H_{ABC}=_AB_{BC}+_CB_{AB}+_BB_{CA}.$$

In order to obtain the solution (2) we consider bulk space filled with the following constant background “magnetic” field

$$H_{ijk}=Hϵ_{ijk}.$$
(4)
The non-vanishing components of the energy-momentum tensor corresponding to the magnetic field (4) in the background (2) are

$`T_{00}=T_{\theta _a\theta _a}={\displaystyle \frac{3}{2}}H^2\mathrm{e}^{2k|z|},`$ (5)
$`T_{ii}=T_{zz}={\displaystyle \frac{3}{2}}H^2.`$ (6)
In addition we assume that there is a source brane located at $`z=0`$ with the following non-vanishing components of the energy-momentum tensor

$$\tau _{00}=\tau _{\theta _a\theta _a}=\sigma _0\delta (z),\tau _{ii}=\sigma _x\delta (z).$$
(7)
The field equations for the two-form field are satisfied by the Ansatz (2) and (4). The Einstein equations lead to the following set of fine-tuning conditions,

$`2\mathrm{\Lambda }=(n+1)^2k^2M^{n+3},H^2=(n+1)k^2M^{n+3},`$ (8)
$`\sigma _0=2knM^{n+3},\sigma _x=2k(n+1)M^{n+3}.`$ (9)
This fine-tuning is somewhat similar to the fine-tuning inherent in the original Randall–Sundrum proposal.

A few comments are in order here. First, our solution may be considered as a limiting case of the hedgehog solutions with p-form fields described in Ref. . The interpretation is quite different, however — dimensions which were treated as small and compact in Ref.  become ordinary infinite (or cosmologically large) spatial dimensions in our setup. On the other hand, small compact extra dimensions $`\theta ^a`$ in the metric (2) correspond to the usual spatial coordinates in the setup of Ref. .

Second, the energy-momentum tensor of the brane involved in our construction violates the positive energy condition, as is clear from Eq. (9) (the coefficient $`\omega `$ in the equation of state $`p=\omega \rho `$ is smaller than $`1`$ for the $`𝐱`$-component of pressure). Presumably, this will not lead to an instability as long as $`_2`$ orbifold symmetry $`zz`$ is imposed in analogy to the first Randall–Sundrum model . It is worth noting that the same type of exotic sources was involved in the construction of Lorentz-violating setups in Ref. . It remains to be understood whether exotic matter violating positive energy condition is a necessary ingredient of scenarios with approximate low-energy Lorentz invariance and localized gravity.

We note finally that, as pointed out in Ref. , metric of the type (2) has an orbifold singularity at $`z=\mathrm{}`$ in the presence of warped extra dimensions $`\theta ^a`$. We need these extra compact dimensions in order to obtain quasilocalized bosonic fields (see Section 4). However, the main effect studied in our paper (tunneling into extra dimensions in the geometry of the type (1)) is not related to this singularity as will be demonstrated in the next Section. This effect is present in a more general case of non-constant $`b(z)`$ as well. In that case one need not add extra compact dimensions, leading to orbifold singularity, because graviton and scalar particles are localized by pure gravity, and gauge field may be localized by some other mechanism (e.g., Dvali–Shifman mechanism ). Finally, as shown in Ref. , one of possible resolutions of this singularity in the context of string theory preserves the key feature of the Randall–Sundrum model relevant to the tunneling into extra dimension — a continuous spectrum of bulk modes starting from zero.

## 3 Tunneling into extra dimension

As the first example of tunneling into extra dimension let us consider quasilocalized fermions. For concreteness, in explicit calculations we make use of the metrics constructed in Section 2. In the end of this Section we show that tunneling into extra dimensions is a general feature of all particles having bulk modes, provided the Lorentz violating metrics (1) have the properties described in Introduction.

In the background of the type (2), fermionic fields are not localized on the brane by the gravitational interactions only, for the same reason as in the original Randall–Sundrum scenario  (see also ). Hence, one invokes the localization mechanism of Refs. . The simplest setup is as follows. One considers a domain wall formed by some scalar field $`\varphi _k`$. This scalar field has a double-well potential with two degenerate vacua at $`\varphi _k=\pm v`$; the domain wall separates the region $`\varphi _k=v`$ at $`z<0`$ from the region $`\varphi _k=v`$ at $`z>0`$. In flat space, a fermionic field which has a Yukawa coupling to the scalar, $`g\varphi _k\overline{\mathrm{\Psi }}\mathrm{\Psi }`$, has an exact zero mode in the domain wall background. This zero mode is topological and its existence does not depend on the details of the profile of the scalar field across the wall. Therefore, it exists also for infinitely thin wall,

$$\varphi _k(z)=v\text{sgn}(z),$$

which is the case we consider in what follows.

The effective action for fermionic fields which are constant along the compact extra dimensions reads (cf. Ref. )

$$S=𝑑zd^4x\sqrt{|g|}\overline{\mathrm{\Psi }}\left(i\gamma ^\alpha _\alpha \frac{ikn\text{ sgn}(z)}{2}\gamma ^z+g\varphi _k\right)\mathrm{\Psi },$$
(10)
where $`_\alpha `$ is the spinor covariant derivative with respect to the metric (2) and $`\alpha =0,i,z`$. For the sake of simplicity we limit ourselves to the case of massless fermion. The treatment of massive fermions would require the introduction of a fermionic doublet in the bulk theory and would make our formulae more complicated without changing physics (cf. Ref. ). After the change of variables

$$\mathrm{\Psi }=\mathrm{e}^{k(n+1)|z|/2}\stackrel{~}{\mathrm{\Psi }}$$

one obtains the following Dirac equation for the 4-spinor $`\stackrel{~}{\mathrm{\Psi }}`$,

$$\left[E\mathrm{e}^{k|z|}\gamma ^0+\gamma ^ip_i+\gamma ^5_zg\varphi _k(z)\right]\stackrel{~}{\mathrm{\Psi }}=0.$$
(11)
which is valid for any number of extra compact dimensions (and in their absence as well). In order to see that there exists a quasilocalized resonance, let us find the complex eigenvalue at which there exists a solution to Eq. (11) with the radiation boundary conditions imposed at $`z\pm \mathrm{}`$. It is convenient to choose the following basis for $`\gamma `$-matrices,

$`\gamma ^0=\left(\begin{array}{cc}0& 1\\ 1& 0\end{array}\right),\gamma ^1=\left(\begin{array}{cc}i\sigma _2& 0\\ 0& i\sigma _2\end{array}\right),\gamma ^2=\left(\begin{array}{cc}0& 1\\ 1& 0\end{array}\right),`$ (18)
$`\gamma ^3=\left(\begin{array}{cc}i\sigma _3& 0\\ 0& i\sigma _3\end{array}\right),\gamma ^5=\left(\begin{array}{cc}\sigma _1& 0\\ 0& \sigma _1\end{array}\right)`$ (23)
and separate the spinor $`\stackrel{~}{\mathrm{\Psi }}`$ into the up and down components

$$\stackrel{~}{\mathrm{\Psi }}=\left(\begin{array}{c}\psi \\ \chi \end{array}\right).$$

Furthermore, we will make use of invariance of our metric under spatial rotations and consider a particle moving in the $`𝐱^3`$-direction, i.e. choose the spatial momentum in the form

$$𝐩=(0,0,p)$$

In terms of two-component spinors $`\psi `$ and $`\chi `$, Eq. (11) translates into a set of coupled equations

$`E\mathrm{e}^{k|z|}\chi +ip\sigma _3\psi \sigma _1_z\psi g\varphi _k\psi =0,`$ (24)
$`E\mathrm{e}^{k|z|}\psi ip\sigma _3\chi +\sigma _1_z\chi g\varphi _k\chi =0.`$ (25)
After eliminating $`\psi `$ one obtains the second order equation for $`\chi `$,

$$\left[E^2\mathrm{e}^{2k|z|}+_z^2k\text{sgn}(z)\left(_z(p\sigma _2+g\varphi _k\sigma _1)\right)g\varphi _k^{}\sigma _1(g\varphi _k^2+p^2)\right]\chi =0.$$
(26)
The differential operator in the left hand side of Eq. (26) coincides with the differential operator entering the second order equation for massive fermions (with mass equal to $`p`$) in the Randall–Sundrum background; the latter has been considered in Ref. . Consequently, we may make use of the result of that work and find that there is a complex eigenvalue determined by the following equation at $`pgv`$,

$$\frac{H_{M/k1/2}^{(1)}\left(E/k\right)}{H_{M/k+1/2}^{(1)}\left(E/k\right)}=\frac{p}{2gv},$$

where $`M`$ is defined in such a way that

$$gv+ip=M\mathrm{e}^{i\alpha }$$

with real $`M`$ and $`\alpha `$. We are interested in the case of small energies and momenta $`E,pk,gv`$. In this case one expands the Bessel functions at small values of the argument and finds

$$E=E_0\frac{i\mathrm{\Gamma }}{2}$$

with

$$E_0=\left(1\frac{k}{2gv}\right)p$$
(27)
and

$$\mathrm{\Gamma }=\frac{2\pi E_0}{[\mathrm{\Gamma }\left(gv/k+1/2\right)]^2}\left(\frac{E_0}{2k}\right)^{2gv/k1}.$$
(28)
Equations (27) and (28) describe a narrow resonance at low energies, provided

$$\frac{gv}{k}>\frac{1}{2}$$

This resonance can be interpreted as a four dimensional metastable particle. The $`\chi `$-component of its wave function is determined by Eq. (26) and $`\psi `$-component of its wave-function may be found then from Eq. (25).

The speed of the quasilocalized massless fermion is equal to

$$c_f=1\frac{k}{2gv}$$

as is clear from Eq. (27). This speed depends on the parameter $`k/2gv`$ and is always smaller then the speed of a massless point-like particle tightly bound to the brane and moving along $`z=0`$ (the latter case corresponds to the limit $`gv\mathrm{}`$). This does not necessarily imply that the maximum speed is not the same for different quasilocalized particles. As we will see below, quasilocalized scalar field, photon and graviton in the background (2) have equal speeds (smaller than the speed of a tightly bound massless particle). In order that the quasilocalized fermion has the same speed, additional fine-tuning of parameters is needed.

Another non-trivial manifestation of the violation of Lorentz invariance is non-zero imaginary part of the fermion energy, as given by Eq. (28). This imaginary part determines the probability for quasilocalized fermion to tunnel into extra dimension. The tunneling probability is small at low energies and grows with energy, as follows from Eq. (28). This effect is present in the absence of extra compact dimensions (at $`n=0`$) and, hence, is not related to the orbifold singularity at infinity.

Let us now present an argument indicating that metastability of all particles having bulk modes is a general property of the metrics of the type (1) with $`a(z)/b(z)0`$ as $`z\mathrm{}`$. Indeed, a mode equation for a bulk field $`\mathrm{\Phi }`$ in the background of this type may be presented in the following general form

$$\left[\mathrm{\Delta }(z)+p^2f(z)E^2\right]\mathrm{\Phi }(z)=0,$$
(29)
where $`\mathrm{\Delta }(z)`$ is a positive semi-definite differential operator and the function $`f(z)=a^2(z)/b^2(z)`$ vanishes at infinity. The key property of models with infinite extra dimensions, essential for our argument, is that the operator $`\mathrm{\Delta }(z)`$ has a continuous spectrum of energies starting from some $`E^2=E_{min}^2`$ at $`p=0`$. This property may be considered as a definition of infinite extra dimension and means that the energy of the particle related to its motion along the $`z`$-direction is not quantized, at least starting from some large enough value. Note that in most of known models with infinite extra dimensions and localized gravity, the continuum spectrum starts from zero energy222See, however, Refs. . We thank P. Tinyakov for pointing out the possibility of non-zero $`E_{min}`$. $`E_{min}=0`$. At non-zero $`p^2>0`$, the term $`p^2f(z)`$ in the left hand side of Eq. (29) vanishes at large $`z`$ and thus it is irrelevant at infinity. Since the continuum eigenvalues are determined by the large $`z`$ asymptotics only, Eq. (29) has the same continuum spectrum for all $`p^2`$ including $`p^2=0`$. This implies that there are no truly localized modes with $`E^2>E_{min}^2`$ (there are no truly localized bound states embedded in the continuum) and all four-dimensional particles become unstable above this energy. A similar argument was presented in Ref.  for massive fields in the Randall–Sundrum background.

## 4 Quasilocalized bosons

As another example of tunneling into extra dimension, we consider propagation of the scalar field $`\varphi `$ with mass $`\mu `$ in the background metric (2). The corresponding field equation has the following form

$$\left[_z^2+(n+1)k\text{sgn}(z)_z+\mu ^2+p^2E^2\mathrm{e}^{2k|z|}\right]\varphi =0,$$
(30)
where $`p`$ is the spatial momentum and $`E`$ is the energy. This equation is analogous to the field equation for massive scalar field in the Randall–Sundrum background. At a given value of $`p^2>0`$, there is a continuum spectrum of modes with arbitrary energies $`E^2>0`$ and no localized modes. However, as in the case of massive fields in the Randall–Sundrum metric, there exists a resonance mode which describes quasilocalized particle. Indeed, let us show that the mode equation, Eq. (30), has a complex eigenvalue when the radiation boundary conditions are imposed at $`z\pm \mathrm{}`$. The solution to Eq. (30) which satisfies the radiation boundary conditions is

$$\varphi (z)=c\mathrm{e}^{k(n+1)|z|/2}H_\nu ^{(1)}\left(\frac{E}{k}\mathrm{e}^{k|z|}\right),$$
(31)
where $`H_\nu ^{(1)}(x)`$ is the Hankel function, the constant $`c`$ is determined by the normalization condition and

$$\nu =\sqrt{\left(\frac{n+1}{2}\right)^2+\left(\frac{p}{k}\right)^2+\left(\frac{\mu }{k}\right)^2}$$
(32)
The first derivative of $`\varphi (z)`$ should be continuous at $`z=0`$, as is clear from Eq. (30),

$$_z\varphi (+0)_z\varphi (0)=0.$$

The latter condition implies the following dispersion relation

$$\frac{E}{k}\frac{H_{\nu 1}^{(1)}\left(\frac{E}{k}\right)}{H_\nu ^{(1)}\left(\frac{E}{k}\right)}+\frac{n+1}{2}\nu =0.$$
(33)
Let us consider small energies and momenta $`\mu ,E,pk`$. Then expanding the Bessel function at small argument one finds

$$E=E_0\frac{i}{2}\mathrm{\Gamma }$$
(34)
with

$$E_0^2=\frac{n1}{n+1}(p^2+\mu ^2)$$
(35)
and

$$\mathrm{\Gamma }=\frac{2\pi E_0}{\mathrm{\Gamma }(n/2)\mathrm{\Gamma }(n/21)}\left(\frac{E_0}{2k}\right)^{n1}.$$
(36)
We see that at $`n>1`$ (this is the case we consider in the rest of the Section) the width of the resonance is much smaller than $`E_0`$. As a result, this resonance can be interpreted as a four-dimensional metastable particle. This particle decays through tunneling into extra dimensions. Violation of Lorentz invariance reveals itself in the fact that the decay rate grows with the energy of the particle. The factor of $`(n1)/(n+1)`$ in the dispersion relation (35) implies that the maximum speed of this particle is smaller than the speed of a tightly bound massless particle moving along the brane at $`z=0`$. In order that this speed be equal to the speed of the quasilocalized fermions, the following fine-tuning condition should be satisfied

$$1\frac{k}{2gv}=\sqrt{\frac{n1}{n+1}}.$$
(37)
It would be desirable to obtain this fine-tuning condition as a consequence of some symmetry in the underlying theory. Another solution of this problem may be to construct a setup where fermions are localized by gravity only, like all other fields. See, e.g., Refs.  for recent attempts in this direction. Alternatively, one can localize fermions on the additional spectator brane like in the Lykken–Randall scenario . This brane should be located in the place where the maximum speed of tightly bound particles in $`𝐱_i`$ directions is the same as the speed of the quasilocalized bosonic modes. Finally, one can consider metrics of the type (1), i.e., more general than those described in Section 2. In that case the (approximate) equality between maximum attainable speeds of fermions and bosons may be attributed to the small difference between the warp factors $`a(z)`$ and $`b(z)`$.

Let us now consider the propagation of a massless vector field $`V_A`$ in the background (2). The field equation in curved space has the form

$$_A\left(\sqrt{|g|}F^{AB}\right)=0,$$
(38)
where $`F_{AB}=_AV_B_BV_A`$. To classify propagating modes of the vector field in the background (2), we find it convenient to work in the gauge

$$V_0=0.$$

In this gauge the $`(0)`$-component of Eq. (38) takes the following form,

$$_0\left[_z\left(\mathrm{e}^{(n1)k|z|}V_z\right)+\mathrm{e}^{(n1)k|z|}_iV_i\right]=0$$
(39)
The fact that the combination in the square bracket in the left hand side of Eq. (39) is constant in time makes it possible to use the residual gauge freedom to set this combination to zero,

$$_z\left(\mathrm{e}^{(n1)k|z|}V_z\right)+\mathrm{e}^{(n1)k|z|}_iV_i=0.$$
(40)
Then one can rewrite the $`(z)`$-component of Eq. (38) in the following form,

$$\left[_z^2+(n1)k\text{sgn}(z)_z+p^2E^2\mathrm{e}^{2k|z|}+2(n1)k\delta (z)\right]V_z=0.$$
(41)
At zero energy and momentum, the field $`V_z=const`$ ceases to be a solution of this equation because of the presence of the extra delta-functional term in the left hand side of Eq. (41), which is absent in Eq. (30). As a result, Eq. (41) does not admit localized modes at $`E=p=0`$ and one has no reason to expect the presence of narrow resonances at small but non-vanishing energy and momenta. Straightforward calculation analogous to one outlined in the beginning of this Section shows that such resonances are indeed absent. Consequently, the $`z`$-component of the vector field (or, equivalently, the longitudinal component of the photon, see Eq. (40)) does not have (quasi)localized modes in our setup.

Now, it is straightforward to check that transverse components $`V_i`$ satisfy the same mode equation as the massless scalar field, Eq. (30) with $`\mu =0`$. Consequently, a quasilocalized transverse photon with the same speed of light as the maximum speed of the scalar particle exists at $`n>1`$.

Components $`V_{\theta ^a}`$ of the vector field do not couple to matter as long as there are no currents along the extra compact dimensions. However, for the sake of completeness, let us consider the propagation of these components as well. Taking ($`\theta ^a`$)-component of Eq. (38) one obtains

$$\left[_z^2+(n1)k\text{sgn}(z)_z+p^2E^2\mathrm{e}^{2k|z|}\right]V_{\theta ^a}=0.$$
(42)
Comparing this equation with the equation (30) for the scalar field we see that these components correspond to quasilocalized scalar particle with the maximum speed equal to $`\sqrt{(n1)/(n3)}`$ at $`n>3`$.

Let us now briefly discuss the quasilocalized graviton mode in our setup. The mode equation for transverse traceless gravitational waves may be obtained by considering metric perturbation of the form

$$\delta ds^2=h(z)h_{ij}(𝐱,t)d𝐱^id𝐱^j$$

with

$$_ih_{ij}=\delta ^{ij}h_{ij}=0$$

It is straightforward to check that these perturbations decouple from perturbations of other components of the metric and of the two-form field $`B_{AB}`$ and satisfy the same mode equation as the scalars, Eq. (30). This is the same situation as in the Randall–Sundrum model  and as in the setup of Ref. . Consequently, there exists a quasilocalized graviton in our setup with the same low-energy properties as the massless scalar boson. We will not study the excitations of other components of the metric and of the two-form $`B_{AB}`$ here; this study would be necessary to understand the complete structure of the low-energy effective action for gravity in our setup and in the setup of Ref. . We hope to return to this issue in future.

We would like to make only a couple of comments in this regard. The first one is that in Ref. , there were suggested two possible explanations of the difference between the speed of the localized graviton and the speed of a tightly bound massless particle moving along the brane at $`z=0`$. The first possibility is that the effective action for gravity explicitly breaks the 4D Lorentz invariance at $`z=0`$, as different coefficients for time and space coordinates in the kinetic term of the graviton are introduced. The second possibility is that the effective action violates the weak equivalence principle by the presence of some extra fields which couple differently to matter and gravity. This may force gravitational waves to propagate differently than the tightly localized particle.

To infer which of these two possibilities is actually realized, we note that the correct value of the maximum speed of the quasilocalized bosons can be obtained in the following empiric way. The scalar field action in the background (2) has the following form for the modes which are constant over compact coordinates $`\theta _a`$

$$S_{sc}𝑑t𝑑𝐱𝑑z\left(\mathrm{e}^{(n1)k|z|}\dot{\varphi }^2\mathrm{e}^{(n+1)k|z|}\left(_i\varphi \right)^2\mathrm{e}^{(n+1)k|z|}\mu ^2\varphi ^2\right).$$
(43)
Note that the field equation (30) admits a normalizable mode $`\varphi (z)=const`$ with zero energy when $`p=\mu =0`$. To obtain the effective four-dimensional action for quasilocalized modes of low energy and momentum, one can plug a field $`\varphi (t,𝐱)`$ independent of $`z`$ in the action (43) and integrate over $`z`$. Then one will obtain precisely the dispersion relation (35) due to different coefficients in front of time and space derivatives in Eq. (43).

The fact that the above empirical way to obtain the low energy effective action for the scalar (and other bosons) gives the correct value for its speed due to the difference in the coefficients in front of time and space derivatives suggests that it is the effect of the first type that takes place (at least in our setup). Certainly, our observation does not exclude the possibility that the second effect may also be present.

The second comment is that one may worry that our setup suffers from a van Dam-Veltman-Zakharov (vDVZ) discontinuity as other theories with quasistable graviton . In this regard it worth noting, that, as it was argued recently, this discontinuity is likely to be an artifact of the linear approximation . Secondly, and more relevant for the class of model we consider, this discontinuity is due to the fact that quasilocalized graviton has extra polarization states in model discussed in . In our models, graviton is strictly massless and may decay not because of non-zero mass but due to the presence of bulk modes with smaller propagation velocities. Consequently, one has no a priori reason to expect extra polarizations, which would give rise to the vDVZ discontinuity. Explicit analysis presented above demonstrates that these polarizations are indeed absent in the photon case. The analysis of the linearized gravity in asymmetrically warped spaces which is necessary to address the same issue for a graviton is beyond the scope of this work.

## 5 Discussion

In this concluding section, let us first briefly discuss potential experimental constraints on the scenario suggested in this paper, by making use of the brane world solutions of Section 2 as a toy model. We assume that the fine-tuning condition (37) is satisfied and neglect possible non-universality in the maximum attainable speed of different quasilocalized particles which may appear due to quantum corrections. The latter issue requires a separate study. Then the limits on the parameters of our model are related to the instability of energetic particles and come from searches for forbidden decays and from physics of high-energy cosmic rays. In all our estimates we take the parameter $`k`$ to be equal to the Planck mass $`M_{Pl}10^{19}`$ GeV. Certainly, one can make all limits weaker by taking larger values of $`k`$. Also, one may expect that these constraints are weaker for more general metrics of the type given in Eq. (1), if the difference between warp factors $`a(z)`$ and $`b(z)`$ is not so drastic as in Eq. (2).

The first limit comes from the stability of electron. We estimate the width of electron against the decay $`e^{}nothing`$ by plugging $`E_0=m_e`$ in the expression (28) for the width of massless fermion and finding the parameter $`gv/k`$ from the fine-tuning condition (37). As a result one finds that for two extra compact dimensions (the smallest number for which the localization of bosons takes place in our toy model), the life-time of electron is of order 100 years, which is certainly unacceptable. However, for three extra compact dimensions this life-time is $`910^{25}`$ years which is larger than the current experimental limit $`\tau _e>4.210^{24}`$ years .

It is not quite obvious that Eq. (28) can be directly applied for estimating the proton life-time. Proton is a composite object and this fact may lead to an additional suppression of the decay rate. A naive application of Eq. (28) with $`E_0=m_p`$ excludes our setup with three extra compact directions, but for four extra compact dimensions gives the value $`\tau _p=9.210^{34}`$ years which is much larger then the experimental limit on the decay $`pnothing`$, $`\tau _p>510^{26}`$ years .

Another set of limits comes from the high-energy cosmic ray data. Observation of $`20`$ TeV photons from the distant blazar Markarian 501  with redshift $`z0.3`$ implies that the life-time of a photon of this energy cannot be much smaller than the age of the Universe (see Ref.  for a recent discussion of limits on the Lorentz-violating parameters coming from the observation of Markarian 501). By making use of the expression (36) for the life-time of bosons, we see that this observation excludes theories with the number of extra compact dimensions smaller than four, just like the proton decay.

The strongest limit may come from the observation of ultra-high energy cosmic rays. If the recently found correlation  of cosmic rays with energies $`E510^{19}`$ eV with BL Lacertae objects is confirmed by further data, then particles of this energy should be stable enough to travel as large distances as $`600`$ Mpc. By making use of Eqs. (36) and (28) one finds that the number of extra compact dimensions in our toy model should be as large as 7, irrespectively of whether primary particles are fermions or bosons (assuming that one can use expressions (36) and (28) for composite objects like proton). This would mean that the total number of dimensions is equal to or greater than 12 and prevent embedding this brane world into string/M-theory. Note, however, that the life-times of particles strongly depend on the parameter $`k`$. Taking this parameter equal to $`5M_{Pl}`$ will allow to have the total number of dimensions equal to 11, a value favored by M-theory.

To conclude, we constructed a setup with an infinite extra dimension where bulk space does not exhibit even approximate $`SO(1,3)`$ isometry but low-energy effective theory is approximately Lorentz invariant. This Lorentz invariance is automatic for scalar field, photon and graviton. Additional fine-tuning of parameters is needed to assure that the maximum attainable speed of fermions is the same. It worth noting that with this fine-tuning, $`SO(1,3)`$ is still strongly violated the bulk.

A peculiar property of our setup is that there are no truly localized modes — all four-dimensional particles are narrow resonances in the spectrum of bulk modes. These resonances have finite probabilities to tunnel into extra dimension. These probabilities depend on the energies of the particles.

In the (generalized) holographic interpretation where bulk modes are described as a hidden (quasi)conformal sector (see, e.g., Refs. ) this type of decay may be attributed to the direct coupling

$$\varphi 𝒪_\varphi $$

between quasilocalized fields $`\varphi `$ and operators $`𝒪_\varphi `$ from the conformal sector. This coupling may be Lorentz invariant like in the case of massive particles in the Randall –Sundrum background  or violating Lorentz invariance like in the examples of this paper. Tunneling into extra dimension is interpreted then as a decay into conformal matter. It worth noting that in this language, emergence of the Lorentz invariant sector at low energies, in a theory without Lorentz invariance at high energies, appears rather non-trivial.

Possibly, one can consider bulk metrics even without $`SO(3)`$ symmetry under spatial rotations. For instance, one can take metric with time and one of the space coordinates warped and two other coordinates not warped. Presumably, one will be able to fine-tune parameters so that all maximum attainable speeds will be equal for different fields in this setup as well. An interesting property of these metrics is that the tunneling probability will depend not only on the energy of the particle, but also on the direction of its motion.

The author is indebted to T. Gherghetta, M. Giovannini, M. Libanov, V. Rubakov, M. Shaposhnikov, S. Sibiryakov, P. Tinyakov and S. Troitsky for helpful discussions of different topics related to this work. The author thanks Institute of Theoretical Physics, University of Lausanne, where this work was done, for kind hospitality. This work was supported in part by RFBR grant 99-01-18410, by the Council for Presidential Grants and State Support of Leading Scientific Schools, grant 00-15-96626, by CRDF grant (award RP1-2103) and by Swiss Science Foundation grant 7SUPJ062239.


text:
# Erratum: Bremsstrahlung from an Equilibrating Quark-Gluon Plasma [Phys. Rev. C62, 014902 (2000)]

PACS numbers: 12.38.Mh, 24.10.Nz, 25.75.-q

Due to a numerical error the rates for bremsstrahlung (4) and annihilation with scattering (5) in , taken from , were overestimated by a factor of 4 . The quantities $`J_{T,L}`$ appearing in (4) and (5) read correctly $`J_T=1.11`$, $`J_L=1.07`$ for two quark flavors $`N_f=2`$ and $`J_T=1.20`$, $`J_L=1.13`$ for $`N_f=3`$, respectively. The photon spectra at RHIC and LHC in equilibrium and chemical non-equilibrium calculated using the corrected rates are shown in Fig.1-4, which replace the corresponding figures in .

The new conclusions following from these figures are the following. The total photon yield from the quark-gluon plasma at RHIC and LHC is reduced by a factor 2-3 compared to . At RHIC the non-equilibrium spectrum is smaller by a factor of 5 at $`p_T=1`$ GeV, but larger by a factor of 2 at 5 GeV, while it is suppressed at LHC by a factor of about 3 at all momenta compared to the equilibrated case, if the same initial energy density is assumed. Therefore we conclude as in  that the equilibrium spectrum is similar to the equlibrating. The annihilation-with-scattering contribution to the spectrum dominates in equilibrium for all momenta, whereas in the equilibrating case the one-loop (Compton scattering, anihilation) contribution is the largest for all momenta and initial conditions (SSPC, HIJING-I, HIJING-II) in contrast to the results found in . The thermal emission in chemical non-equilibrium exceeds now the prompt photon yield for $`p_T<3.5`$ GeV at RHIC as well as LHC.