format: text

text:
# Growth of a peanut shaped bulge via resonant trapping of stellar orbits in the vertical Inner Lindblad Resonances

## 1 Introduction

The existence of spiral galaxy bulges with boxy or peanut shapes has been recognized for many years through the study of edge-on galaxies (e.g., Burbridge & Burbridge 1959; de Vaucouleurs 1974). Recent work has found that edge-on galaxies with prominent boxy or peanuts shaped bulges in most cases show kinematic evidence for non-circular motions associated with a bar, whereas galaxies lacking boxy or peanut shaped bulges do not show kinematic evidence for a bar (Bureau & Freeman, 1999; Merrifield & Kuijken, 1999). A few cases exist where highly inclined systems exhibit both bars and peanuts (e.g., NGC 7582, Quillen et al. 1997 and NGC 4442, Bettoni & Galletta 1994). The link between a galactic bar and a boxy/peanut shaped bulge has been firmly established.

Mechanisms proposed to explain the formation of boxy and peanut shaped structures include accretion of small satellite galaxies (Binney & Petrou, 1985), bar-buckling via the fire-hose type instability (Toomre, 1966; Raha et al., 1991; Merritt & Sellwood, 1994; Fridman & Polyachenko, 1984) and resonant heating (Pfenniger, 1984, 1985; Combes et al., 1990). The fire-hose (bar-buckling) instability is a global time dependent instability which relies on the centrifugal force caused by highly eccentric orbits in the plane caused by the bar and is damped via a dissipative process similar to Landau damping (Fridman & Polyachenko, 1984). The resonant heating model has been inspired by the study and classification of orbit families in 3-dimensional systems (Pfenniger & Friedli, 1991) and through the phenomena displayed in N-body simulations (Combes et al., 1990).

No study has yet considered the evolution of 3 dimensional stellar orbits in a barred system that is varying with time. In this paper we investigate the possibility that the growth of the bar itself traps particles in the vertical Inner Lindblad Resonances (ILRs). We develop a simple Hamiltonian analytical model for these resonances and use it to predict the vertical distribution of a bar following the growth of the bar. Our approach of considering resonance trapping is similar to that explored by Sridhar & Touma (1996) for disk heating. However their model is for the general case of a growing axisymmetric disk, whereas our model applies specifically to a growing bar perturbation.

## 2 Dynamical Model

To exihibit resonant phenomena Contopoulos (1975) showed that the Hamiltonian must be expanded to at least third order in the epicyclic approximation. When this is done there are higher order cross terms in the action variables and the model will exhibit phenomenology such as the bifurcation of orbit families seen in characteristic diagrams which are created through numerical integration of orbits.

We first consider the Hamiltonian lacking non-axisymmetric or time dependent perturbations

$$H_0(L,p_r,p_z;\theta ,r,z)=\frac{L^2}{2r^2}+\frac{p_r^2}{2}+\frac{p_z^2}{2}+V_0(r,z)$$
(1)
where $`V_0`$ is the gravitational potential which is assumed to be axisymmetric and $`L,p_r,p_z`$ are the momenta conjugate to the cylindrical coordinates $`\theta ,r,z`$. For the 2 dimensional problem restricted to the Galactic plane ($`z=0,p_z=0`$), Contopoulos (1975) showed how to put the Hamiltonian in the form

$$K_0(I_1,I_2;\theta _1,\theta _2)=h+\kappa I_1+\mathrm{\Omega }I_2+aI_1^2+2bI_1I_2+cI_2^2\mathrm{}$$
(2)
in a third order epicyclic approximation. The action variables $`I_1=\frac{1}{2\pi }\dot{r}𝑑r`$ and $`I_2=J_0J_c`$ are integrals of motion. $`J_0`$ is the particle’s angular momentum, and $`J_c`$ is the angular momentum of a particle in a circular orbit at a radius $`r_c`$ which is the radius of a circular orbit with energy $`h`$. $`\theta _1`$ is the epicyclic angle and $`\theta _2`$ is the azimuth of the epicyclic center. $`\theta _1`$ and $`\theta _2`$ are the angle variables conjugate to $`I_1`$ and $`I_2`$. $`\kappa `$ is the epicyclic frequency and $`\mathrm{\Omega }`$ the angular rotation rate; both are evaluated at $`r=r_c,z=0`$.

To extend the theory into the third dimension we must add additional terms to the restricted Hamiltonian which depend upon $`z`$;

$$H_0=K_0+H_{0z}.$$
(3)
We expand the potential $`V_0`$ about $`z`$ and $`r_c`$ defining the variables

$`\nu ^2`$ $`=`$ $`{\displaystyle \frac{^2V_0}{z^2}}`$ (4)
$`\lambda `$ $`=`$ $`{\displaystyle \frac{^4V_0}{z^4}}`$
which are evaluated at $`z=0,r=r_c`$. Inserting this into the full Hamiltonian we find

$`H_{0z}(`$ $`L,p_r,p_z;\theta ,r,z)={\displaystyle \frac{1}{2}}(p_z^2+\nu ^2z^2)+{\displaystyle \frac{\lambda z^4}{4!}}`$
$`+{\displaystyle \frac{\nu ^2z^2}{2}}\left[r_c\mathrm{\Omega }^2(rr_c)+{\displaystyle \frac{1}{2}}(\kappa ^23\mathrm{\Omega }^2)(rr_c)^2\right]`$
where we have made use of the relations $`\mathrm{\Omega }^2=\frac{1}{r}\frac{V_0}{r}`$, $`\kappa ^2=\frac{^2V_0}{r^2}+3\mathrm{\Omega }^2`$ and both $`\mathrm{\Omega }`$ and $`\kappa `$ are evaluated at $`r=r_c,z=0`$.

For this Hamiltonian following Contopoulos (1975) we can choose a third action variable $`I_3`$ to represent the amplitude of vertical oscillations.

$`z`$ $``$ $`\left({\displaystyle \frac{2I_3}{\nu }}\right)^{1/2}\mathrm{cos}(\theta _3)`$ (6)
$`{\displaystyle \frac{dz}{dt}}`$ $``$ $`\left(2I_3\nu \right)^{1/2}\mathrm{sin}(\theta _3)`$
where $`\nu `$ is the vertical oscillation frequency and $`\theta _3`$ is the angle associated with the vertical oscillations. The above expressions are given to first order in $`I_3^{1/2}`$ and require higher order terms to be exact.

After a couple of canonical transformations (see appendix) the Hamiltonian to second order in $`I_3^{1/2}`$ takes the form

$$H_{0z}(I_1,I_2,I_3;\theta _1,\theta _2,\theta _3)=\nu I_3+\frac{\lambda I_3^2}{32\nu ^2}+(\kappa ^23\mathrm{\Omega }^2)\frac{\nu I_1I_3}{4\kappa }.$$
(7)
Since this is independent of the angles $`\theta _1,\theta _2,\theta _3`$, the momenta $`I_1,I_2,I_3`$ are conserved quantities and we say we have put the non-perturbed Hamiltonian, (Eqn. 2), in action angle form.

### 2.1 Perturbation

We now consider the form of the perturbation of the gravitational potential caused by a bar. In the plane of the galaxy we can expand the gravitational potential in terms of Fourier components

$$V_1(r,\theta ,z=0)=\underset{m}{}A_m\mathrm{cos}(m(\theta \mathrm{\Omega }_bt))$$
(8)
(for example see Quillen et al. 1994). The strongest term due to the bar should be the $`m=2`$ term with $`A_2<0`$ so that the bar major axis lies along $`\theta \mathrm{\Omega }_bt=0,\pi .`$ For small $`z`$ we assume that the potential can be expanded so that we can write for each $`m`$ Fourier component

$$V_{1m}(r,\theta ,z)=A_m\mathrm{cos}(m(\theta \mathrm{\Omega }_bt))\left(1\frac{z^2}{2h_1^2}\right)$$
(9)
where $`h_1`$ is the scale height of the perturbation. We also assume that $`A_m`$ and $`h_1`$ vary only weakly with radius. Since $`z^2\frac{2I_3}{\nu }\mathrm{cos}^2(\theta _3)`$ we can rewrite the perturbation components to second order in $`I_3^{1/2}`$

$$V_{1m}(I_3,\theta _3;I_2,\theta _2)=\frac{A_mI_3}{4h_1^2\nu }\mathrm{cos}(m(\theta _2\mathrm{\Omega }_bt)\pm 2\theta _3)+\mathrm{}$$
(10)
We focus on the particular resonance that we expect is important and ignore all the other terms.

$`H(`$ $`I_2,I_3;\theta _2,\theta _3)=\nu I_3+{\displaystyle \frac{\lambda }{32\nu ^2}}I_3^2+\mathrm{\Omega }I_2`$ (11)
$`+I_3{\displaystyle \frac{A_m}{4h_1^2\nu }}\mathrm{cos}(m(\theta _2\mathrm{\Omega }_bt)2\theta _3)+\mathrm{}`$
To go into a coordinate system with an angle that librates when a particle is trapped in the resonance we do a canonical transformation with the following generating function

$$F_2(J_3,J_2;\theta _3,\theta _2)=J_2\theta _2\frac{J_3}{2}[m(\theta _2\mathrm{\Omega }_bt)2\theta _3]$$
(12)
so that our resonant angle and new momenta

$`\varphi `$ $`=`$ $`\theta _3{\displaystyle \frac{m}{2}}(\theta _2\mathrm{\Omega }_bt)`$ (13)
$`I_2`$ $`=`$ $`J_2J_3{\displaystyle \frac{m}{2}}`$
$`I_3`$ $`=`$ $`J_3`$
and $`\theta _2`$ is unchanged. Our generating function is time dependent; $`F_2/t=\frac{m}{2}\mathrm{\Omega }_bJ_3`$. Following this canonical transformation our Hamiltonian becomes

$`H(`$ $`J_2,J_3;\theta _2,\varphi )=[\nu {\displaystyle \frac{m}{2}}(\mathrm{\Omega }\mathrm{\Omega }_b)]J_3+{\displaystyle \frac{\lambda }{32\nu ^2}}J_3^2`$ (14)
$`+J_3{\displaystyle \frac{A_m}{4h_1^2\nu }}\mathrm{cos}(2\varphi )+\mathrm{\Omega }J_2+\mathrm{}`$
The motion in the two different degrees of freedom are now decoupled so we can neglect $`J_2`$ and write the Hamiltonian in the simpler form

$$H(J_3;\varphi )=\nu ^{}J_3+aJ_3^2ϵJ_3\mathrm{cos}(2\varphi )$$
(15)
where

$`\nu ^{}`$ $``$ $`\nu {\displaystyle \frac{m}{2}}(\mathrm{\Omega }\mathrm{\Omega }_b)`$ (16)
$`a`$ $``$ $`{\displaystyle \frac{\lambda }{32\nu ^2}}`$
$`ϵ`$ $``$ $`{\displaystyle \frac{A_m}{4h_1^2\nu }}`$
Equation(15) has resonant term proportional to $`J_3`$ instead of proportional $`J_3^{1/2}`$ as was the case for the inner Lindblad resonance, considered by Contopoulos (1975), and for e-e’ orbit-orbit resonances commonly studied in Celestial mechanics (Murray & Dermott, 1999). However the Hamiltonian is similar to that of J+2:J orbital resonances studied by Borderies & Goldreich (1984). Orbits captured in the 2:1 vertical ILR were seen in the numerical simulations of Combes et al. (1990) however they expected the resonance to be first order in $`J_3^{1/2}`$ rather than second order as we have found here. By symmetry the gravitational potential when expanded should not contain terms proportional to $`z`$, and so terms $`J_3^{1/2}`$, unless we consider a global instability such as the fire-hose bar buckling instability. Indeed Pfenniger & Friedli (1991) found that a temporal $`z`$-symmetry breaking (due to a collective effect) would make the resonance first order. Friedli & Pfenniger (1990) reported that N-body simulations which forced $`z`$-symmetry did grow boxy/peanut shaped bulges but more slowly than those which did not force the symmetry.

Unless the galaxy is extremely thin and the bulge small we expect that the 2:1 vertical ILR will occur in the inner region of the bar (Pfenniger & Friedli 1991; Combes et al. 1990). It is also possible that the 1:1 vertical ILR will be present. The 2:1 vertical ILR produces banana-shaped orbits, referred to as BAN in the characteristic diagrams of Pfenniger & Friedli (1991) and the 1:1 vertical ILR produces titled orbits that are referred to as anomalous (ANO) by Pfenniger & Friedli (1991). If the galaxy is very thin it is also possible that there will be orbits associated with the 4:1 vertical ILR (Pfenniger, 1985).

If we concentrate on the 2:1 vertical ILR then the perturbation term from the bar that is relevant is the $`m=4`$ Fourier term; if we focus on the 1:1 vertical ILR then the $`m=2`$ term is relevant. It has long been known that bars exhibit significant $`m=4`$ Fourier components and contain boxy or square contours when projected onto the plane of the galaxy (Athanassoula, 1990). From Figure 4 by Quillen et al. (1994) we estimate that the $`m=4`$ Fourier component of the barred galaxy NGC 4314 the potential peaks at $`A_40.04`$ in units of $`v_c^2`$ and at $`A_20.16`$ in the same units. We can use these as typical values for the potential perturbations in a barred galaxy.

> Fig. 1– Contour levels of the Hamiltonian in Equation(17) have been plotted where $`x=\sqrt{2j_3}\mathrm{cos}\varphi `$ and $`y=\sqrt{2j_3}\mathrm{sin}\varphi `$. Only one resonance region exists for $`\frac{\overline{\nu }}{\overline{ϵ}}>1`$, two exist for $`1<\frac{\overline{\nu }}{\overline{ϵ}}<1`$ and three for $`\frac{\overline{\nu }}{\overline{ϵ}}<1`$. The resonance bifurcates at $`\frac{\overline{\nu }}{\overline{ϵ}}=\pm 1`$ giving banana-shaped orbits at $`\varphi =0`$ and upside down banana-shaped orbits at $`\varphi =\pi `$. The bar strength sets $`\overline{ϵ}`$ and $`\overline{\nu }`$ varies with radius from the center of the resonance where $`\overline{\nu }=0`$. Between $`\frac{\overline{\nu }}{\overline{ϵ}}=11`$ no small $`j_3`$ circulating orbits exist. We can think of the progression of resonance sizes between $`\frac{\overline{\nu }}{\overline{ϵ}}11`$ as describing the radial variation of the width of the peanut. We expect maximum vertical amplitudes at $`\frac{\overline{\nu }}{ϵ}1`$ which is on the outer, larger radius side of the resonance. As the bar grows phase space moves downwards from plot to plot and particles are trapped and then lifted upwards. Likewise if the bar slows down the same thing will happen.

We now transform Equation (15) so that our coefficients are unitless. We put length in units of $`r_{res}`$, the center of the resonance where $`\nu ^{}=0`$ and time in units of $`1/\nu `$. Our action variables are in units of $`r_{res}^2\nu `$ so we define $`j_3J_3/r_{res}^2\nu `$. We divide the entire Hamiltonian by $`r_{res}^2\nu ^2`$ so that our new Hamiltonian becomes

$$h(j_3,\varphi )/(ar_{res}^2)=j_3^2+\overline{\nu }j_3\overline{ϵ}j_3\mathrm{cos}(2\varphi )$$
(17)
where

$`\overline{\nu }`$ $``$ $`{\displaystyle \frac{32\nu ^{}\nu }{r_{res}^2\lambda }}{\displaystyle \frac{32\nu ^{}}{\nu }}{\displaystyle \frac{h_1^2}{r_{res}^2}}`$ (18)
$`\overline{ϵ}`$ $``$ $`{\displaystyle \frac{8A_m}{r_{res}^2h_1^2\lambda }}{\displaystyle \frac{8A_m}{v_c^2}}{\displaystyle \frac{v_c^2}{\nu ^2r_{res}^2}}`$
and we have assumed that the vertical form of the potential can be approximated by $`\lambda \frac{\nu ^2}{h_1^2}`$. $`\lambda `$ should be negative because the vertical oscillations are expected to be slower for particles traveling well above the galactic plane.

For a bar with a peak $`\frac{A_4}{v_c^2}0.04`$, as estimated from NGC 4314. $`\nu r_{res}2\mathrm{\Omega }r_{res}2v_c`$ at the the 2:1 resonance, and $`\nu r_{res}v_c`$ at the 1:1 resonance. We estimate that $`\overline{ϵ}`$ peaks at $`0.08`$ for both resonances because $`A_24A_4`$. Barred galaxies have an observed ratio of bar length to peanut length of $`2.7\pm 0.3`$ (Lütticke et al., 2000). Measurements of the ratio of bar length to thickness give a typical value of $`10`$ (Sellwood & Wilkinson, 1993) with the measurement of Lütticke et al. (2000) being on the high end at $`14\pm 4`$, and the Milky Way bar likely to be lower than the typical value. Using these typical values we estimate that $`h_1/r_{res}0.3`$.

$`\overline{\nu }`$ $``$ $`3{\displaystyle \frac{\nu ^{}}{\nu }}\left({\displaystyle \frac{h_1/r_{res}}{0.3}}\right)^2`$ (19)
$`\overline{ϵ}`$ $``$ $`0.08\left({\displaystyle \frac{A_m/v_c^2}{0.04}}\right)\left({\displaystyle \frac{2}{\nu r_{res}/v_c}}\right)^2.`$
We expect the resonance to be important in the region where $`\overline{\nu }=0\overline{ϵ}`$ or across a range $`dr`$;

$$\frac{dr\nu _{,r}^{}}{\nu }=\frac{A_m}{4\nu ^2h_1^2}$$
(20)
where <sub>,r</sub> denotes a derivative with respect to radius. If $`\overline{\nu }`$ and $`\nu ^{}`$ varies slowly with radius then the resonance can be felt over a significant range of radius.

> Fig. 2– A cartoon of the edge-on galaxy labeling the approximate location of the peanut in terms of the values of $`\frac{\overline{\nu }}{\overline{ϵ}}`$.

> Fig. 3– Particle distribution following growth of a perturbation with $`\overline{ϵ}=0.1`$ for different values of $`\overline{\nu }`$. The initial dispersion in $`j_3`$ was $`\sigma =0.1`$. The perturbation was grown in $`P=3`$ periods and then the integration continued for 5 times longer than the growth period so that orbits become more evenly distributed in phase. $`x`$ and $`y`$ axes are defined as in Fig. 1.

Given the definition of our variables we estimate that the expectation value

$$z^2j_3r_{res}^2.$$
(21)
From Figure 1 we infer that when $`\frac{\overline{\nu }}{\overline{ϵ}}=1`$ the minimum value of vertical oscillations occurs for the fixed points at $`j_3=1`$. We therefore expect the maximum height of the the peanut to be at $`\frac{\overline{\nu }}{\overline{ϵ}}=1`$ where $`<j_3>\overline{ϵ}`$. For $`\overline{ϵ}0.08`$ this implies a peanut width of order $`2\sqrt{\overline{ϵ}}r_{res}0.6r_{res}`$ where we gain the factor of 2 when we consider orbits of both signs ($`\varphi =0`$ and $`\varphi =\pi `$). This is sufficiently large to explain the observed vertical extent of boxy/peanut shaped bulges.

### 2.2 Structure of the resonance

As shown by Borderies & Goldreich (1984) the resonance contains 3 separate regions. To show the separate regions we use a coordinate system $`x=\sqrt{2j_3}\mathrm{cos}\varphi `$ and $`y=\sqrt{2j_3}\mathrm{sin}\varphi `$. Contours of constant $`h`$ (Eqn. 17) are shown in Figure 1. For $`\frac{\overline{\nu }}{\overline{ϵ}}<1`$ three fixed points exist, for $`1<\frac{\overline{\nu }}{\overline{ϵ}}<1`$ two fixed points exist and for $`\frac{\overline{\nu }}{\overline{ϵ}}>1`$ only one fixed point exists. Orbits circulating around $`\varphi =0`$ correspond to banana-shaped orbits with positive $`z`$ at the ends of the bar, and orbits correspond to banana-shaped orbits with negative $`z`$ at the ends of the bar. As shown in Figure 8b of Pfenniger & Friedli (1991) banana shaped orbits appear on the characteristic diagram at a particular value of the Hamiltonian, consistent with them appearing at a particular value of $`\frac{\overline{\nu }}{\overline{ϵ}}`$. For orbits circulating around the origin, the resonant angle, $`\varphi `$, circulates; referred to as oscillation as opposed to libration in analogy with a pendulum.

We consider the signs of each term in Eqns.(15,17). $`\nu >0`$ but we expect that $`\lambda ,A_4,A_2<0`$. Consequently $`a<0`$ and $`ϵ,\overline{ϵ}>0`$ and $`\overline{\nu }`$ has the opposite sign of $`\nu ^{}`$. Because $`\nu `$ and $`\mathrm{\Omega }`$ increase at small radii we expect $`\nu ^{}>0`$ and $`\overline{\nu }<0`$ for $`r>r_{res}`$. We see from Fig.1 that near $`\overline{\nu }=\overline{ϵ}`$ no orbits exist with low values of $`j_3`$. This implies that all orbits will rise out of the plane of the galaxy. As $`\overline{\nu }`$ decreases the mininum size of the vertical oscillations increases until $`\frac{\overline{\nu }}{\overline{ϵ}}<1`$, then orbits with small amplitude vertical oscillations appear again. The vertical extent of the galaxy should increase until a particular radius and then drop abruptly (see Figure 2). This would naturally correspond to a bowtie or peanut shape when viewed edge-on.

### 2.3 Growing the bar

To explore the simple model given in Eqn.(17) we integrate particle trajectories assuming that the perturbation is growing. In other words we assume that $`\overline{ϵ}`$ begins at zero and reaches a maximum value a few rotation periods later. We begin the integrations with a particle distribution $`p(j_3)\mathrm{exp}(j_3^2/(2\sigma ^2))`$ and with angle $`\varphi `$ randomly chosen between 0 and $`2\pi `$. 4000 particles were integrated using Eqn.(17) for the simulations shown in Figures 3–6. The growth time for the perturbation is given in periods where $`P=\frac{2\pi }{\nu }|a|r_{res}^2`$. Since $`|a|r_{res}^2\frac{r_{res}^2}{h_1^232}1/3`$, $`P`$ is about 1/3 the bar growth time in units of the vertical oscillation period.

In the adiabatic limit the capture probability of a region of phase space depends upon the ratio between the rate that phase space volume is swept across the separatrix bounding this region divided by the rate that phase space volume is swept across the entire separatrix bounding the entire resonance (Henrard, 1982; Borderies & Goldreich, 1984). For $`\overline{\nu }<0`$ phase space initially looks like the panel on the bottom right of Fig. 1 and then moves upward on this figure. When the separatrix touches the particle distribution centered at the origin, particles are captured into librating regions near $`\varphi =0`$ and $`\pi `$ (see Figures 3–4). For $`\overline{\nu }>0`$ phase space initially looks like the panel on the top left of Figure 1 and then moves downward on this figure. When $`\overline{\nu }\overline{ϵ}`$ particles at low $`j_3`$ are captured into the librating regions around $`\varphi =0,\pi `$. Figures 3–4 illustrate that smoother particle distributions are achieved when the growth rate is faster (shorter number of periods $`P`$) as we would expect when the variation is no longer adiabatic, and when the initial momentum distribution is wider (larger $`\sigma `$).

Given a distribution in $`j_3`$ and $`\varphi `$ resulting from the growth of the perturbation we can predict the vertical distribution as a function of azimuthal angle in the plane of the galaxy. For a value of the resonant angle $`\varphi `$ and an azimuthal angle $`\theta \mathrm{\Omega }_bt`$ in the plane of the galaxy in the frame rotating with the bar, $`\theta _3`$ is determined via Eqn.(13) and $`z`$ is then determined from the definition of our action angle variables (Eqn. 6). In Figure 5 we show vertical distributions for the action angle distributions shown in Figures 3 and 4b for $`m=2`$ and in Figure 6 we show the equivalent but for $`m=4`$.

We see from Figures 5 and 6 that the vertical height reaches a maximum near $`\frac{\overline{\nu }}{\overline{ϵ}}1`$ as we expected from the form of the resonance. The height increases slowly between $`\frac{\overline{\nu }}{\overline{ϵ}}1`$ to $`1`$ and then drops swiftly, confirming conceptually our bowtie picture shown in Fig. 2. For $`m=2`$ (shown in Figure 5) corresponding to the 1:1 vertical ILR, the major axis of the bar becomes bulbous, as would be expected from a sum of anomalous orbits of both orientations (shown in Figure 7). For $`m=4`$ (shown in Figure 5) corresponding to the 2:1 vertical ILR, both the major and minor axes of the bar have increased vertical heights. Again this is expected from a sum of banana-shaped orbits of both orientations (see Figure 7).

Observations of peanut shaped galaxies where both the bar and the peanut are observed (e.g., Quillen et al. 1997) suggest that the ends of the peanut are aligned with the bar. So it could be that the 1:1 resonance orbit family is more likely to explain the bulbous ends of boxy/peanut shaped bulges. However when the radial oscillations of the galaxy are taken into account the banana-shaped orbits may still be viable, particularly if the extent of their vertical oscillation is smaller along the minor axis than it is along the major axis.

> Fig. 7– Anomalous orbits at the 1:1 resonance are shown on the top and give increased widths at the ends of the bar. Banana shaped orbits at the 2:1 resonance are shown on the bottom and give increased widths at the ends and sides of the bar. The $`x`$ axis is assumed to lie along the major axis of the bar.

We remind the reader that in Section 2.1 we dropped the last term in Eqn.(7) which is proportional to $`I_1I_3`$. As a bar grows the epicyclic amplitude $`I_1`$ of the stars will grow. This has the effect of shifting the location of the vertical resonance so that $`\nu ^{}`$ becomes

$$\nu ^{}=\nu \frac{m}{2}(\mathrm{\Omega }\mathrm{\Omega }_b)+(\kappa ^23\mathrm{\Omega }^2)\frac{\nu I_1}{4\kappa }.$$
(22)
For a flat rotation curve $`\kappa ^2=2\mathrm{\Omega }^2`$ so we expect that $`\nu ^{}`$ will be more negative than expected if this additional term is neglected. This has the affect of moving the resonance outward in radius. One reason that the 2:1 ILR resonance was considered a logical choice for exciting the peanut is that when the galaxy is thin the 1:1 may only occur at extremely small radii, well inside the galaxy bulge. However the estimates for the resonance locations were made assuming $`I_10`$ (e.g., Pfenniger & Norman 1990). The point here is that the growth of the epicyclic amplitude will shift the location of the vertical resonances outwards so that the lower order resonances may manifest at larger radii than previously considered. It is possible that the growth of the bar causes the vertical resonances to sweep through the inner part of the galaxy.

## 3 Summary and Discussion

We have presented a simple Hamiltonian dynamical model for the vertical inner Lindblad resonances in the presence of a bar perturbation. The Hamiltonian model is similar to that introduced by Contopoulos (1975) to describe the 2:1 Lindblad resonances in the plane of the galaxy but uses the vertical action angle variables. Using this model we consider the growth of a boxy/peanut shaped bulge via resonance trapping in the vertical inner Lindblad resonances caused by the growth of the bar. Following the growth of the bar, our model predicts that the height of the peanut increases as a function of radius until it reaches a maximum at the outermost radius which is set by the condition $`\frac{\overline{\nu }}{\overline{ϵ}}=1`$. Our model naturally accounts for the sharp outer edge of observed boxy/peanut shaped bulges. The maximum total vertical width of the peanut is of order $`2\sqrt{\overline{ϵ}}r_{res}`$ which we estimate for a typical bar could be about two thirds $`r_{res}`$, where $`r_{res}`$ is the radial location of the resonance. This model is therefore capable of accounting for the observed widths of boxy/peanut shaped bulges.

The scenario explored here suggests that the boxy/peanut shaped bulge is grown at the same time as the bar forms. We predict that recently formed bars should manifest strong boxy/peanut shaped bulges and barred galaxies should never be found without boxy/peanut shaped bulges. If the boxy/peanut shapes take nearly a Hubble time to grow, as found in the N-body simulations of Combes et al. (1990), then there should be barred galaxies which lack boxy/peanut shaped bulges. However Bureau & Freeman (1999) considered a control sample of edge-on galaxies lacking boxy/peanut bulges. No examples of barred galaxies that lack a boxy/peanut shaped bulge have yet been identified. Pfenniger & Friedli (1991) found that the boxy/peanut only required 3-4 bar growth periods to grow and this could be significantly shorter, only a few Gyrs. If this timescale were more appropriate then the lack of barred galaxies which do not display boxy/peanut shaped bulges is less of a problem.

One way to probe the formation timescale of the boxy/peanut is to search for barred galaxies which have recently formed bars. If boxy/peanuts require many rotation periods to form then these barred galaxies should have normal (non-boxy) bulges. Because of the high rate of star formation along the bar Quillen et al. (1997) inferred that NGC 7582 was a case of a recently formed bar. They pointed out that the galaxy bulge did show a strong prominent peanut and so argued that boxy/peanut shaped bulges are formed quickly after the onset of a bar. Observations of edge-on galaxies are therefore consistent with a rapid boxy/peanut formation mechanism such as we have proposed here.

Our scenario does not require the galaxy to buckle in order to form the boxy/peanut shaped bulge. The fire-hose instability scenario for bar-buckling (beautifully illustrated in 3D N-body simulations by Raha et al. 1991) predicts that the galaxy will literally be bent in a U-shape while the galaxy buckles, however no examples of U-shaped edge-on galaxies have been identified in galaxy surveys (with the exception of galaxies which are believed to be ram pressure stripped in clusters). It is likely that the period of time during which the galaxy is U-shaped is short. However galaxy collisions have a similar estimated lifetime and are detected in nearby galaxy surveys. We note that Friedli & Pfenniger (1990) reported that N-body simulations with fixed $`z`$-symmetry did grow boxy/peanut shaped bulges, though at a slower rate than unconstrained simulations.

We have constructed an analytical framework which can be used to predict the height distribution of the peanut as a function of radius and the azimuthal angle in the galaxy plane. The nature of the dependence is primarily determined by the resonance responsible for the vertical excitations. A 3 dimensional study of a nearby boxy bulge (such as that seen in our galaxy) could determine which resonance is most likely to be responsible. The shape of the boxy/peanut shaped bulge is also sensitive to the dependence of the gravitational potential on height from the mid-plane and on the way the bar grew. Further study of boxy/peanut shaped bulges may allow us to place constraints on the vertical shape of the galaxy and on the early evolution of the bar.

The toy model introduced in this paper was designed to illustrate the process of resonance capture during the growth of a bar. We have argued that the growth of the bar itself causes stars to become trapped into vertical resonances and so lifted out of the plane of the galaxy. However, simultaneous growth of a bar and boxy/peanut shaped bulge has not yet been observed in three dimensional N-body simulations of non-bending bars. This could in part be because N-body simulations are are typically thicker than thin bars such as NGC 7582. Moreover, N-body simulations (Pfenniger & Friedli, 1991; Friedli & Pfenniger, 1990; Combes et al., 1990) have shown that the boxy/peanut continues to grow well after the bar is stable. Since resonance capture is likely to be a natural consequence of bar growth, it might be useful to reexamine N-body simulations and verify if orbits are trapped into vertical resonances during the bar growth period. After the bar stabilizes, the analytical framework developed here might also be applied toward predicting the evolution of stellar orbits caused by secular evolution of a bar. For example a slow decrease in pattern speed will also trap particles in the vertical resonances and cause them to be lifted into higher orbits. In this case $`\overline{\nu }`$ decreases with time so that phase space moves downwards in Figure 1, causing similar phenomenolgy as growing the bar; particles trapped into resonances can be lifted higher as the bar evolves.

This work would not have been carried out without helpful discussions with Andrew Anissi and Larry Helfer. A. C. Q. gratefully thanks the Technion for hospitality and support during the fall of 2001. To put the Hamiltonian into action angle variables to first order we use the generating function
$$F_1(z,\theta ^{})=\frac{\nu }{2}z^2\mathrm{tan}\theta ^{}$$
(23)
Following this canonical transformation the $`z`$ part of the Hamiltonian restricted to $`r=r_c`$

$$H_z(p_z;z)=\frac{1}{2}(p_z^2+\nu ^2z^2)+\frac{\lambda z^4}{4!}$$
(24)
becomes

$$H_z(I^{};\theta ^{})=\nu I_3^{}+\frac{\lambda I^2}{6\nu ^2}\mathrm{cos}^4(\theta ^{})$$
(25)
To put the Hamiltonian into a form that doesn’t depend on the angle we must do another canonical transformation. Note that $`\mathrm{cos}^4\theta =\frac{1}{8}\mathrm{cos}4\theta +\frac{1}{2}\mathrm{cos}2\theta +\frac{3}{8}`$.

We try the following generating function

$$F_2(\theta ^{},I_3)=I_3\frac{\lambda }{6\nu ^2}I_3^2(\frac{1}{4}\mathrm{sin}2\theta ^{}+\frac{1}{32}\mathrm{sin}4\theta ^{})$$
(26)
which leads to new variables $`I_3,\theta _3`$ such that

$`I^{}`$ $`=`$ $`I_3+{\displaystyle \frac{\lambda }{6\nu }}I_3^2\left({\displaystyle \frac{1}{2}}\mathrm{cos}2\theta ^{}+{\displaystyle \frac{1}{8}}\mathrm{sin}4\theta ^{}\right)`$ (27)
$`\theta _3`$ $`=`$ $`\theta ^{}{\displaystyle \frac{\lambda }{6\nu ^2}}I_3^{}\left({\displaystyle \frac{1}{4}}\mathrm{sin}2\theta ^{}+{\displaystyle \frac{1}{32}}\mathrm{sin}4\theta ^{}\right)`$
To fourth order in $`I_3^{1/2}`$ the Hamiltonian becomes

$$H_z(I_3;\theta _3)=\nu I_3+\frac{\lambda I_3^2}{32\nu ^2}$$
(28)
which is independent of $`\theta _3`$ so that $`I_3`$ is a conserved quantity and we find that we have successfully transfered the Hamiltonian into action angle variables.

Via a similar canonical transformation the term proportional to $`(rr_c)z^2`$ in Equation (2) disappears but the term proportional to $`(rr_c)^2z^2`$ remains yielding the additional term $`(\kappa ^23\mathrm{\Omega }^2)\frac{\nu I_1I_3}{4\kappa }`$ which we have included in Eqn. (7).


text:
# 1 Introduction

## 1 Introduction

The Randall-Sundrum (RS) model  offers a new approach to the hierarchy problem. This scheme proposes that our four-dimensional world is embedded in a five-dimensional spacetime described by the metric

$$ds^2=e^{2\sigma (y)}\eta _{\mu \nu }dx^\mu dx^\nu dy^2,$$
(1)
where $`\sigma (y)=k|y|`$, and with the $`5^{th}`$ dimensional coordinate $`y=r_c\varphi `$ being compactified on an $`S^1/Z_2`$ orbifold bounded by branes of opposite tension at the fixed points $`y=0`$ (known as the Planck brane) and $`y=\pi r_c`$ (TeV-brane). The parameter $`k`$ describes the curvature of the space (with the five-dimensional curvature invariant being given by $`=20k^2`$) and is of order the five-dimensional Planck scale, $`M_5`$, so that no additional hierarchy exists. Self-consistency of the classical theory requires  that $`||M_5^2`$ so that quantum gravitational effects can be neglected. The space between the two branes is $`AdS_5`$ and their separation, $`\pi r_c`$, can be naturally stabilized  with $`kr_c1112`$; we employ $`kr_c=11.27`$ in the numerical results that follow. For such values of $`kr_c`$ any mass of order the Planck scale on the $`y=0`$ brane appears to be suppressed by an amount $`e^{\pi kr_c}10^{15}`$ on the TeV brane. The presence of the exponential warp factor $`e^{\sigma (y)}`$ thus naturally generates the hierarchy between the Planck and electroweak (EW) scales. The scale of physics on the TeV-brane is given by $`\mathrm{\Lambda }_\pi =\overline{M}_{Pl}e^{kr_c\pi }TeV`$, where $`\overline{M}_{Pl}`$ is the reduced Planck scale. Integration over the extra dimension of the five dimensional RS action yields the relationship between the 4-dimensional Planck scale and the scales $`k`$ and $`M_5`$:

$$\overline{M}_{Pl}^2=M_5^3/k.$$
(2)
Together, this relation and the inequality $`||M_5^2`$ imply that the ratio $`k/\overline{M}_{Pl}`$ cannot be too large, and suggests that $`k/\overline{M}_{Pl}\begin{array}{c}<\hfill \\ \hfill \end{array}0.11`$.

In the original RS framework, gravity propagates freely throughout the bulk while the Standard Model (SM) fields are constrained to the TeV-brane. The graviton KK states have non-trivial wave functions in the extra dimension due to the warp factor, and have masses given by $`m_n=x_nke^{\pi kr_c}`$, where the $`x_n`$ are the unequally spaced roots of the Bessel function $`J_1`$ and $`n`$ labels the KK excitation level. The first graviton excitation thus naturally has a mass of order a TeV. The $`n>0`$ KK states couple to fields on the TeV-brane with a strength of $`\mathrm{\Lambda }_\pi ^1`$. The graviton KK states can thus be produced in colliders as TeV-scale resonances with TeV-1-size couplings to matter .

For additional freedom in model building, the original RS model has been extended to allow various subsets of the SM fields to reside in the bulk in the limit that the back-reaction on the metric can be neglected. This possibility allows for new techniques to address gauge coupling unification, supersymmetry breaking, the neutrino mass spectrum, and the fermion mass hierarchy. Placing the gauge fields of the SM alone in the bulk is problematic , as all of the gauge KK excitations then have large couplings to the remaining fields on the TeV-brane; these couplings take on the value $`\sqrt{2\pi kr_c}g8.4g`$, where $`g`$ is the corresponding SM gauge coupling. EW precision data then constrain the masses of the first KK gauge states to be in excess of 25-30 TeV, thus requiring $`\mathrm{\Lambda }_\pi `$ to be in excess of 100 TeV . This introduces a new hierarchy between $`\mathrm{\Lambda }_\pi `$ and the EW scale, and therefore this scenario is highly disfavored. It was subsequently shown that these constraints can be softened by also placing the SM fermions in the bulk  and giving them a common five-dimensional mass $`m=k\nu `$. This leads to further model building possibilities provided $`\nu `$ is in the range $`0.8\begin{array}{c}<\hfill \\ \hfill \end{array}\nu \begin{array}{c}<\hfill \\ \hfill \end{array}0.3`$ ; for larger values of $`\nu `$ the former strong coupling regime is again entered, while for smaller values potential problems with perturbation theory can arise . In the absence of fine-tuning, these scenarios require that the Higgs field which breaks the symmetry of the SM remains on the TeV brane, as when bulk Higgs fields are employed, the experimentally observed pattern of $`W`$ and $`Z`$ masses cannot be reproduced. In this case, the $`W`$ and $`Z`$ obtain a common KK mass in addition to the usual contribution from the Higgs vev. It is then impossible to simultaneously maintain the two tree-level SM relationships $`M_Z\mathrm{cos}\theta _w=M_W`$ and $`e=g\mathrm{sin}\theta _w`$ .

In this paper we re-examine the possibility of allowing the SM fermions to propagate in the RS bulk. We show that if the third generation of fermions resides in the bulk, then large mixing between the fermion zero modes and their KK tower states is induced by the SM Higgs vev and the large top Yukawa coupling. This mixing results in contributions to $`\delta \rho `$ or $`T`$  which greatly exceeds the bound set by current precision EW measurements . The only way to circumvent this problem is to raise the mass of the first KK gauge state above 25 TeV for any value of $`\nu `$ in its viable range, which again implies a higher value of $`\mathrm{\Lambda }_\pi `$. Unless we are willing to fine tune $`\mathrm{\Lambda }_\pi `$, we must then require the third generation fields to remain on the TeV-brane so that they have no KK excitations. If we treat the three generations symmetrically we must localize all of the fermions on the TeV-brane, and also confine the SM gauge sector to the TeV brane as discussed in the previous paragraph. We instead propose here a ‘mixed’ scenario which places the first two generations of fermions in the bulk and localizes the third on the wall. We find that mixing of the KK towers of these lighter generations with their zero modes does not yield a dangerously large value of $`\delta \rho `$ provided that $`\nu \begin{array}{c}>\hfill \\ \hfill \end{array}0.6`$. Furthermore, we show that values of $`\nu `$ near $`0.4`$ to $`0.5`$ may help explain the mass hierarchies $`m_c/m_t`$ and $`m_s/m_b`$. We explore the possible signatures of this scenario at the LHC and future linear colliders,as well as in precision measurements and flavor changing neutral currents (FCNC); we find that the same parameter space which addresses the fermion mass hierarchies also allows a Higgs boson with a mass of 500 GeV, and is otherwise invisible at the LHC.

The outline of this paper is as follows. In Section II we give a brief overview of the mechanics of the RS model which we will need for subsequent calculations. In Section III we examine the contributions to the $`\rho `$ parameter when the third generation is in the bulk and show that this scenario is highly disfavored. We examine the present bounds on the KK mass spectrum in our mixed scenario that arise from precision EW data in Section IV. We demonstrate that SM Higgs masses as large as 500 GeV are now allowed by the electroweak fit since these contributions can be partially ameliorated from those of the KK states. Section V explores the implications of this scenario for the LHC, while Section VI examines the signatures at a future $`e^+e^{}`$ linear collider and at GigaZ. In particular, we show that the KK states in this model lie outside the kinematically limited range of the LHC but yield observable indirect effects at a linear collider. Finally, we discuss constraints from FCNC in Section VII, and present our conclusions in Section VIII.

## 2 The Standard Model Off the Wall

We present here a cursory formulation of the RS model in the case where the SM gauge and fermion fields propagate in the bulk; we refer the reader to  for a thorough introduction.

We begin by considering a $`SU`$(N) gauge theory defined by the action

$$S_A=\frac{1}{4}d^5x\sqrt{G}G^{MK}G^{NL}F_{KL}F_{MN},$$
(3)
where $`G^{\alpha \beta }=e^{2\sigma }\left(\eta ^{\alpha \beta }+\kappa _5h^{\alpha \beta }\right)`$, $`\sqrt{G}|det(G_{MN})|^{1/2}=e^{4\sigma }`$, with $`\kappa _5=2M_5^{3/2}`$, $`\eta _{\alpha \beta }`$ being the Minkowski metric with signature -2, and $`h_{\alpha \beta }`$ represents the graviton fluctuations. $`F_{MN}`$ is the 5-dimensional field strength tensor given by

$$F_{MN}=_MA_N_NA_M+ig_5[A_M,A_N],$$
(4)
and $`A_M`$ is the matrix valued 5-dimensional gauge field and $`g_5`$ is the corresponding 5-dimensional gauge coupling. We impose the gauge condition $`A_5=0`$; this is consistent with 5-dimensional gauge invariance , and with the $`Z_2`$-odd parity assigned to $`A_5`$ to remove its zero mode from the TeV-brane action. To derive the effective 4-dimensional theory we expand $`A_\mu `$ as

$$A_\mu (x,\varphi )=\underset{n=0}{\overset{\mathrm{}}{}}A_\mu ^{(n)}\frac{\chi ^{(n)}(\varphi )}{\sqrt{r_c}},$$
(5)
and require that the bulk wavefunctions $`\chi ^{(n)}`$ satisfy the orthonormality constraint

$$_\pi ^\pi 𝑑\varphi \chi ^{(m)}\chi ^{(n)}=\delta ^{mn}.$$
(6)
We obtain a tower of massive KK gauge fields $`A_\mu ^{(n)}`$, with $`n1`$, and a massless zero mode $`A_\mu ^{(0)}`$. The KK masses $`m_n^A`$ are determined by the eigenvalue equation

$$\frac{1}{r_c^2}\frac{d}{d\varphi }\left(e^{2\sigma }\frac{d}{d\varphi }\chi ^{(n)}\right)=\left(m_n^A\right)^2\chi ^{(n)}.$$
(7)
This yields $`m_n^A=x_n^Ake^{kr_c\pi }`$ on the TeV-brane, where the $`x_n^A`$ are given in , with the first few numerical values being given by $`x_1^A2.45`$, $`x_2^A5.57`$, $`x_3^A8.70`$. Explicit expressions for the bulk wavefunctions $`\chi ^{(n)}`$ also contain the first order Bessel functions $`J_1`$ and $`Y_1`$, and can be found in ; we note here only that the zero mode wavefunction is $`\varphi `$ independent with $`\chi ^{(0)}=1/\sqrt{2\pi }`$.

We now add a fermion field charged under this gauge group, and able to propagate in the bulk. The action for this field is

$$S_F=d^4xdy\sqrt{G}[V_n^M(\frac{i}{2}\overline{\mathrm{\Psi }}\gamma ^n\underset{M}{\overset{}{D}}\mathrm{\Psi }+\mathrm{h}.\mathrm{c}.)\mathrm{sgn}(y)m\overline{\mathrm{\Psi }}\mathrm{\Psi }],$$
(8)
where h.c. denotes the hermitian conjugate, $`V_\mu ^M=e^\sigma \delta _\mu ^M`$, $`V_5^5=1`$, $`\gamma ^n=(\gamma ^\mu ,i\gamma _5)`$, $`D_M`$ is the covariant derivative, and $`m`$ is the 5-dimensional Dirac mass parameter. This 5-dimensional fermion is necessarily vector-like; we wish to obtain a chiral zero mode from its KK expansion. We follow  and expand the chiral components of the 5-dimensional field as

$$\mathrm{\Psi }_{L,R}(x,\varphi )=\underset{n=0}{\overset{\mathrm{}}{}}\psi _{L,R}^{(n)}(x)\frac{e^{2\sigma }}{\sqrt{r_c}}f_{L,R}^{(n)}(\varphi ),$$
(9)
and require the orthonormality conditions

$$_\pi ^\pi e^\sigma f_L^{(m)}f_L^{(n)}=_\pi ^\pi e^\sigma f_R^{(m)}f_R^{(n)}=\delta ^{mn}.$$
(10)
The $`Z_2`$ symmetry of the 5-dimensional mass term in the action forces $`f_L^{(n)}`$ and $`f_R^{(n)}`$ to have opposite $`Z_2`$ parity; we choose $`f_L^{(n)}`$ to be $`Z_2`$ even and $`f_R^{(n)}`$ to be $`Z_2`$ odd. As shown in , this removes $`f_R^{(0)}`$ from the TeV-brane action, and we obtain the chiral zero mode $`f_L^{(0)}`$ necessary for construction of the SM. The KK states form a tower of massive vector fermions. The zero mode wavefunction is

$$f_L^{(0)}=\frac{e^{\nu \sigma }}{N_0^L},$$
(11)
where $`\nu =m/k`$ and is expected to be of order unity, and $`N_0^L`$ is determined from the orthonormality constraint of Eq. 10. Explicit expressions for the KK fermion masses and wavefunctions are given in ; we note here that $`m_n^F=m_n^A`$ when $`\nu =0.5`$, and that $`m_n^F>m_n^A`$ for all other values of $`\nu `$.

Inserting the KK expansions of both the gauge and fermion fields into the covariant derivative term in Eq. 8, we find that the ratios of fermion-gauge KK couplings to the corresponding 4-dimensional coupling are

$$C_{f\overline{f}A}^{mnq}=\sqrt{2\pi }_\pi ^\pi 𝑑\varphi e^\sigma f_L^{(m)}f_L^{(n)}\chi ^{(q)},$$
(12)
where $`m,n,q`$ label the excitation state. The coefficients $`C_{f\overline{f}A}^{00n}`$ and $`C_{f\overline{f}A}^{01n}`$ are shown in Fig. 1 for $`n=1,\mathrm{},4`$ as functions of $`\nu `$. Notice that $`C_{f\overline{f}A}^{001}`$ vanishes at $`\nu =0.5`$ and remains small for $`\nu <0.5`$; this fact will be crucial in our later analysis.

In addition, the ratios of the KK triple gauge couplings (TGCs) to the TGC of the 4-dimensional theory are given by

$$C_{AAA}^{mnq}=\frac{g^{(mnq)}}{g}=\sqrt{2\pi }_\pi ^\pi 𝑑\varphi \chi ^{(m)}\chi ^{(n)}\chi ^{(q)},$$
(13)
where we have identified $`g=g_5/\sqrt{2\pi r_c}`$. Using the zero mode wavefunction $`\chi ^{(0)}=1/\sqrt{2\pi }`$ and the orthonormality constraint of Eq. 6, we find that $`C_{AAA}^{n00}=0`$ when $`n>0`$; no coupling exists between two zero mode gauge particles and a KK gauge state.

We will also require the couplings between gauge KK states and the fermion fields which are localized on the TeV-brane . The relevant action is

$$S_F=d^4xd\varphi \sqrt{G}\left[V_n^M(\frac{i}{2}\overline{\psi }\gamma ^n\underset{M}{\overset{}{D}}\psi +\mathrm{h}.\mathrm{c}.)\right]\delta (\varphi \pi ).$$
(14)
Inserting the expansion of Eq. 5 into this expression, letting $`\psi e^{3\sigma /2}\psi `$, and setting $`g=g_5/\sqrt{2\pi r_c}`$, we find that the ratio of the nth KK gauge coupling to localized fermions relative to the corresponding SM coupling is

$$C_{f\overline{f}A}^n=\frac{\chi ^{(n)}(\pi )}{\chi ^{(0)}(\pi )}.$$
(15)
Utilizing the approximate expressions for the KK gauge wavefunctions in , these become

$$C_{f\overline{f}A}^n\left(1\right)^{n+1}\sqrt{2\pi kr_c}.$$
(16)
We now consider the final ingredient required for construction of the SM, the Higgs boson. As discussed in the introduction, the Higgs field must be confined to the TeV-brane to correctly break the electroweak symmetry. Its action can therefore be expressed as

$$S_H=d^4x𝑑y\sqrt{G}\left\{G^{MN}D_MH\left(D_NH\right)^{}V(H)\right\}\delta (yr_c\pi ),$$
(17)
where $`V(H)`$ is the Higgs potential and $`D_M`$ the covariant derivative. To properly normalize the Higgs field kinetic term we must rescale $`He^\sigma H`$; we then expand $`H`$ around its vev, $`v`$, insert the expansion of Eq. 5 into the covariant derivative, and identify $`g=g_5/\sqrt{2\pi r_c}`$. We find the gauge field mass terms

$$S_{H,mass}=\frac{1}{2}\underset{m,n=0}{\overset{\mathrm{}}{}}a_{mn}d^4xm_{A,0}^2A_\mu ^{(m)}A^{(n),\mu },$$
(18)
where $`m_{A,0}`$ is the gauge field mass of the 4-dimensional theory corresponding to the zero-mode of the gauge KK tower, and

$$a_{mn}=2\pi \chi ^{(m)}(\pi )\chi ^{(n)}(\pi ).$$
(19)
We must diagonalize the full mass matrix, including the contributions arising from the KK reduction, to obtain the physical spectrum; we will do so for the SM gauge fields in a later section.

We now examine the mixing between fermion KK states induced by the Higgs field. When fermion fields are confined to the TeV-brane, no such mixing occurs; we therefore consider only the case where the fermions propagate in the bulk. The coupling between the Higgs and KK fermions is

$$S_{f\overline{f}H}=\frac{\lambda ^{^{}}}{k}d^4xdy\sqrt{G}\{H^{}\mathrm{\Psi }_D\mathrm{\Psi }_S^c+\mathrm{h}.\mathrm{c}.\}\delta (yr_c\pi ),$$
(20)
where $`\lambda ^{^{}}`$ is the 5-dimensional Yukawa coupling, and $`k`$ has been introduced to make $`\lambda ^{^{}}`$ dimensionless. Both $`\mathrm{\Psi }_D`$ and $`\mathrm{\Psi }_S^c`$ are left-handed Weyl fermions; we have introduced the subscripts $`D`$ and $`S`$ for these fields to indicate that in the SM, the Higgs couples $`SU(2)_L`$ doublets to $`SU(2)_L`$ singlets. After diagonalization of the mass matrix, the hermitian conjugates of the singlet fields will combine with the appropriate doublets to form Dirac fermions. Since, for our analysis, we are interested only in the contributions to the fermion masses arising from this action, we again rescale the Higgs field by $`e^\sigma `$, set the Higgs field equal to its vev and expand the left-handed fermion wavefunctions as in Eq. 9. Identifying

$$\lambda =\frac{\lambda ^{^{}}\left(f_L^{(0)}(\pi )\right)^2e^{kr_c\pi }}{kr_c}$$
(21)
as the 4-dimensional Yukawa coupling, we find the fermion mass terms

$$S_{f,mass}=\underset{m,n=0}{\overset{\mathrm{}}{}}b_{mn}d^4x\{m_{f,0}\psi _D^{(m)}\psi _S^{c,(n)}+\mathrm{h}.\mathrm{c}.\},$$
(22)
where $`m_{f,0}`$ is the zero mode mass obtained when the KK states decouple, and

$$b_{mn}=\frac{f_L^{(m)}(\pi )f_L^{(n)}(\pi )}{\left(f_L^{(0)}(\pi )\right)^2}.$$
(23)
Since $`f_L^{(n)}(\pi )`$ is approximately the same function for all $`n1`$, we will set $`b_{0n}=\sqrt{f}`$ and $`b_{mn}=f`$, with $`mn0`$, in our analysis, where $`f`$ is a $`\nu `$ dependent quantity that measures the strength of the mixing. This parameter is explicitly given by

$$f=2\frac{1e^{k\pi r_c(1+2\nu )}}{1+2\nu }.$$
(24)
These mass terms must be diagonalized in conjunction with the contributions from the KK reduction of Eq. 8; we will do so for the SM $`b`$ and $`t`$ quarks in the next section.

To complete our discussion of the RS model, we must briefly discuss the KK gravitons it contains. We parameterize the 5-dimensional metric as

$$G_{\alpha \beta }=e^{2\sigma }\left(\eta _{\alpha \beta }+\kappa _5h_{\alpha \beta }\right),$$
(25)
where $`\kappa _5=2M_5^{3/2}`$, $`\eta _{\alpha \beta }`$ is the Minkowski metric with signature -2, and the fluctuations of the bulk radius have been neglected. We then expand the graviton field $`h_{\alpha \beta }`$ as

$$h_{\alpha \beta }(x,\varphi )=\underset{n=0}{\overset{\mathrm{}}{}}h_{\alpha \beta }^{(n)}(x)\frac{\chi _G^n(\varphi )}{\sqrt{r_c}},$$
(26)
and impose the orthonormality constraint

$$_\pi ^\pi 𝑑\varphi e^{2\sigma }\chi _G^{(m)}\chi _G^{(n)}=\delta ^{mn}.$$
(27)
The explicit forms of the KK graviton wavefunctions contain the second order Bessel functions $`J_2`$ and $`Y_2`$, and can be found in . Expressing the graviton masses as $`m_n^G=x_n^Gke^{kr_c\pi }`$, we find the numerical values $`x_1^G3.83`$, $`x_2^G7.02`$, $`x_3^G10.17`$ for the first few states which are given by $`J_1(x_n^G)=0`$; notice that $`m_n^G>m_n^A`$. The couplings of the KK gravitons to fermions are given by

$$C_{f\overline{f}G}^{mnq}=_\pi ^\pi 𝑑\varphi \frac{e^\sigma f^{(m)}f^{(n)}\chi _G^{(q)}}{\sqrt{kr_c}}.$$
(28)
The $`C_{f\overline{f}G}^{00n}`$ are presented in Fig. 2 as functions of $`\nu `$. These couplings are relatively small, particularly when $`\nu 0.5`$; this, and their large mass, render the KK gravitons unimportant in our analysis.

We now have the tools necessary to build the SM within the RS framework. In the next section we will discuss the diagonalization of the $`t`$ and $`b`$ quark mass matrices, and the KK contributions to the $`\rho `$ parameter. To whet the reader’s appetite, we note that the off-diagonal mass matrix elements $`b_{mn}`$, with $`m,n1`$ and $`mn`$, range from $`10`$ when $`\nu =0.4`$ to $`700`$ when $`\nu =0.6`$. This induces large mixing between the zero mode top quark and its KK tower, and creates couplings between the zero mode $`b`$ quark and the top quark KK states. The large mass splitting between these states results in drastic alterations of $`\rho `$, and renders the placement of third generation quarks in the bulk inconsistent with EW measurements for a wide range of KK masses.

## 3 Fermion Mixing and the $`\rho `$ Parameter

Since the top quark is quite heavy, with a mass $`m_t175`$ GeV, we expect the mixing between it and its KK tower to be stronger than that for the lighter fermions, and we focus here upon it and its isodoublet partner, the bottom quark. As mentioned previously, after performing the KK reduction of the 5-dimensional fermion field we obtain a chiral zero mode and a vector-like KK tower; this spectrum is presented pictorially for the top quark in Table 1.

We have exchanged the left-handed $`SU(2)_L`$ Weyl singlets introduced in the preceding section for their right-handed conjugates, but the reader should remember that these states are still described by the 5-dimensional wavefunctions $`f_L^{(n)}(\varphi )`$. The top quark mass matrix receives two distinct contributions: the diagonal KK couplings between the doublet tower states and the singlet tower states, and off-diagonal mixings between the left-handed doublets and right-handed singlets arising from the Higgs coupling on the TeV-brane. We present below the mass matrix with only the zero modes and the first two KK levels included; the infinite-dimensional case can be obtained by a simple generalization. Working in the weak eigenstate basis defined by the vectors $`\mathrm{\Psi }_L^t=(T_L^{(0)},T_L^{(1)},t_L^{(1)},T_L^{(2)},t_L^{(2)})`$ and $`\mathrm{\Psi }_R^t=(t_R^{(0)},t_R^{(1)},T_R^{(1)},t_R^{(2)},T_R^{(2)})`$, we find

$$_t=\left(\begin{array}{ccccc}m_{t,0}& \sqrt{f}m_{t,0}& 0& \sqrt{f}m_{t,0}& 0\\ \sqrt{f}m_{t,0}& fm_{t,0}& m_1& \sqrt{f}m_{t,0}& 0\\ 0& m_1& 0& 0& 0\\ \sqrt{f}m_{t,0}& fm_{t,0}& 0& fm_{t,0}& m_2\\ 0& 0& 0& m_2& 0\end{array}\right).$$
(29)
$`m_{t,0}`$ is the mass of the zero mode in the infinite KK mass limit, and $`m_1`$ and $`m_2`$ are the masses of the first two KK fermion states in the limit of vanishing Higgs couplings; $`f`$ is the mixing strength introduced in the previous section. We fix $`m_{t,0}`$ by demanding that the lowest lying eigenvalue of this matrix reproduce the measured top quark mass, $`m_t=174.3`$ GeV. Due to the large values of the off-diagonal elements, we diagonalize this mass matrix numerically, rather than analytically to $`𝒪(m_{t,0}/m_1)`$. We must necessarily truncate the KK expansion at some level; we have performed our analysis twice, once including only the first KK level and once keeping the first two levels, and have checked that adding more states only strengthens our conclusions. We examine the parameter region $`0.3\begin{array}{c}>\hfill \\ \hfill \end{array}\nu \begin{array}{c}>\hfill \\ \hfill \end{array}0.55`$; the range $`\nu >0.3`$ is strongly constrained by contact interaction searches at LEP , and the values $`\nu \begin{array}{c}<\hfill \\ \hfill \end{array}0.55`$ are prohibited by extrapolation of the results obtained below. This is essentially the same region studied in , where it was shown that the LHC will be able to probe the shift in the $`Zt\overline{t}`$ coupling for fermion KK mass values $`m_1\begin{array}{c}<\hfill \\ \hfill \end{array}15`$ TeV. We will find that the region where $`\nu 0.3`$ and $`m_1\begin{array}{c}<\hfill \\ \hfill \end{array}30100`$ TeV is already disfavoured by current measurements.

The Lagrangian containing the top quark mass terms and its interactions with the $`Z`$ and $`W^\pm `$ gauge bosons is

$``$ $`=`$ $`(\overline{\mathrm{\Psi }}_L^t_t\mathrm{\Psi }_R^t+\mathrm{h}.\mathrm{c}.)+\overline{\mathrm{\Psi }}_L^t\overline{)}ZC_{t,L}^Z\mathrm{\Psi }_L^t+\overline{\mathrm{\Psi }}_R^t\overline{)}ZC_{t,R}^Z\mathrm{\Psi }_R^t`$ (30)
$`+\overline{\mathrm{\Psi }}_L^t\overline{)}W^{}C_L^W\mathrm{\Psi }_L^b+\overline{\mathrm{\Psi }}_R^t\overline{)}W^{}C_R^W\mathrm{\Psi }_R^b+\mathrm{}+\mathrm{h}.\mathrm{c}..`$
We have introduced the basis $`\mathrm{\Psi }_L^b`$ and $`\mathrm{\Psi }_R^b`$ for the bottom quark in analogy with those for the top quark. The $`C_j^i`$ are matrices containing the couplings of the various top quark states to the $`Z`$ and $`W^\pm `$; letting $`g`$ denote the SM electroweak coupling, $`c_W`$ the cosine of the weak mixing angle, and $`g_L`$ and $`g_R`$ the couplings of the usual left-handed and right-handed SM fermions to the $`Z`$ boson, we find

$`C_{t,L}^Z`$ $`=`$ $`{\displaystyle \frac{g}{c_W}}\mathrm{diag}(g_L,g_L,g_R,g_L,g_R),`$
$`C_{t,R}^Z`$ $`=`$ $`{\displaystyle \frac{g}{c_W}}\mathrm{diag}(g_R,g_R,g_L,g_R,g_L),`$
$`C_L^W`$ $`=`$ $`{\displaystyle \frac{g}{\sqrt{2}}}\mathrm{diag}(1,1,0,1,0),`$
$`C_R^W`$ $`=`$ $`{\displaystyle \frac{g}{\sqrt{2}}}\mathrm{diag}(0,0,1,0,1).`$ (31)
In obtaining these matrices we have treated the $`T_R^{(n)}`$ as $`SU(2)_L`$ doublets and the $`t_L^{(n)}`$ as singlets as denoted in Table 1. We diagonalize $`_t`$ with the two unitary matrices $`U_L^t`$ and $`U_R^t`$,

$$_t^D=U_L^t_t\left(U_R^t\right)^{}.$$
(32)
Diagonalization of the matrix $`_t_t^{}`$ determines $`U_L^t`$ up to an overall phase matrix, while diagonalization of $`_t^{}_t`$ similarly fixes $`U_R^t`$. The mass eigenstate basis is obtained by multiplication of the weak eigenstate basis by the appropriate transformation matrix:

$$\mathrm{\Psi }_L^tU_L^t\mathrm{\Psi }_L^t,\mathrm{\Psi }_R^tU_R^t\mathrm{\Psi }_R^t.$$
(33)
The coupling matrices undergo a similar shift,

$`C_{t,L}^Z`$ $``$ $`U_L^tC_{t,L}^Z\left(U_L^t\right)^{},`$
$`C_{t,R}^Z`$ $``$ $`U_R^tC_{t,R}^Z\left(U_R^t\right)^{},`$
$`C_L^W`$ $``$ $`U_L^tC_L^W\left(U_L^b\right)^{},`$
$`C_R^W`$ $``$ $`U_R^tC_R^W\left(U_R^b\right)^{}.`$ (34)
We have also implicitly performed an identical diagonalization of the bottom quark mass matrix.

This procedure induces off-diagonal elements in both the $`Z`$ and $`W^\pm `$ coupling matrices; consequently, fermions of widely varying masses enter the vacuum polarization graphs contributing to the $`Z`$ and $`W^\pm `$ self energies. Such a scenario typically generates unacceptable contributions to the $`\rho `$ parameter , defined as

$$\rho =\frac{\mathrm{\Pi }_W\left(q^2=0\right)}{M_W^2}\frac{\mathrm{\Pi }_Z\left(q^2=0\right)}{M_Z^2},$$
(35)
where $`\mathrm{\Pi }_X\left(q^2\right)`$ is the $`X`$ boson self energy function. We set $`\mathrm{\Delta }\rho =\rho \rho _{SM}`$, where $`\rho _{SM}`$ is the contribution from the SM $`(t,b)`$ doublet, and calculate $`\rho `$ for our two cases: once including the shifted top and bottom quark zero modes and the first KK level only, and once including the zero modes and the first two KK states. $`\mathrm{\Delta }\rho `$ is then a measure of the deviation from the SM prediction. The results are presented in Fig. 3 as functions of $`\nu `$ for several choices of $`m_1`$. The 95% CL exclusion limit  of $`\mathrm{\Delta }\rho \begin{array}{c}<\hfill \\ \hfill \end{array}2\times 10^3`$ is also indicated. Notice that $`\mathrm{\Delta }\rho `$ increases when we add the second KK level in our analysis; thus adding more states only increases $`\mathrm{\Delta }\rho `$ further, and our neglect of these higher modes is justified. It is clear from the lower graph in Fig. 3 that consistency with the 95% CL exclusion limit restricts $`m_1`$ to the range $`m_1\begin{array}{c}>\hfill \\ \hfill \end{array}25`$ TeV for all values of $`\nu `$ in the previously allowed range, and requires $`m_1\begin{array}{c}>\hfill \\ \hfill \end{array}100`$ TeV when $`\nu 0.4`$. When $`\nu <0.5`$, including the range $`\nu 0.55`$ that we have not presented, the corrections to $`\rho `$ are so large that the perturbative definition of the $`Z`$ and $`W^\pm `$ gauge bosons is no longer valid. We stress that these restrictions are lower bounds on the actual constraints as including more KK levels in our analysis will only strengthen these results. These results imply $`\mathrm{\Lambda }_\pi \begin{array}{c}>\hfill \\ \hfill \end{array}100`$ TeV , with the exact choice depending on the value of $`\nu `$, to avoid unacceptable contributions to $`\mathrm{\Delta }\rho `$; the resulting hierarchy between the EW scale and the fundamental RS scale thus strongly disfavors allowing the third generation quarks to propagate in the bulk.

This restriction applies only to the top and bottom quarks; the first and second generations are much less massive, and the large mixing induced above by the top quark Yukawa coupling does not appear when considering these states. We have numerically checked that the contributions of bulk first and second generation quarks are consistent with the constraints on $`\mathrm{\Delta }\rho `$, and hence the placement of the first two generations in the bulk is still allowed. But, is such a setup motivated? Does any interesting physics result from this construction? The answer to both questions is unequivocally yes, as we will demonstrate in the next section.

## 4 The Third Generation On the Wall and the EW Precision Observables

A handful of authors have attempted to construct models explaining the quark and neutrino mass matrices within the framework of the RS model . These ideas generically require the placement of fermions at different locations in the 5-dimensional bulk. We have already shown that the third generation quarks must lie on the TeV-brane; if we permit the first two generations to propagate in the bulk, can we explain the hierarchy between the Yukawa coupling of the top quark and those of the lighter quarks?

We consider first the coupling of the Higgs to a fermion field confined to the TeV-brane; the relevant action is

$$S_{f\overline{f}H}^{wall}=\lambda ^{wall}d^4xdy\sqrt{G}\{H^{}\psi _D\psi _S^c+\mathrm{h}.\mathrm{c}.\}\delta (yr_c\pi ),$$
(36)
where $`\lambda ^{wall}`$ is the Yukawa coupling of the localized fermion, chosen to be of $`𝒪(1)`$. To derive the 4-dimensional action we must rescale $`\psi e^{3\sigma /2}\psi `$ and $`He^\sigma H`$ as before. The mass of this field is then $`m^{wall}=\lambda ^{wall}v/\sqrt{2}`$. We have shown in Eq. 21 that the 4-d Yukawa coupling that determines the zero-mode mass of a bulk fermion is

$$\lambda ^{bulk}=\frac{\lambda ^{^{}}\left(f_L^{(0)}(\pi )\right)^2e^{kr_c\pi }}{kr_c}.$$
(37)
We now assume that the *fundamental* coupling that enters the 5-d bulk fermion action, $`\lambda ^{^{}}`$, is also of order unity as is $`\lambda ^{wall}`$. The factor $`\left(f_L^{(0)}\right)^2e^{kr_c\pi }/kr_c`$ then suppresses $`\lambda ^{bulk}`$ with respect to $`\lambda ^{wall}`$; we find $`\lambda ^{bulk}\left(10^110^2\right)\lambda ^{wall}`$ when $`0.55\begin{array}{c}<\hfill \\ \hfill \end{array}\nu \begin{array}{c}<\hfill \\ \hfill \end{array}0.35`$, using the zero mode wavefunction given in Eq. 11. Choosing a value of $`\nu `$ in this region, ameliorates the hierarchy between the second and third generation quark Yukawa couplings. To be more explicit, with the top quark on the TeV brane and $`\lambda ^{wall}`$ of order unity, we expect a top mass near its experimental value. On the other hand, for the charm quark in the bulk, we expect a much smaller mass even if the bulk Yukawa coupling $`\lambda ^{}`$ is also of order unity. As an example, assuming $`\lambda ^{}=\lambda ^{wall}`$ and taking $`\nu =0.5`$ one obtains $`m_t/m_c2\pi kr_c70`$ which is within a factor of 2 to 3 of the experimental value. A similar argument applies to the $`m_b/m_s`$ ratio. The localization of the third generation on the TeV brane while keeping the first two in the bulk may thus help explain the fermion mass hierarchy. We do not attempt here to build a more detailed flavor model incorporating the off-diagonal CKM matrix elements, but instead examine the consequences of this simple situation. We focus on the region $`0.6\nu 0.3`$, extending slightly for completeness the range preferred by the quark Yukawa hierarchy. We study next the effects of KK gauge boson mixing on the EW precision observables; we will find that large mixing similar to that appearing in the top quark mass matrix relaxes the upper bound on the Higgs boson mass obtained in the standard EW fit .

The mass terms for the $`W^\pm `$ and $`Z`$ can be obtained using Eq. 18 as a template; we find

$$S_{mass}=\underset{m,n=0}{\overset{\mathrm{}}{}}a_{mn}d^4x\left(m_{W,0}^2W_0^{+(m)}W_0^{(n)}+\frac{1}{2}m_{Z,0}^2Z_0^{(m)}Z_0^{(n)}\right),$$
(38)
where the $`a_{mn}`$ are given by Eq. 19 and we have for notational simplicity omitted the Lorentz indices of the gauge fields. The resulting $`W^\pm `$ mass matrix is

$$_W^2=m_1^2\left(\begin{array}{cccc}a_{11}x_W& a_{12}x_W& a_{13}x_W& \mathrm{}\\ a_{12}x_W& b_1^2+a_{22}x_W& a_{23}x_W& \mathrm{}\\ a_{13}x_W& a_{23}x_W& b_2^2+a_{33}x_W& \mathrm{}\\ \mathrm{}& \mathrm{}& \mathrm{}& \end{array}\right),$$
(39)
where $`m_1`$ is the first KK gauge mass, $`x_W=m_{W,0}^2/m_1^2`$, and $`b_i=m_i/m_1`$ is the ratio of the $`i`$th KK mass to the first. The mass matrix for the $`Z`$ is obtained by substituting $`m_{W,0}m_{Z,0}`$. The subscripts on the fields $`W_0^\pm `$ and $`Z_0`$, and on the masses $`m_{W,0}`$ and $`m_{Z,0}`$, indicate that these are not the physical fields and masses; they are the zero-mode fields and masses in the infinite KK limit. To obtain the physical spectrum we must diagonalize the mass matrices while respecting the appropriate constraints; these are the same as those developed for the interpretation of precision measurements at the $`Z`$-pole . This prescription states that the following quantities are inputs to radiative corrections and to fits to the precision EW data: $`\alpha `$ as measured in Thomson scattering, $`G_F`$ as defined by the muon lifetime, $`M_Z`$ as determined from the $`Z`$ line shape, $`m_t`$ as measured at the Tevatron, and $`m_H`$, which is currently a free parameter. All other observables, such as the $`W^\pm `$ mass, $`M_W`$, and the width for the decay $`Zl^+l^{}`$, $`\mathrm{\Gamma }_l`$, are derived from these measured quantities; we must compare the RS model predictions for these parameters with the values obtained by experiment.

We examine the six relatively uncorrelated observables $`M_W`$, $`\mathrm{sin}^2\theta _{\mathrm{eff}}`$, $`\mathrm{\Gamma }_l`$, $`R_b`$, $`R_c`$, and $`\mathrm{sin}^2\theta _{\nu \mathrm{N}}`$, and discuss in detail our procedure for deriving the RS model predictions for these quantities and then compare these predictions to the measured values. We consider tree level KK and loop level SM contributions to these observables, and assume that contributions from KK loops are higher order and therefore negligible. Our analysis differs slightly from those performed in models with KK gauge bosons arising from $`\mathrm{TeV}^1`$-sized extra dimensions . Here, the parameters $`a_{mn}`$ of Eq. 19 which enter the mass matrices are rather large; the $`a_{1n}`$, with $`n>1`$, have the approximate value $`\sqrt{2\pi kr_c}8.4`$, while the elements $`a_{mn}`$, with $`m,n>1`$ and $`mn`$, have the approximate value $`2\pi kr_c71`$. Although the ratios $`x_{W(Z)}=m_{W,0(Z,0)}^2/m_1^2`$ that appear in the mass matrices may be small, they are multiplied by these large coefficients, and to avoid errors we diagonalize the matrices and handle shifts of the precision observables numerically to all orders in $`x_{W,Z}`$, rather than performing the analysis analytically to $`𝒪(x_{W,Z})`$. This necessitates a truncation of the mass matrices; we work with $`30\times 30`$ matrices, and have verified that increasing the size to $`60\times 60`$ produces a negligible change in our results.

We first determine the parameter $`m_{Z,0}`$ by diagonalizing the $`Z`$ mass matrix and demanding that the lowest eigenvalue reproduce the measured $`Z`$ mass, $`M_Z`$ . Armed with $`m_{Z,0}`$, we consider next the muon lifetime, through which the input parameter $`G_F`$ is defined. The relevant decay is $`\mu ^{}e^{}\nu _e\overline{\nu }_\mu `$. In the SM this proceeds at tree level through $`W`$ exchange; it proceeds here through the exchange of the entire $`W^{(n)}`$ KK tower. $`G_F`$ therefore becomes

$$\frac{G_F}{\sqrt{2}}=\frac{g^2}{8M_W^2}+\frac{g_0^2}{8}\underset{n}{}\frac{c_n}{m_n^2},$$
(40)
where the first term arises from the exchange of the zero mode and the second term from the higher KK states, and the $`c_n`$ encapsulate the couplings of the KK gauge states to zero mode leptons. Some clarification of this expression is required. $`g`$ is the coupling of the *physical* $`W`$ obtained *after* diagonalization, whereas $`g_0`$ is the coupling that appears in the Lagrangian *before* diagonalization. To make this distinction explicit we express $`g`$ as

$$g=g_0\left\{1G(m_{W,0})\right\},$$
(41)
where $`G(m_{W,0})`$ accounts for the admixture of KK states in the physical $`W^{(0)}`$ boson. After EW symmetry breaking, $`g_0^2=4\pi \alpha /s_{w,0}^2`$, where $`s_{w,0}`$ is the sine of the weak mixing angle obtained before including mixing effects: $`s_{w,0}^2=1m_{W,0}^2/m_{Z,0}^2`$. Substituting these relations into Eq. 40, we arrive at the condition

$$1=\frac{\pi \alpha }{\sqrt{2}G_FM_W^2s_{w,0}^2}\left\{1G(m_{W,0})\right\}^2+\frac{\pi \alpha }{\sqrt{2}s_{w,0}^2}H(m_{W,0}),$$
(42)
where we have introduced the dimensionless quantity

$$H(m_{W,0})=\frac{1}{G_F}\underset{n}{}\frac{c_n}{m_n^2}.$$
(43)
In the SM, after radiative corrections are included,

$$\frac{\pi \alpha }{\sqrt{2}G_F}\frac{\pi \alpha }{\sqrt{2}G_F\left(1\mathrm{\Delta }r\right)}=m_{W,SM}^2\left[1\frac{m_{W,SM}^2}{M_Z^2}\right].$$
(44)
To incorporate radiative corrections in our analysis, we make this substitution in Eq. 42, and use the $`m_{W,SM}`$ calculated by ZFITTER  using $`M_Z`$ as an input. We find the relation

$$1=\frac{m_{W,SM}^2\left[1\frac{m_{W,SM}^2}{M_Z^2}\right]}{M_W^2\left[1\frac{M_W^2}{M_Z^2}\right]}\left\{1G(m_{W,0})\right\}^2+\frac{\pi \alpha }{\sqrt{2}\left(1\frac{m_{W,0}^2}{m_{Z,0}^2}\right)}H(m_{W,0}).$$
(45)
The only unknown quantity in this equation is $`m_{W,0}`$; the physical $`W^\pm `$ mass, $`M_W`$, is derived from $`m_{W,0}`$ through diagonalization of the $`W^\pm `$ mass matrix. We now scan over $`m_{W,0}`$ until we find a solution to this equation; the $`m_{W,0}`$ that furnishes this solution also predicts a $`M_W`$ that can be compared with experiment.

We next compute the KK contributions to the effective coupling $`\mathrm{sin}^2\theta _{eff}`$, which appears in the dressed $`Zl^+l^{}`$ vertex . In the SM,

$$\mathrm{sin}^2\theta _{eff,SM}=\kappa ^Z\mathrm{sin}^2\theta _{w,SM},$$
(46)
where $`\theta _w`$ is the on-shell weak mixing angle, $`\mathrm{sin}^2\theta _{w,SM}=1m_{W,SM}^2/M_Z^2`$, and $`\kappa ^Z`$ contains a subset of the radiative corrections to the decay $`Zl^+l^{}`$. In the RS model, the weak mixing angle that appears in the $`Zl^+l^{}`$ vertex is $`s_{w,0}`$; this is unaffected by diagonalization because $`s_{w,0}`$ enters the coupling of every KK excitationstate. The RS model expression for $`\mathrm{sin}^2\theta _{eff}`$ is

$$\mathrm{sin}^2\theta _{eff}=\kappa ^Zs_{w,0}^2=\mathrm{sin}^2\theta _{eff,SM}\left(\frac{1\frac{m_{W,0}^2}{m_{Z,0}^2}}{1\frac{m_{W,SM}^2}{M_Z^2}}\right),$$
(47)
where in the last step we have incorporated the ZFITTER predictions for $`\mathrm{sin}^2\theta _{eff,SM}`$ and $`m_{W,SM}`$ to correctly account for the SM radiative corrections.

The shifts of the remaining observables occur in a similar fashion as in the two examples given above, and hence we discuss them only briefly here. The width of the decay $`Z\overline{f}f`$ is

$$\mathrm{\Gamma }_f=\frac{g^2M_Z}{96\pi c_w^2}C_f\left\{\left[14|Q_f|\mathrm{sin}^2\theta _{eff}+8Q_f^2\mathrm{sin}^4\theta _{eff}\right]\left(1+\frac{2m_f^2}{M_Z^2}\right)3\frac{m_f^2}{M_Z^2}\right\},$$
(48)
where $`C_f`$ encapsulates kinematic factors, color sums for final state quarks, and factorizable radiative corrections . This formula is valid in both the SM and the RS model, with the proviso that in the RS case $`g`$ describes the coupling of the $`Z^{(0)}`$ obtained after diagonalization, $`c_wc_{w,0}`$ and $`\mathrm{sin}^2\theta _{eff}`$ is given by that described in the previous paragraph. Our previous results can be adapted to compute the shift in $`\mathrm{\Gamma }_l`$. The change in the ratio of the $`Z\overline{c}c`$ width to the total hadronic width, $`R_c=\mathrm{\Gamma }_c/\mathrm{\Gamma }_h`$, can also be computed by following the outline presented for calculating the $`\mathrm{\Gamma }_l`$ shift. The derivation of the shift in $`R_b=\mathrm{\Gamma }_b/\mathrm{\Gamma }_h`$ proceeds similarly, except that the couplings of the higher gauge KK modes to the brane localized bottom quarks are those presented in Eqs. 15 and 16. Finally, $`\mathrm{sin}^2\theta _{\nu \mathrm{N}}`$ is determined experimentally through the measurement of $`R`$, which is the following ratio of neutrino-nucleon neutral and charged current scattering events:

$$R=\frac{\sigma _{NC}^\nu \sigma _{NC}^{\overline{\nu }}}{\sigma _{CC}^\nu \sigma _{CC}^{\overline{\nu }}}.$$
(49)
It becomes

$$R=\frac{1}{2}\mathrm{sin}^2\theta _{\nu N}$$
(50)
at tree level in the SM, where the $`W^\pm `$ and $`Z`$ coupling constants have cancelled in the ratio. When RS corrections are included, the gauge boson couplings no longer cancel because of different mixing effects in the $`W^\pm `$ and $`Z`$ mass matrices, and $`\mathrm{sin}^2\theta _{\nu N}s_{w,0}^2`$. Again, these corrections to $`\mathrm{sin}^2\theta _{\nu \mathrm{N}}`$ can be easily obtained from our above results.

Having computed these corrections, we can now compare the RS model predictions for the precision observables with the values actually measured. We perform a $`\chi ^2`$ fit to the data, with the Higgs boson mass $`m_H`$ and the first KK gauge mass $`m_1`$ as free parameters. The LEP Electroweak Working Group has quoted an upper limit on the Higgs mass in the SM of $`m_H<222`$ GeV at the 95% confidence level , which we find corresponds to $`\chi ^2=23.3`$. Following , we normalize our results by choosing this $`\chi ^2`$ value as our benchmark; we claim that the predictions are disfavoured at the 95% CL if $`\chi ^2>23.3`$, and that the model fits the precision data otherwise. We use the input parameter values

$`M_Z`$ $`=`$ $`91.1875\mathrm{GeV},`$
$`G_F`$ $`=`$ $`1.16637\times 10^5\mathrm{GeV}^2,`$
$`\alpha (m_e)`$ $`=`$ $`1/137.036,`$ (51)
and the experimental observable values and errors

$`M_W`$ $`=`$ $`80.451\pm 0.033\mathrm{GeV},`$
$`\mathrm{sin}^2\theta _{eff}`$ $`=`$ $`0.23152\pm 0.00017,`$
$`\mathrm{\Gamma }_l`$ $`=`$ $`83.991\pm 0.087\mathrm{MeV},`$
$`R_b`$ $`=`$ $`0.21646\pm 0.00065,`$
$`R_c`$ $`=`$ $`0.1719\pm 0.0031,`$
$`\mathrm{sin}^2\theta _{\nu \mathrm{N}}`$ $`=`$ $`0.2277\pm 0.0016,`$ (52)
as presented in . The results of these fits are presented in Fig. 4 as functions of both $`m_H`$ and $`m_1`$, and for four representative values of the fermion bulk mass parameter $`\nu `$. We have allowed $`m_H`$ to range from 115 GeV to 1 TeV; higher Higgs masses are inconsistent with perturbative unitarity. This bound is modified slightly by KK gauge boson and graviton exchanges, but we have neglected these effects here. The six observables contribute to the fit with widely varying strengths; $`\mathrm{sin}^2\theta _{eff}`$ is very sensitive to deviations arising from RS physics throughout the entire $`m_H`$, $`m_1`$ region, while $`R_c`$ does not significantly affect the $`\chi ^2`$ value for any choice of parameters. $`R_b`$ is drastically altered when $`\nu 0.5`$, where the light fermion couplings to KK gauge states either vanish or become small, but is less affected for larger values of $`\nu `$. $`M_W`$, $`\mathrm{\Gamma }_l`$, and $`\mathrm{sin}^2\theta _{\nu \mathrm{N}}`$ are somewhat less sensitive than $`\mathrm{sin}^2\theta _{eff}`$, and vary in relative importance as $`m_H`$ and $`m_1`$ are changed. The allowed values of $`m_H`$ vary with $`\nu `$, but it is clear from Fig. 4 that for $`\nu 0.5`$ Higgs masses in the range $`300600`$ GeV are permitted for $`n=1`$ KK gauge masses of $`1115`$ TeV. A heavy Higgs has the effect of decreasing $`M_W`$, while the RS mixing effects increase it, and this compensation allows the predicted values of $`M_W`$, $`\mathrm{sin}^2\theta _{eff}`$, and $`\mathrm{\Gamma }_l`$ to be brought into good agreement with the measured values by tuning $`m_1`$. Shifts in $`R_b`$ arising from the confinement of the third generation quarks to the TeV-brane prevent larger values of $`m_H`$ from providing a good fit to the EW precision data. For each choice of $`\nu `$ and $`m_H`$ there exists a range of allowed $`m_1`$ values that fits the EW precision data; the lowest allowed value of $`m_1`$ as a function of $`\nu `$ is presented in Fig. 5 for several choices of $`m_H`$. The drastic difference between the $`m_H=300`$ and $`400`$ GeV curves arises from the sharp distinction between allowed and disallowed KK masses imposed by the cut at $`\chi ^2=23.3`$. The sharp rise for lower $`\nu `$ values and higher Higgs masses is due almost entirely to $`R_b`$. We present in Table 2 a summary of the allowed $`m_1`$ ranges for various choices of $`\nu `$ and $`m_H`$.

This relaxation of the upper bound on $`m_H`$ is akin to that observed in ; the factors of 8.4 and 71 that appear in the off-diagonal elements of the $`W^\pm `$ and $`Z`$ mixing matrices here allow the effect to occur for much larger KK masses. At this point the reader may wonder whether these high $`m_1`$ values can be probed at future colliders. We will show in the next section that they are indeed invisible at the LHC; however, the large KK gauge boson couplings to third generation quarks produces observable effects over most of the allowed parameter space at future $`e^+e^{}`$ colliders.

## 5 Searches at the LHC

We now discuss the prospects for detecting the gauge KK states which are consistent with our EW fit at the LHC. The primary discovery mode for new heavy gauge bosons at hadron colliders is high invariant mass Drell-Yan lepton pair production; at the LHC the relevant processes are $`pp\gamma ^{(1)},Z^{(1)}\mu ^+\mu ^{}`$, $`e^+e^{}`$. The contributing parton level processes are $`q\overline{q}\mu ^+\mu ^{}`$, $`e^+e^{}`$. We present $`d\sigma /dm_{ll}`$ for this process (with $`m_{ll}`$ being the invariant mass of the final state lepton pair) in Fig. 6 for the parameter choices $`\nu =0.6,0.5,0.4,0.3`$ and $`m_1=8,10`$ TeV. These values are representative of the allowed region for $`\nu `$, but the gauge KK masses are lighter than those allowed by the EW fit. If the rates are unobservable at these points in parameter space, then the RS effects are undetectable for all interesting cases. The resonances are wide in this case primarily because of the large couplings of the KK gauge states to top and bottom quarks. With the $`100\mathrm{fb}^1`$ of integrated luminosity envisioned for the LHC, the KK contributions to Drell-Yan production are indeed invisible. We present the expected number of excess events including both the $`\mu ^+\mu ^{}`$ and $`e^+e^{}`$ channels for this value of integrated luminosity and for the two choices of $`\nu `$ which produce the largest cross section in Table 3. Here, we have integrated over the invariant mass bins in which there is an excess of events over the SM predictions; this corresponds to the cuts $`m_{ll}\begin{array}{c}>\hfill \\ \hfill \end{array}5`$ TeV when $`m_1=8`$ TeV and $`\nu =0.3`$, $`m_{ll}\begin{array}{c}>\hfill \\ \hfill \end{array}6.5`$ TeV when $`m_1=8`$ TeV and $`\nu =0.4`$, $`m_{ll}\begin{array}{c}>\hfill \\ \hfill \end{array}6`$ TeV when $`m_1=10`$ TeV and $`\nu =0.3`$, and $`m_{ll}\begin{array}{c}>\hfill \\ \hfill \end{array}8`$ TeV when $`m_1=10`$ TeV and $`\nu =0.4`$. We have not attempted to study the depletion of events at lower $`m_{ll}`$ because the event rates at the affected invariant masses are too low. Two effects are hindering the detection of the KK contributions: the small couplings of zero mode fermions to KK gauge states for $`\nu 0.5`$, and the high KK masses which require the parton subprocesses to occur at energies where the quark distribution functions are small. Even with an order of magnitude increase in integrated luminosity, the production of the first gauge KK excitation that is consistent with the EW precision data is unobservable.

Another possible production mechanism for the KK gauge bosons at the LHC is $`W^+W^{}`$ fusion, $`ppWW+2`$ jets$`V^{(1)}+2`$ jets. The relevant triple gauge couplings, $`W^{+(0)}W^{(0)}\gamma ^{(1)}`$ and $`W^{+(0)}W^{(0)}Z^{(1)}`$, are induced by mixing effects. We present the strengths of these vertices normalized to the SM couplings $`W^+W^{}\gamma `$ and $`W^+W^{}Z`$ in Fig. 7. Very slight $`\nu `$ and $`m_H`$ dependences enter these vertices; here we fix $`\nu =0.3`$ and $`m_H=115`$ GeV, which maximizes their strength. For $`m_111`$ TeV, these couplings are a fraction, $`10^3`$, of their SM strengths. This, and the fact that the $`W^+W^{}`$ fusion process is higher order in the EW coupling constant, render this a poor place in which to search for KK effects.

The only remaining possibility for detecting the KK states at the LHC is via deviations in top and bottom quark production. These processes are dominated at high energies by gluon initiated interactions; however, these do not receive any modifications from gluon KK states since $`g^{(0)}g^{(0)}g^{(n)}`$ couplings do not exist. We thus only examine top quark production, which receives a larger contribution from quark initiated processes, and where the large couplings of the KK gauge states to third generation quarks enter. The invariant mass distribution is presented in Fig. 8 for $`m_1=10`$ TeV and $`\nu =0.4`$. Since the KK couplings to wall fermions do not decrease with KK level, we have checked that the contributions from including multiple states in the KK tower does not significantly enhance the effect. In fact, summing the first five KK contributions slightly decreases the cross section from that where only the first level is included due to the factor of $`(1)^n`$ that enters the coupling of the $`n`$th KK level to top quarks. The expected number of excess events at the LHC is $`0.14`$, assuming a cut on the invariant mass of the final state top quarks of $`m_{tt}\begin{array}{c}>\hfill \\ \hfill \end{array}3.5`$ TeV. As in the previous case of Drell-Yan production, this event rate is undetectable even with an order of magnitude increase in integrated luminosity. The slight depletion of events at lower invariant masses is similarly unobservable. We must therefore conclude that the KK excitations which relax the precision EW upper bound on the Higgs mass are invisible at the LHC.

## 6 Searches at Future TeV-scale Linear Colliders

We examine here whether KK gauge boson exchanges can be observed at a future $`e^+e^{}`$ collider with $`\sqrt{s}=5001000`$ GeV and $`L=5001000\mathrm{fb}^1`$. Since the anticipated center-of-mass energies are well below the 11 TeV KK gauge mass defining the lower edge of the allowed range from the EW fit, we study the off-resonance modification of fermion pair production, $`e^+e^{}f\overline{f}`$. $`Z`$ pair production receives no KK gauge contribution, while the $`\gamma ^{(1)}`$ and $`Z^{(1)}`$ exchanges in $`e^+e^{}W^+W^{}`$ suffer from the weak triple gauge vertices displayed in Fig. 7. We perform a $`\chi ^2`$ fit to the total rate, binned angular distribution, and binned $`𝒜_{LR}`$ for fermion pair production to estimate the search reaches possible at TeV-scale linear colliders. We assume an 80% electron beam polarization, a $`10^{}`$ angular cut, statistical errors and a 0.1% luminosity error. We also use the following reconstructions efficiencies: a 100% $`\tau `$ efficiency, a 70% $`b`$ quark efficiency, a 50% $`t`$ quark efficiency, and a 40% $`c`$ quark efficiency. The $`\chi ^2`$ values obtained in this analysis are shown in Fig. 9 for several choices of $`\sqrt{s}`$ and $`L`$.

We see from Fig. 9 that the effects of KK exchange exceed the 95% CL exclusion limit for all $`\nu `$ values in the allowed region and for $`m_115`$ TeV; the modifications when $`\nu 0.4`$ reach the $`5\sigma `$ discovery limit. The parameter space $`\nu 0.5`$ and $`m_1>15`$ TeV, part of which provides a good fit to the EW precision data, falls below the exclusion limit. It is possible that this difficulty can be alleviated with the inclusion of more observables. We note that a small hierarchy between the EW scale and $`\mathrm{\Lambda }_\pi `$ begins to develop in this region, and it is consequently not as favored as the $`m_115`$ TeV range. We note that the ordering of the $`\nu =0.6`$ and $`0.5`$ curves in the lower figure of Fig. 9 is correct.

We now subject this model to future high precision tests. Planned $`e^+e^{}`$ colliders are designed for operation on the $`Z`$-pole for a period sufficient to collect $`10^9`$ $`Z`$ events. This program, known as GigaZ, will reduce the error in $`\mathrm{sin}^2\theta _{eff}`$ to the $`10^5`$ level and the error in $`R_b`$ by a factor of 5 . A phase of operation on the $`W^\pm `$ pair production threshold is also planned, which will reduce the error in the measurement of $`M_W`$ to 6 MeV. We now return to our analysis of EW precision data and study the effects of this error reduction, keeping the central values for the observables unchanged from the present and focusing on $`\mathrm{sin}^2\theta _{eff}`$, $`M_W`$, and $`R_b`$. Figures 10 and 11 display our results in the $`\mathrm{sin}^2\theta _{eff}`$ versus $`M_W`$ plane and the $`\mathrm{sin}^2\theta _{eff}`$ versus $`R_b`$ plane for $`\nu =0.5`$ and $`0.4`$. These figures show the current and expected experimental precisions, SM predictions, and RS model results for several different Higgs masses and $`m_1`$ choices.

It is difficult to predict what the status of fits to the EW precision data will be after the GigaZ program concludes, as a small shift in the experimental central values assisted by the small anticipated errors can drastically alter the current situation. If the central values remain unchanged, it is clear from these figures that the improved precision in the measurement of $`R_b`$ will disfavor the heavier Higgs solutions, and require a large value of $`m_1`$, which reintroduces a hierarchy between $`\mathrm{\Lambda }_\pi `$ and the EW scale. However, the RS predictions for $`\mathrm{sin}^2\theta _{eff}`$ and $`M_W`$ match experiment better than the SM results and can accommodate a heavy Higgs, and the global fit to the EW observables may prefer this solution. Whatever scenario is realized, it is certainly true that the entire parameter space, including the region inaccessible in off-resonance fermion pair production, can be probed at GigaZ.

## 7 Constraints from FCNCs

The placement of fermions in different locations in extra dimensions, on the TeV brane or in the bulk, leads to potentially dangerous FCNC since the Glashow-Weinberg-Paschos conditions  for the natural absence of FCNC are no longer satisfied. These conditions are violated automatically whenever fermions of different generations are treated asymmetrically by some form of new physics and mixing occurs between the relevant states. Within the RS scenario that we have constructed, these FCNC can arise from a number of potential sources, not all of which present the same level of danger. A detailed analysis of FCNC effects is certainly beyond the scope of this paper and requires a specific flavor model as input; we simply outline the potential sources of FCNC and provide a few estimates of their size.

The most obvious sources of FCNC are from the exchanges of gauge bosons. The states in the gauge KK towers can feel the different fermion generation localities, and through intergenerational mixing can then induce FCNC. Furthermore, the couplings of the wall fields to the KK gauge states are enhanced by a factor of $`\sqrt{2\pi kr_c}`$. Since zero mode KK gauge states in the limit of vanishing mixing are constrained by construction to have the same couplings to fermions as do the SM gauge bosons, such fields can only induce FCNC through the small admixture of KK weak eigenstates introduced by mixing. These effects are suppressed by small mixing angles, and are not as important as those arising from the KK towers themselves. We therefore expect that the KK gauge state contributions represent the greatest source of potentially dangerous FCNC.

Graviton KK towers can also probe the different locations of the SM fermion generations and induce FCNC-like couplings. However, in this case the potentially dangerous contributions are much smaller since ($`i`$) graviton-induced FCNC take the form of dimension-8 operators, in contrast to the dimension-6 KK gauge contributions, and lead to amplitudes which are suppressed by factors of order $`m_{K,D,B}^2/\mathrm{\Lambda }_\pi ^2`$. This is an enormous degree of suppression since we have shown that $`\mathrm{\Lambda }_\pi \begin{array}{c}>\hfill \\ \hfill \end{array}10`$ TeV in the scenario presently under consideration. ($`ii`$) Unlike KK gauge fields, the graviton KK couplings to wall fields are not enhanced by the factor $`\sqrt{2\pi kr_c}`$.

How large are the KK gauge tower contributions? The answer depends upon which gauge boson we are examining. We neglect in this analysis the small mixing between the first and second generation fermions and their KK towers. Let $`g_{L,R}^a`$ represent the couplings of a particular fermion with electric charge $`Q`$ to one of the neutral SM gauge bosons labeled by the index $`a`$. We write the fermion couplings to KK gauge states as $`g_{L,R}^ac^n(\nu _i)`$, where $`\nu _i`$ is the $`i`$th generation bulk mass parameter and $`n`$ labels the gauge KK tower level. Note that the functions $`c^n`$ in the present model are independent of chirality and the gauge boson under consideration. The fact that the $`c^n(\nu _i)`$ are different for each $`i`$ generates the FCNC terms when we transform to the mass eigenstate basis. Let $`U_{L,R}`$ represent the matrices performing the bi-unitary transformation required to diagonalize the appropriate fermion mass matrix. The off-diagonal couplings in the mass eigenstate basis are then given by

$$(Q_{L,R}^n)_{ij}^a=g_{L,R}^a\underset{k}{}(U_{ik})_{L,R}c^n(\nu _k)(U_{kj}^{})_{L,R}.$$
(53)
For the specific model discussed in the previous sections we have $`c^n(\nu _1)=c^n(\nu _2)c^n(\nu _3)`$, and we use the unitarity of the $`U`$’s to rewrite these couplings as

$$(Q_{L,R}^n)_{ij}^a=g_{L,R}^a[c^n(\nu _3)c^n(\nu _1)](U_{i3}U_{3j}^{})_{L,R}.$$
(54)
With the third generation on the wall and the first and second in the bulk in the region $`0.6\nu _10.3`$, it is clear that $`|c^n(\nu _3)c^n(\nu _1)|\sqrt{2\pi kr_c}`$ for all $`n`$; at worst, the size of the off-diagonal couplings in our model is given by

$$(Q_{L,R}^n)_{ij}^a=\sqrt{2\pi kr_c}g_{L,R}^a(U_{i3}U_{3j}^{})_{L,R},$$
(55)
which is independent of $`n`$. The $`U_{ij}`$ arise from some complete theory of flavor that must reproduce the experimentally measured CKM matrix $`V_{ij}`$. We therefore expect $`U_{ij}V_{ij}`$ and ee adopt this approximation in our estimates below.

The most stringent constraints on FCNC arise from low energy processes such as meson-antimeson mixing and rare decays ; we present here our estimate for $`K\overline{K}`$ mixing. The above interaction generates a coupling which can be symbolically written as

$$=2\pi kr_c\underset{a}{}\underset{n}{}(J_L^a+J_R^a)^2/m_n^2,$$
(56)
where $`J_{L,R}^a=g_{L,r}^aV_{i3}V_{3j}^{}\overline{f}_i\gamma _\mu P_{L,R}f_j`$, $`m_n`$ is the mass of the $`n`$th KK gauge state, and we have summed all KK contributions. Recalling the lore that we can accurately approximate the matrix element of the two currents in the vacuum insertion approximation, we see that the KK gluon towers do not contribute. This is due to the fact that these states only couple to currents with non-zero color while both the meson and the vacuum are color singlets. Thus we need to consider only the $`Z`$ and $`\gamma `$ tower exchanges. Using $`_nm_n^21.5m_1^2`$ , $`V_{13}V_{32}^{}A^2(1\rho )^2\lambda ^5`$ in the Wolfenstein parameterization and

$$<K|J_LJ_R|\overline{K}>=\mathrm{\Omega }_K<K|J_LJ_L|\overline{K}>=\mathrm{\Omega }_K<K|J_RJ_R|\overline{K}>,$$
(57)
with $`\mathrm{\Omega }_K7`$  for the current-current matrix element, we arrive at

$$\frac{\mathrm{\Delta }m_{KK}^{RS}}{\mathrm{\Delta }m_{KK}^{SM}}0.0098[1+0.73\mathrm{\Omega }_K]\left(\frac{11TeV}{m_1}\right)^20.06,$$
(58)
which is within the uncertainty of the SM result . From this estimate we see that, at least for the $`K\overline{K}`$ system, the RS FCNC contributions are rather small. We have also studied $`B\overline{B}`$ mixing and obtain similar results.

Once a realistic theory of flavor within this RS model context is constructed, we can perform a more detailed and quantitative analysis of the potential impact of FCNC. It will be interesting to examine if existing bounds can provide further constraints on the RS model parameters within such a framework.

## 8 Discussions and Conclusions

In this paper we have re-examined the placement of SM fermions in the full 5-dimensional bulk of the Randall-Sundrum spacetime. We have found that mixing between the top quark zero mode and its KK tower, induced by the large top quark mass, yields shifts in the $`\rho `$ parameter that are inconsistent with current measurements. To obviate these bounds we must take the fundamental RS scale $`\mathrm{\Lambda }_\pi \begin{array}{c}>\hfill \\ \hfill \end{array}100`$ TeV, reintroducing the hierarchy between the Planck and EW scales and thus destroying the original motivation for the RS model. We instead proposed a mixed scenario which localizes the third generation of quarks, and presumably leptons, on the TeV-brane and allows the lighter two generations to propagate in the RS bulk. For values of the bulk mass parameter in the region $`0.55\begin{array}{c}<\hfill \\ \hfill \end{array}\nu \begin{array}{c}<\hfill \\ \hfill \end{array}0.35`$, the same values allowed by both contact interaction searches and $`\rho `$ parameter constraints arising from the first two generations, the fermions mass hierarchies $`m_c/m_t`$ and $`m_s/m_b`$ are naturally reproduced.

We next explored the consequences of this proposal for current precision EW measurements. We studied modifications of the electroweak observables caused by both mixing of the SM gauge bosons with their corresponding KK towers and the exchanges of higher KK states; we found that with KK masses $`m_111`$ TeV and bulk mass parameters $`\nu 0.5,0.4`$ a Higgs boson with mass $`m_H\begin{array}{c}<\hfill \\ \hfill \end{array}500`$ GeV can provide a good fit to the precision electroweak data. An analysis of the fit showed that the large couplings between the zero mode bottom quark and KK gauge bosons induced large shifts in $`R_b`$ that prevented a heavier Higgs from being consistent with the precision data.

We then examined the signatures of this scenario at future high energy colliders. We found that the parameter region consistent with the precision electroweak data does not lead to any new physics signatures at the LHC; the expected event excess in both Drell-Yan and gauge boson fusion processes are statistically insignificant with the envisioned integrated luminosities, and the predicted modification of the $`t\overline{t}`$ production cross section is similarly unobservably small. The only new physics that the LHC would possibly observe is a Higgs boson apparently heavier than that allowed by the SM electroweak fits. By contrast, the parameter range $`m_1\begin{array}{c}<\hfill \\ \hfill \end{array}15`$ TeV and $`\nu \begin{array}{c}<\hfill \\ \hfill \end{array}0.3`$ can be probed in fermion pair production processes at a future $`e^+e^{}`$ collider with center-of-mass energy of $`5001000`$ GeV, while the region $`m_1\begin{array}{c}<\hfill \\ \hfill \end{array}2530`$ TeV and $`\nu \begin{array}{c}<\hfill \\ \hfill \end{array}0.3`$ is testable at GigaZ. For larger KK first excitation masses, we reintroduce the hierarchy between $`\mathrm{\Lambda }_\pi `$ and the electroweak scale.

Finally, we considered the possible constraints on this scenario arising from low energy FCNC. The asymmetric treatment of the three fermion generations allows KK $`Z`$-boson exchanges to mediate FCNC interactions. We estimated the contributions of such effects to meson-antimeson mixing, and found that their size is within the theoretical errors inherent to meson mixing. However, a detailed analysis of FCNC effects requires a full model of flavor, which we have not constructed.

In summary, we have found that the experimental restrictions on placing SM matter in the RS bulk lead naturally to a very interesting region of parameter space. This parameter region provides a geometrical origin for the fermion Yukawa hierarchies, and allows a heavy Higgs boson to be consistent with precision measurements while remaining otherwise invisible at the LHC. We believe that such features render this model worthy of further study.

Acknowledgements

F.P. would like to thank K. Melnikov for discussions. The work of F.P. was supported in part by the National Science Foundation Graduate Research Program.