text:
# Quark-Lepton Unification and Lepton Flavor Non-Conservation from a TeV-scale Seesaw Neutrino Mass Texture

## I Introduction

In a previous paper LOTW1 , we argued that the $`Z`$-pole data from $`e^+e^{}`$ colliders LEP/SLD:2002  and the $`\nu _\mu `$ ($`\overline{\nu }_\mu `$) scattering data from the NuTeV experiment at Fermilab Zeller:2001hh  could be brought into agreement if the Higgs boson is heavy, and if the $`Z\nu _{\mathrm{}}\nu _{\mathrm{}}`$ couplings are suppressed by a factor of $`(1\epsilon _{\mathrm{}})`$ while the $`W\mathrm{}\nu _{\mathrm{}}`$ couplings are suppressed by a corresponding factor of $`(1\epsilon _{\mathrm{}}/2)`$, where $`\mathrm{}=e,\mu ,\tau `$. Assuming lepton flavor universality of the suppression parameters $`\epsilon _e=\epsilon _\mu =\epsilon _\tau \epsilon `$, the value of $`\epsilon `$ required to fit the data is

$$\epsilon =0.0030\pm 0.0010.$$
(1)
Such a suppression could result if the neutrinos mix with heavy gauge singlet states. However, the mixing angle required in the case of mixing with a single heavy state,

$$\theta =0.055\pm 0.010,$$
(2)
is much larger than would be expected from a conventional seesaw model seesaw . Explicitly, the seesaw model relates the mass eigenvalues and the mixing angle by

$$\frac{m_{\mathrm{light}}}{m_{\mathrm{heavy}}}\theta ^2.$$
(3)
Reasonable values of the masses,

$$m_{\mathrm{light}}0.1\mathrm{eV},m_{\mathrm{heavy}}100\mathrm{GeV},$$
(4)
result in a mixing angle

$$\theta 10^6,$$
(5)
which is too small by orders of magnitude.

This difficulty is absent in models which introduce more than one sterile neutrino per active flavor Gronau:ct . However, even in models with the same number of sterile neutrinos as active neutrinos, Chang, Ng, and Ng Chang:1994hz  have shown that the seesaw constraint can be circumvented by allowing mixing among generations: there are sufficient degrees of freedom in the neutrino mixing matrix that the masses of the light and heavy neutrino states can be tuned independently of their mixing angles.

In the following, we will present explicit examples of mass matrices (textures) that demonstrate this feature. We show that the light neutrinos can be made massless while maintaining large mixing angles with the heavy states, the masses of which can also be adjusted at will. We also derive the relation between the suppression parameter $`\epsilon `$, mixing angles, and the mass parameters in the textures.

A natural question to ask is: what are the masses of the heavy states in these models? Since the mixing angles are dimensionless, they fix only certain ratios between the mass parameters in the texture. The masses of the heavy states are therefore, in principle, unconstrained. Quark–lepton unification, as appears, for example, in the Pati–Salam model Pati:uk , can fix the order of the Dirac masses in the texture to be $`100`$ GeV. In combination with the experimental limits on the suppression parameter, Eq. (1), this allows the heavy state masses to be of order a few TeV. Such states may be directly observable at the Large Hadron Collider (LHC), currently under construction at CERN.

The effect of the heavy states on the flavor–changing process $`\mu e\gamma `$ Chang:1994hz ; mutoegammaold ; mutoegamma ; mutoegammarecent  places a strong constraint on the model. The upper bound on the branching fraction

$$B(\mu e\gamma )=\frac{\mathrm{\Gamma }(\mu e\gamma )}{\mathrm{\Gamma }(\mu e\nu \overline{\nu })},$$
(6)
from the MEGA collaboration mega  requires that the product $`\epsilon _e\epsilon _\mu `$ be strongly suppressed. Since either $`\epsilon _e`$ or $`\epsilon _\mu `$ must remain sizable to account for the discrepancy between the $`Z`$-pole and NuTeV data, the violation of lepton flavor universality among the $`\epsilon _{\mathrm{}}`$’s is required.

Finally, we compare our results to those of a recent analysis of our proposal by Glashow Glashow:2003wt .

## II One Generation Case

First, let us trace the problem with the seesaw mechanism to its origin. We assume the same number of $`SU(2)_L`$ active neutrinos ($`\nu `$) and sterile neutrinos ($`\chi `$). We further adopt the ‘seesaw’ Ansatz in which the Majorana masses of the active neutrinos are all set to zero. For a single generation, the seesaw mass matrix can only be

$$\left[\nu \chi \right]\left[\begin{array}{cc}0& m\\ m& M\end{array}\right]\left[\begin{array}{c}\nu \\ \chi \end{array}\right].$$
(7)
Note that there are only two free parameters: $`m`$ and $`M`$. This mass matrix can be diagonalized with the orthogonal transformation $`O`$:

$$O^\mathrm{T}\left[\begin{array}{cc}0& m\\ m& M\end{array}\right]O=\left[\begin{array}{cc}mt& 0\\ 0& m/t\end{array}\right],$$
(8)
in which

$$O=\left[\begin{array}{cc}c& s\\ s& c\end{array}\right],$$
(9)
and

$$s=\mathrm{sin}\theta ,c=\mathrm{cos}\theta ,t=\mathrm{tan}\theta ,\mathrm{tan}2\theta =\frac{2m}{M}.$$
(10)
Alternately, to make both mass eigenvalues positive we can diagonalize using the unitary matrix $`U`$:

$$U=\left[\begin{array}{cc}ic& s\\ is& c\end{array}\right]$$
(11)
which yields

$$U^\mathrm{T}\left[\begin{array}{cc}0& m\\ m& M\end{array}\right]U=\left[\begin{array}{cc}mt& 0\\ 0& m/t\end{array}\right].$$
(12)
The product of the two mass eigenvalues is always equal to $`m^2`$ regardless the value of $`M`$. Thus the nomenclature ‘seesaw model’: enhancing one of the mass eigenvalues suppresses the other. In particular, when $`mM`$ the two mass eigenvalues are

$`m_{\mathrm{light}}`$ $``$ $`mt={\displaystyle \frac{M}{2}}\left(\sqrt{1+{\displaystyle \frac{4m^2}{M^2}}}1\right){\displaystyle \frac{m^2}{M}},`$ (13)
$`m_{\mathrm{heavy}}`$ $``$ $`{\displaystyle \frac{m}{t}}={\displaystyle \frac{M}{2}}\left(\sqrt{1+{\displaystyle \frac{4m^2}{M^2}}}+1\right)M.`$ (14)
The mass eigenstates and the original generation eigenstates are related through

$$\left[\begin{array}{c}\nu \\ \chi \end{array}\right]=U\left[\begin{array}{c}n\\ N\end{array}\right]=\left[\begin{array}{cc}\hfill icn+sN& \\ \hfill isn+cN& \end{array}\right],$$
(15)
where the light and heavy states are denoted $`n`$ and $`N`$, respectively. The suppression factor $`(1\epsilon )`$ is just the squared magnitude of the coefficient of the light state $`n`$ in the active neutrino $`\nu `$,

$$1\epsilon =\mathrm{cos}^2\theta 1\theta ^21\frac{m^2}{M^2}.$$
(16)
Observe that

$$\theta ^2\frac{m^2}{M^2}\frac{m_{\mathrm{light}}}{m_{\mathrm{heavy}}},$$
(17)
which is the problem anticipated in the introduction. This stems from the fact that the original mass matrix had just two free parameters, so the two mass eigenvalues $`m_{\mathrm{light}}m^2/M`$, $`m_{\mathrm{heavy}}M`$, and the mixing angle $`\theta m/M`$ are necessarily related.

## III Two Generation Case

With the introduction of additional generations, the mass matrix has more free parameters and the mixing angle and the mass eigenvalues are in general independent. Consider first the two generation case. The most general seesaw mass matrix is

$$\left[\begin{array}{cc}\mathrm{𝟎}_{2\times 2}& 𝐦_{2\times 2}\\ 𝐦_{2\times 2}^\mathrm{T}& 𝐌_{2\times 2}\end{array}\right],$$
(18)
where $`𝐦_{2\times 2}`$ is the Dirac mass matrix which couples the active and sterile neutrinos, and $`𝐌_{2\times 2}`$ is the symmetric Majorana mass matrix for the steriles. As a particular instance, take the following generation universal form:

$$\left[\nu _1\nu _2\chi _1\chi _2\right]\left[\begin{array}{cccc}\hfill 0& \hfill 0& \hfill m& \hfill m\\ \hfill 0& \hfill 0& \hfill m& \hfill m\\ \hfill m& \hfill m& \hfill M& \hfill 0\\ \hfill m& \hfill m& \hfill 0& \hfill M\end{array}\right]\left[\begin{array}{c}\nu _1\\ \nu _2\\ \chi _1\\ \chi _2\end{array}\right].$$
(19)
This mass matrix is manifestly rank two, so two of the mass eigenvalues will be zero regardless the values of $`m`$ and $`M`$. This matrix can be diagonalized as

$$O^\mathrm{T}\left[\begin{array}{cccc}\hfill 0& \hfill 0& \hfill m& \hfill m\\ \hfill 0& \hfill 0& \hfill m& \hfill m\\ \hfill m& \hfill m& \hfill M& \hfill 0\\ \hfill m& \hfill m& \hfill 0& \hfill M\end{array}\right]O=\left[\begin{array}{cccc}0& 0& 0& 0\\ 0& 0& 0& 0\\ 0& 0& m/sc& 0\\ 0& 0& 0& m/sc\end{array}\right],$$
(20)
in which

$$O=\left[\begin{array}{cccc}\hfill c^2& \hfill s^2& \hfill sc& \hfill sc\\ \hfill s^2& \hfill c^2& \hfill sc& \hfill sc\\ \hfill sc& \hfill sc& \hfill c^2& \hfill s^2\\ \hfill sc& \hfill sc& \hfill s^2& \hfill c^2\end{array}\right],$$
(21)
and

$$s=\mathrm{sin}\theta ,c=\mathrm{cos}\theta ,t=\mathrm{tan}\theta ,\mathrm{tan}2\theta =\frac{2m}{M}.$$
(22)
The non-zero mass eigenvalues are

$$m_{\mathrm{heavy}}=\pm \frac{m}{sc}=\pm \sqrt{M^2+4m^2}.$$
(23)
Note that the orthogonal matrix which diagonalizes our mass matrix is not unique since we are free to rotate the zero mass eigenstates into each other. However, the above form is the one that reduces to the unit matrix in the limit $`m/M0`$ and facilitates the extraction of the suppression factor.

The mass eigenstates and the generation eigenstates are related through

$$\left[\begin{array}{c}\nu _1\\ \nu _2\\ \chi _1\\ \chi _2\end{array}\right]=O\left[\begin{array}{c}n_1\\ n_2\\ N_1\\ N_2\end{array}\right]=\left[\begin{array}{c}\hfill c^2n_1s^2n_2+scN_1+scN_2\\ \hfill s^2n_1+c^2n_2+scN_1+scN_2\\ \hfill scn_1scn_2+c^2N_1s^2N_2\\ \hfill scn_1scn_2s^2N_1+c^2N_2\end{array}\right].$$
(24)
The suppression factors for the $`Z\nu _1\nu _1`$ and $`Z\nu _2\nu _2`$ vertices can be read off from this expression:

$$1\epsilon =\mathrm{cos}^4\theta 12\theta ^21\frac{2m^2}{M^2}.$$
(25)
Thus, in contrast with the one generation case, the mass of the heavy states, $`\sqrt{M^2+4m^2}`$, and the suppression factor, $`\epsilon 2m^2/M^2`$, can be adjusted independently since the two free parameters in this model, $`m`$ and $`M`$, are not constrained in any way by the light neutrino masses.

Of course, the light mass eigenstates are exactly massless in this model so it is unrealistic as it stands. However, this can be remedied by adding a small perturbation to the mass matrix. For instance, consider the following perturbation:

$$\left[\begin{array}{cccc}0& 0& (1+\delta )m& (1+\delta )m\\ 0& 0& (1+\delta )m& (1+\delta )m\\ (1+\delta )m& (1+\delta )m& (1+\delta )M& 0\\ (1+\delta )m& (1+\delta )m& 0& (1+\delta )M\end{array}\right],$$
(26)
where $`\delta 1`$. This increases the rank of the matrix by one and renders one of the massless states massive. Partial diagonalization can be obtained as

$$O^\mathrm{T}O^\mathrm{T}\left[\begin{array}{cccc}0& 0& (1+\delta )m& (1+\delta )m\\ 0& 0& (1+\delta )m& (1+\delta )m\\ (1+\delta )m& (1+\delta )m& (1+\delta )M& 0\\ (1+\delta )m& (1+\delta )m& 0& (1+\delta )M\end{array}\right]OO^{}=\left[\begin{array}{cccc}0& 0& 0& 0\\ 0& & & \\ 0& & & \\ 0& & & \end{array}\right],$$
(27)
where $`O`$ is the matrix given in Eq. (21), and $`O^{}`$ is the matrix which mixes the first two columns maximally,

$$O^{}=\left[\begin{array}{cccc}\frac{1}{\sqrt{2}}& \frac{1}{\sqrt{2}}& 0& 0\\ \frac{1}{\sqrt{2}}& \frac{1}{\sqrt{2}}& 0& 0\\ 0& 0& 1& 0\\ 0& 0& 0& 1\end{array}\right].$$
(28)
The matrix $``$ has the form

$$=M\left[\begin{array}{ccc}4\delta \mu & 4\sqrt{2}\delta \mu r& 4\sqrt{2}\delta \mu r\\ 4\sqrt{2}\delta \mu r& \sqrt{1+4r^2}+\delta (1+2\mu )& 2\delta \mu \\ 4\sqrt{2}\delta \mu r& 2\delta \mu & \sqrt{1+4r^2}+\delta (1+2\mu )\end{array}\right],$$
(29)
where

$$r\frac{m}{M},\mu \frac{r^2}{1+4r^2}.$$
(30)
Complete diagonalization of $``$ requires further mixings by angles of order $`\delta `$. In the limit $`\delta 1`$, they can be neglected.

As another example, consider the perturbation

$$\left[\begin{array}{cccc}0& 0& m& m\\ 0& \delta M& (1\delta )m& (1\delta )m\\ m& (1\delta )m& M& 0\\ m& (1\delta )m& 0& M\end{array}\right],$$
(31)
where we have allowed for a small ($`\delta 1`$) but non-zero Majorana mass for one of the active neutrinos. This matrix can be partially diagonalized as

$`O^{\prime \prime \mathrm{T}}O^\mathrm{T}\left[\begin{array}{cccc}0& 0& m& m\\ 0& \delta M& (1\delta )m& (1+\delta )m\\ m& (1\delta )m& M& 0\\ m& (1+\delta )m& 0& M\end{array}\right]OO^{\prime \prime }`$ (36)
$`=`$ $`M\left[\begin{array}{cccc}4\delta r^4& 0& 2\delta r^3& 2\delta r^3\\ 0& \delta \left(1+2r^24r^4\right)& 2\delta r^3& 2\delta r^3\\ 2\delta r^3& 2\delta r^3& \sqrt{1+4r^2}\delta \mu & \delta \mu \\ 2\delta r^3& 2\delta r^3& \delta \mu & \sqrt{1+4r^2}\delta \mu \end{array}\right]+𝒪(\delta r^5),`$ (41)
where $`O`$ is the matrix given in Eq. (21), $`O^{\prime \prime }`$ is the matrix which mixes the first two columns with angle $`\theta ^{}`$,

$$O^{\prime \prime }=\left[\begin{array}{cccc}\mathrm{cos}\theta ^{}& \mathrm{sin}\theta ^{}& 0& 0\\ \mathrm{sin}\theta ^{}& \mathrm{cos}\theta ^{}& 0& 0\\ 0& 0& 1& 0\\ 0& 0& 0& 1\end{array}\right],\mathrm{tan}2\theta ^{}2c^2s^2(c^2s^2)\frac{2m^2}{M^2},$$
(42)
and the parameters $`r`$ and $`\mu `$ are as defined in Eq. (30). Again, complete diagonalization requires further mixings by angles of order $`\delta `$ which can be neglected.

As these examples illustrate, minute perturbations to the mass matrix, Eq. (19), can generate a variety of masses and mixings for the light neutrinos. Note that the extra mixings by $`O^{}`$ or $`O^{\prime \prime }`$ in these models have no effect on the suppression parameter, Eq. (25), which remains common for $`\nu _1`$ and $`\nu _2`$. This is to be expected as long as the perturbations are small and the mass matrix retains an approximate generation universal form. However, that is not to say that the suppression is flavor universal since the flavor eigenstates can be some mixture of $`\nu _1`$ and $`\nu _2`$ due to the mixing among the charged leptons. In general,

$`\left[\begin{array}{c}\nu _e\\ \nu _\mu \end{array}\right]`$ $`=`$ $`𝒰\left[\begin{array}{c}\nu _1\\ \nu _2\end{array}\right]=\left[\begin{array}{cc}\hfill \mathrm{cos}\varphi & \hfill \mathrm{sin}\varphi \\ \hfill \mathrm{sin}\varphi & \hfill \mathrm{cos}\varphi \end{array}\right]\left[\begin{array}{c}\nu _1\\ \nu _2\end{array}\right]`$ (51)
$`=`$ $`\left[\begin{array}{c}(c^2\mathrm{cos}\varphi +s^2\mathrm{sin}\varphi )n_1(c^2\mathrm{sin}\varphi +s^2\mathrm{cos}\varphi )n_2+\mathrm{}\\ (c^2\mathrm{sin}\varphi s^2\mathrm{cos}\varphi )n_1+(c^2\mathrm{cos}\varphi s^2\mathrm{sin}\varphi )n_2+\mathrm{}\end{array}\right].`$ (54)
This mixing breaks the universality of the suppression factors. For the $`Z\nu _e\nu _e`$ and $`We\nu _e`$ vertices, the suppression factor is

$`1\epsilon _e`$ $`=`$ $`(c^2\mathrm{cos}\varphi +s^2\mathrm{sin}\varphi )^2+(s^2\mathrm{cos}\varphi +c^2\mathrm{sin}\varphi )^2`$ (55)
$`=`$ $`c^4+2s^2c^2\mathrm{sin}2\varphi +s^4`$ (56)
$``$ $`12\theta ^2(1\mathrm{sin}2\varphi )`$ (57)
$``$ $`1\epsilon (1\mathrm{sin}2\varphi ),`$ (58)
while that for the $`Z\nu _\mu \nu _\mu `$ and $`W\mu \nu _\mu `$ vertices is

$`1\epsilon _\mu `$ $`=`$ $`(c^2\mathrm{sin}\varphi s^2\mathrm{cos}\varphi )^2+(c^2\mathrm{cos}\varphi s^2\mathrm{sin}\varphi )^2`$ (59)
$`=`$ $`c^42s^2c^2\mathrm{sin}2\varphi +s^4`$ (60)
$``$ $`12\theta ^2(1+\mathrm{sin}2\varphi )`$ (61)
$``$ $`1\epsilon (1+\mathrm{sin}2\varphi ).`$ (62)
Note that the sum $`\epsilon _e+\epsilon _\mu `$ is independent of the mixing angle $`\varphi `$. We will see later that this breaking of flavor universality is important in satisfying the constraint from $`\mu e\gamma `$.

## IV Three Generation Case

Extension of our model to the three generation case is straightforward. The most general seesaw mass matrix for the three generation case is

$$\left[\begin{array}{cc}\mathrm{𝟎}_{3\times 3}& 𝐦_{3\times 3}\\ 𝐦_{3\times 3}^\mathrm{T}& 𝐌_{3\times 3}\end{array}\right],$$
(63)
where $`𝐦_{3\times 3}`$ is the Dirac mass matrix, and $`𝐌_{3\times 3}`$ is the symmetric Majorana mass matrix. Consider the following Ansatz:

$$\left[\nu _1\nu _2\nu _3\chi _1\chi _2\chi _3\right]\left[\begin{array}{cccccc}0& 0& 0& \alpha m& \beta m& \gamma m\\ 0& 0& 0& \alpha m& \beta m& \gamma m\\ 0& 0& 0& \alpha m& \beta m& \gamma m\\ \alpha m& \alpha m& \alpha m& \alpha M& 0& 0\\ \beta m& \beta m& \beta m& 0& \beta M& 0\\ \gamma m& \gamma m& \gamma m& 0& 0& \gamma M\end{array}\right]\left[\begin{array}{c}\nu _1\\ \nu _2\\ \nu _3\\ \chi _1\\ \chi _2\\ \chi _3\end{array}\right],$$
(64)
where $`\alpha +\beta +\gamma =0`$. This matrix is manifestly at most rank three so that at least three of the mass eigenvalues are zero. This matrix can be block diagonalized as

$$O^\mathrm{T}\left[\begin{array}{cccccc}0& 0& 0& \alpha m& \beta m& \gamma m\\ 0& 0& 0& \alpha m& \beta m& \gamma m\\ 0& 0& 0& \alpha m& \beta m& \gamma m\\ \alpha m& \alpha m& \alpha m& \alpha M& 0& 0\\ \beta m& \beta m& \beta m& 0& \beta M& 0\\ \gamma m& \gamma m& \gamma m& 0& 0& \gamma M\end{array}\right]O=\left[\begin{array}{cc}0& 0\\ 0& \end{array}\right],$$
(65)
where

$$O=\left[\begin{array}{cccccc}1\frac{2}{3}s^2& \frac{2}{3}s^2& \frac{2}{3}s^2& \frac{2}{3}sc& \frac{2}{3}sc& \frac{2}{3}sc\\ \frac{2}{3}s^2& 1\frac{2}{3}s^2& \frac{2}{3}s^2& \frac{2}{3}sc& \frac{2}{3}sc& \frac{2}{3}sc\\ \frac{2}{3}s^2& \frac{2}{3}s^2& 1\frac{2}{3}s^2& \frac{2}{3}sc& \frac{2}{3}sc& \frac{2}{3}sc\\ \frac{2}{3}sc& \frac{2}{3}sc& \frac{2}{3}sc& 1\frac{2}{3}s^2& \frac{2}{3}s^2& \frac{2}{3}s^2\\ \frac{2}{3}sc& \frac{2}{3}sc& \frac{2}{3}sc& \frac{2}{3}s^2& 1\frac{2}{3}s^2& \frac{2}{3}s^2\\ \frac{2}{3}sc& \frac{2}{3}sc& \frac{2}{3}sc& \frac{2}{3}s^2& \frac{2}{3}s^2& 1\frac{2}{3}s^2\end{array}\right],$$
(66)
$$=\left[\begin{array}{ccc}\alpha (M+2mt)& \gamma mt& \beta mt\\ \gamma mt& \beta (M+2mt)& \alpha mt\\ \beta mt& \alpha mt& \gamma (M+2mt)\end{array}\right],$$
(67)
and

$$s=\mathrm{sin}\theta ,c=\mathrm{cos}\theta ,t=\mathrm{tan}\theta ,\mathrm{tan}2\theta =\frac{3m}{M}.$$
(68)
Note the difference in the definition of $`\mathrm{tan}2\theta `$ from the two and one generation cases, Eqs. (10) and (22). Again, we have the freedom to rotate the three zero mass eigenstates into each other so the above form for $`O`$ is not unique. However, as before, this is the choice which reduces to the unit matrix in the limit $`m/M0`$ and facilitates the extraction of the suppression parameter. Complete diagonalization requires an additional multiplication by a matrix of the form

$$O^{}=\left[\begin{array}{cc}I& 0\\ 0& 𝒜\end{array}\right],$$
(69)
so that

$$O_{}^{}{}_{}{}^{\mathrm{T}}\left[\begin{array}{cc}0& 0\\ 0& \end{array}\right]O^{}=\left[\begin{array}{cc}0& 0\\ 0& 𝒜^\mathrm{T}𝒜\end{array}\right]=\left[\begin{array}{cc}0& 0\\ 0& 𝒟\end{array}\right],$$
(70)
where

$$𝒟=\left[\begin{array}{ccc}M_1& 0& 0\\ 0& M_2& 0\\ 0& 0& M_3\end{array}\right].$$
(71)
The exact expression for $`𝒜`$ and the eigenvalues $`M_i`$ ($`i=1,2,3`$) will depend on the values of $`(\alpha ,\beta ,\gamma )`$. However, note that

$$𝒱OO^{}=\left[\begin{array}{cc}𝒪_{11}& 𝒪_{12}\\ 𝒪_{21}& 𝒪_{22}\end{array}\right]\left[\begin{array}{cc}I& 0\\ 0& 𝒜\end{array}\right]=\left[\begin{array}{cc}𝒪_{11}& 𝒪_{12}𝒜\\ 𝒪_{21}& 𝒪_{22}𝒜\end{array}\right],$$
(72)
so we see that $`𝒪_{11}`$, the upper left $`3\times 3`$ block of $`𝒱=OO^{}`$, is not affected by what the actual form of $`𝒜`$ is. Therefore, we can read off the common suppression factor for the $`Z\nu _1\nu _1`$, $`Z\nu _2\nu _2`$, and $`Z\nu _3\nu _3`$ vertices from our expression for $`O`$ and we find

$$1\epsilon =\left(1\frac{2}{3}\mathrm{sin}^2\theta \right)^21\frac{4}{3}\theta ^21\frac{3m^2}{M^2}.$$
(73)
Again, this can be adjusted independently of the heavy mass eigenvalues. Masses for the light eigenstates can be generated via minute perturbations to our mass matrix as in the two generation case, but we omit the details. Mixing among the charged leptons will cause $`\nu _1`$, $`\nu _2`$, and $`\nu _3`$ to mix:

$`\left[\begin{array}{c}\nu _e\\ \nu _\mu \\ \nu _\tau \end{array}\right]=𝒰\left[\begin{array}{c}\nu _1\\ \nu _2\\ \nu _3\end{array}\right],`$ (80)
where $`𝒰`$ is the unitary mixing matrix. This mixing breaks the flavor universality of the suppression parameters. We find that the suppression parameter for flavor $`f=e,\mu ,\tau `$ is given by

$`\epsilon _f`$ $`=`$ $`\epsilon \left|𝒰_{f1}+𝒰_{f2}+𝒰_{f3}\right|^2`$ (81)
$`=`$ $`\epsilon \left[1+2\mathrm{}\left(𝒰_{f1}^{}𝒰_{f2}+𝒰_{f2}^{}𝒰_{f3}+𝒰_{f3}^{}𝒰_{f1}\right)\right].`$ (82)
Note that as in the two generation case, the sum of the suppression parameters is independent of the mixing,

$$\epsilon _e+\epsilon _\mu +\epsilon _\tau =3\epsilon =\frac{9m^2}{M^2}.$$
(83)
The mixing of the heavy states with the light states is given by $`𝒪_{12}𝒜`$, so we need the explicit form of $`𝒜`$. Let us work out a few examples:

1. $`(\alpha ,\beta ,\gamma )=(0,1,1)`$.

$$=\left[\begin{array}{ccc}0& mt& mt\\ mt& (M+2mt)& 0\\ mt& 0& (M+2mt)\end{array}\right],$$
(84)
$$𝒜=\left[\begin{array}{ccc}c_\varphi ^2s_\varphi ^2& \sqrt{2}s_\varphi c_\varphi & i\sqrt{2}s_\varphi c_\varphi \\ \sqrt{2}s_\varphi c_\varphi & c_\varphi ^2& is_\varphi ^2\\ \sqrt{2}s_\varphi c_\varphi & s_\varphi ^2& ic_\varphi ^2\end{array}\right],$$
(85)
where

$$c_\varphi =\mathrm{cos}\varphi ,s_\varphi =\mathrm{sin}\varphi ,\mathrm{tan}2\varphi \frac{2\sqrt{2}t^2}{3+t^2},$$
(86)
and

$$𝒜^\mathrm{T}𝒜=\sqrt{M^2+6m^2}\left[\begin{array}{ccc}0& 0& 0\\ 0& 1& 0\\ 0& 0& 1\end{array}\right],$$
(87)
$`𝒪_{12}𝒜`$ $`=`$ $`{\displaystyle \frac{2}{3}}sc\left[\begin{array}{ccc}(\mathrm{cos}2\varphi \sqrt{2}\mathrm{sin}2\varphi )& (\mathrm{cos}2\varphi +\frac{1}{\sqrt{2}}\mathrm{sin}2\varphi )& i(\mathrm{cos}2\varphi +\frac{1}{\sqrt{2}}\mathrm{sin}2\varphi )\\ (\mathrm{cos}2\varphi \sqrt{2}\mathrm{sin}2\varphi )& (\mathrm{cos}2\varphi +\frac{1}{\sqrt{2}}\mathrm{sin}2\varphi )& i(\mathrm{cos}2\varphi +\frac{1}{\sqrt{2}}\mathrm{sin}2\varphi )\\ (\mathrm{cos}2\varphi \sqrt{2}\mathrm{sin}2\varphi )& (\mathrm{cos}2\varphi +\frac{1}{\sqrt{2}}\mathrm{sin}2\varphi )& i(\mathrm{cos}2\varphi +\frac{1}{\sqrt{2}}\mathrm{sin}2\varphi )\end{array}\right]`$ (91)
$`=`$ $`{\displaystyle \frac{m}{\sqrt{M^2+6m^2}}}\left[\begin{array}{ccc}\mathrm{cos}2\theta & 1& i\\ \mathrm{cos}2\theta & 1& i\\ \mathrm{cos}2\theta & 1& i\end{array}\right].`$ (95)
This model includes a massless sterile neutrino while the massive states are degenerate.
2. $`(\alpha ,\beta ,\gamma )=(1,1,2)`$.

$$=\left[\begin{array}{ccc}(M+2mt)& 2mt& mt\\ 2mt& (M+2mt)& mt\\ mt& mt& 2(M+2mt)\end{array}\right],$$
(96)
$$𝒜=\left[\begin{array}{ccc}\frac{1}{\sqrt{2}}& \frac{1}{\sqrt{2}}c_\phi & \frac{i}{\sqrt{2}}s_\phi \\ \frac{1}{\sqrt{2}}& \frac{1}{\sqrt{2}}c_\phi & \frac{i}{\sqrt{2}}s_\phi \\ 0& s_\phi & ic_\phi \end{array}\right],$$
(97)
$$𝒜^\mathrm{T}𝒜=\left[\begin{array}{ccc}M& 0& 0\\ 0& M_{}& 0\\ 0& 0& M_+\end{array}\right],$$
(98)
where

$$c_\phi =\mathrm{cos}\phi ,s_\phi =\mathrm{sin}\phi ,\mathrm{tan}2\phi \frac{4\sqrt{2}t^2}{9+7t^2},M_\pm \frac{3\sqrt{M^2+8m^2}\pm M}{2},$$
(99)
and

$`𝒪_{12}𝒜`$ $`=`$ $`{\displaystyle \frac{2}{3}}sc\left[\begin{array}{ccc}0& (\sqrt{2}c_\phi s_\phi )& i(\sqrt{2}s_\phi +c_\phi )\\ 0& (\sqrt{2}c_\phi s_\phi )& i(\sqrt{2}s_\phi +c_\phi )\\ 0& (\sqrt{2}c_\phi s_\phi )& i(\sqrt{2}s_\phi +c_\phi )\end{array}\right]`$ (103)
$`=`$ $`{\displaystyle \frac{\sqrt{6}m}{\sqrt{M_+^2M_{}^2}}}\left[\begin{array}{ccc}0& \sqrt{M/M_{}}& i\sqrt{M/M_+}\\ 0& \sqrt{M/M_{}}& i\sqrt{M/M_+}\\ 0& \sqrt{M/M_{}}& i\sqrt{M/M_+}\end{array}\right].`$ (107)
Note that the three mass eigenvalues are different in this case and that the heavy state with mass $`M`$ decouples from the light states.
3. $`(\alpha ,\beta ,\gamma )=(1,\omega ,\omega ^2)`$, $`\omega ^3=1`$.

$``$ $`=`$ $`\left[\begin{array}{ccc}(M+2mt)& \omega ^2mt& \omega mt\\ \omega ^2mt& \omega (M+2mt)& mt\\ \omega mt& mt& \omega ^2(M+2mt)\end{array}\right]`$ (111)
$`=`$ $`\left[\begin{array}{ccc}1& 0& 0\\ 0& \omega ^2& 0\\ 0& 0& \omega \end{array}\right]\left[\begin{array}{ccc}(M+2mt)& mt& mt\\ mt& (M+2mt)& mt\\ mt& mt& (M+2mt)\end{array}\right]\left[\begin{array}{ccc}1& 0& 0\\ 0& \omega ^2& 0\\ 0& 0& \omega \end{array}\right],`$ (121)
$$𝒜=\left[\begin{array}{ccc}1& 0& 0\\ 0& \omega & 0\\ 0& 0& \omega ^2\end{array}\right]\left[\begin{array}{ccc}\frac{1}{\sqrt{3}}& \frac{1}{\sqrt{6}}& \frac{1}{\sqrt{2}}\\ \frac{1}{\sqrt{3}}& \frac{1}{\sqrt{6}}& \frac{1}{\sqrt{2}}\\ \frac{1}{\sqrt{3}}& \frac{2}{\sqrt{6}}& 0\end{array}\right],$$
(122)
$$𝒜^\mathrm{T}𝒜=\left[\begin{array}{ccc}M& 0& 0\\ 0& \sqrt{M^2+9m^2}& 0\\ 0& 0& \sqrt{M^2+9m^2}\end{array}\right],$$
(123)
$$𝒪_{12}𝒜=\sqrt{\frac{3}{2}}\frac{m\omega ^2}{\sqrt{M^2+9m^2}}\left[\begin{array}{ccc}0& 1& i\\ 0& 1& i\\ 0& 1& i\end{array}\right].$$
(124)
In this case, the heavy state of mass $`M`$ decouples from the light states while the other two states have degenerate mass.

As these examples illustrate, there is a wide range of possibilities for the masses and mixings of the heavy states.

## V The Dirac Masses and Quark–Lepton Unification

We next consider the consequences of setting the Majorana masses to zero in our model. Our mass matrix will be

$$\left[\begin{array}{cccccc}0& 0& 0& \alpha m& \beta m& \gamma m\\ 0& 0& 0& \alpha m& \beta m& \gamma m\\ 0& 0& 0& \alpha m& \beta m& \gamma m\\ \alpha m& \alpha m& \alpha m& 0& 0& 0\\ \beta m& \beta m& \beta m& 0& 0& 0\\ \gamma m& \gamma m& \gamma m& 0& 0& 0\end{array}\right]$$
(125)
which is manifestly rank 2. The non–zero eigenvalues are

$$\pm m\sqrt{|\alpha |^2+|\beta |^2+|\gamma |^2}.$$
(126)
Therefore, this mass matrix leads to one massive, and two massless Dirac fermions.

If we assume that the up-type quarks have the same Dirac mass texture as the neutrinos, which would be the case in the Pati-Salam Pati:uk  model, we obtain one massive quark which can be identified with the $`t`$, and two massless quarks which can be identified with the $`u`$ and the $`c`$. Recall that a similar mechanism is at work in the ‘democratic’ democratic ; democratic2  quark mass texture,

$$\left[\overline{q}_1\overline{q}_2\overline{q}_3\right]\left[\begin{array}{ccc}m& m& m\\ m& m& m\\ m& m& m\end{array}\right]\left[\begin{array}{c}q_1\\ q_2\\ q_3\end{array}\right],$$
(127)
which is manifestly rank 1, and has eigenvalues $`0`$, $`0`$, and $`3m`$, which are identified as the $`u`$, $`c`$ and $`t`$ quark masses. Thus, our model provides an alternative to the ‘democratic’ model to explain why the $`t`$ quark mass is so large compared to the other quarks.

For this idea to work, it must be the case that

$$m100\mathrm{GeV},$$
(128)
to predict the correct $`t`$ mass. Since $`\epsilon 0.003`$ LOTW1  requires

$$\frac{m}{M}0.03,$$
(129)
we can predict

$$M3\mathrm{TeV}.$$
(130)
This places the heavy states within range of observation at the LHC.

## VI Constraint from $`\mu 𝐞\gamma `$

The flavor changing process $`\mu e\gamma `$ places a strong constraint on our model. The current limit on the decay branching fraction is from the MEGA collaboration mega

$$B(\mu e\gamma )=\frac{\mathrm{\Gamma }(\mu e\gamma )}{\mathrm{\Gamma }(\mu e\nu \overline{\nu })}<1.2\times 10^{11},\text{(90\% confidence level)}.$$
(131)
The theoretical value of this branching fraction is given by mutoegamma

$`B(\mu e\gamma )`$ $`=`$ $`{\displaystyle \frac{3\alpha }{32\pi }}\left|{\displaystyle \underset{i,j=1}{\overset{3}{}}}𝒰_{ei}^{}𝒰_{\mu j}T_{ij}\right|^2,`$ (132)
$`T_{ij}`$ $`=`$ $`{\displaystyle \underset{k=1}{\overset{6}{}}}𝒱_{ik}^{}𝒱_{jk}F(m_k^2/m_W^2),`$ (133)
where $`𝒰`$ is the mass diagonalization matrix for the charged leptons, Eq. (80), and $`𝒱=OO^{}`$ is the corresponding matrix for the neutrinos, Eq. (72). $`m_k`$ is the mass of the $`k`$–th neutrino mass eigenstate, and the function $`F(x)`$ is given by

$$F(x)=2(x+2)I^{(3)}(x)2(2x1)I^{(2)}(x)+2xI^{(1)}(x)+1,$$
(134)
in which

$$I^{(n)}(x)=_0^1𝑑z\frac{z^n}{z+(1z)x}.$$
(135)
$`F(x)`$ is plotted in Figure 1. Using the unitarity of $`𝒱`$ we can express $`T_{ij}`$ as a sum over the heavy states only,

$$T_{ij}=\underset{k=\mathrm{heavy}}{}𝒱_{ik}^{}𝒱_{jk}\left[F(m_k^2/m_W^2)F(0)\right],$$
(136)
where the masses of the light states are all set to zero. If we assume $`m_{\mathrm{heavy}}O(\text{TeV})`$, it is evident from Figure 1 that

$$F(m_{\mathrm{heavy}}^2/m_W^2)F(\mathrm{})=\frac{4}{3}.$$
(137)
Then,

$$T_{ij}2\underset{k=\mathrm{heavy}}{}𝒱_{ik}^{}𝒱_{jk}.$$
(138)
For the sake of simplicity, let us first consider the two generation case we considered in section III. From Eq. (21), we find

$$T_{ij}=4s^2c^2\left[\begin{array}{cc}1& 1\\ 1& 1\end{array}\right]2\epsilon \left[\begin{array}{cc}1& 1\\ 1& 1\end{array}\right].$$
(139)
Then, taking $`𝒰`$ to be the mixing matrix in Eq. (54), we obtain

$$B(\mu e\gamma )=\frac{3\alpha }{32\pi }\left|2\epsilon \mathrm{cos}2\varphi \right|^2=\frac{3\alpha }{8\pi }\epsilon ^2\mathrm{cos}^22\varphi =\frac{3\alpha }{8\pi }\epsilon _e\epsilon _\mu .$$
(140)
Assuming $`\epsilon =0.003`$, we find

$$\frac{3\alpha }{8\pi }\epsilon ^22\times 10^8,$$
(141)
which is more than three orders of magnitude above the MEGA limit, Eq. (131). Note that had we assumed the heavy states to be as light as the $`Z`$, then this prefactor would have been suppressed by an order of magnitude as can be gleaned from Figure 2. But even then, it would have been two orders of magnitude above the experimental limit. Thus, in order to suppress this process we need

$$\mathrm{cos}2\varphi 0\varphi 45^{}.$$
(142)
This mixing breaks the flavor universality of the suppression parameters maximally.

For the three generation model, the overall normalization of $`T_{ij}`$ depends on the number of heavy states that contribute to the sum. For the three cases considered earlier, we find

$$T_{ij}=2\eta \epsilon \left[\begin{array}{ccc}1& 1& 1\\ 1& 1& 1\\ 1& 1& 1\end{array}\right],$$
(143)
with

$$\eta =\{\begin{array}{cc}2/3,\hfill & (\alpha ,\beta ,\gamma )=(0,1,1),\hfill \\ 1,\hfill & (\alpha ,\beta ,\gamma )=(1,1,2),(1,\omega ,\omega ^2).\hfill \end{array}$$
(144)
Note that in the $`(\alpha ,\beta ,\gamma )=(0,1,1)`$ case, there are only two massive states that contribute to $`\eta `$. Thus,

$$\underset{i,j=1}{\overset{3}{}}𝒰_{ei}^{}𝒰_{\mu j}T_{ij}=2\eta \epsilon \left[𝒰_{e1}+𝒰_{e2}+𝒰_{e3}\right]^{}\left[𝒰_{\mu 1}+𝒰_{\mu 2}+𝒰_{\mu 3}\right],$$
(145)
and

$`B(\mu e\gamma )={\displaystyle \frac{3\alpha }{8\pi }}\eta ^2\epsilon ^2\left|𝒰_{e1}+𝒰_{e2}+𝒰_{e3}\right|^2\left|𝒰_{\mu 1}+𝒰_{\mu 2}+𝒰_{\mu 3}\right|^2={\displaystyle \frac{3\alpha }{8\pi }}\eta ^2\epsilon _e\epsilon _\mu .`$ (146)
The only difference from the two generation case is the factor $`\eta ^2`$. From this expression it is evident that to suppress $`B(\mu e\gamma )`$ the mixing matrix must be chosen to suppress either $`\epsilon _e`$ or $`\epsilon _\mu `$. That is, we must identify one of the flavor eigenstates, $`\nu _e`$ or $`\nu _\mu `$, as a linear combination of $`\nu _1`$, $`\nu _2`$, and $`\nu _3`$ which almost completely decouples from the heavy states.

This result implies that in order to satisfy the experimental limit on $`\mu e\gamma `$, the mixing angles in $`𝒰`$ must be generically large. These mixing angles should not be confused with the mixing angles in the MNS matrix Maki:mu  which are the angles measured in neutrino oscillation experiments SK ; SNO ; K2K . The MNS matrix in our notation is given by

$$V_{\mathrm{MNS}}=𝒰𝒱_{\mathrm{upper}\mathrm{left}},$$
(147)
where $`𝒱_{\mathrm{upper}\mathrm{left}}`$ is the upper left $`3\times 3`$ block of $`𝒱`$. (Note that $`V_{\mathrm{MNS}}`$ is not unitary since $`𝒱_{\mathrm{upper}\mathrm{left}}`$ is but a portion of an unitary matrix $`𝒱`$.) Therefore, in order to make a connection to the experimental data one must take into account the mixings among the light neutrinos in $`𝒱`$ also. In the current model, as long as the light neutrinos are exactly massless and degenerate, the MNS matrix is ill defined. The perturbations which break the degeneracy and allow non-zero masses must be chosen judiciously to obtain the correct $`𝒱`$ which would lead to the correct MNS matrix.

## VII Conclusions and Discussion

In this paper, we have shown explicit examples which demonstrate how large mixings between light active neutrinos and heavy sterile states can be realized within the seesaw framework, i.e. with the same number of active and sterile neutrinos, by a judicious choice of the mass texture. This required the presence of more than one generation so that there exist sufficient degrees of freedom in the mass texture to permit independent tuning of the mixing angles and mass eigenvalues Chang:1994hz . Indeed, even though the mass textures we considered, Eqs. (19) and (64), were constrained to a generation universal form, there were still enough degrees of freedom left to accomplish this objective.

Due to the independence of the mass eigenvalues and the mixing angles in these models, the only experimental constraint on the masses of the heavy states is that they must be heavier than the $`Z`$ LOTW1 . However, if we invoke quark–lepton universality and require the up–type quarks to share the same Dirac mass texture as the neutrinos, then we find that:

1. it provides a non–democratic explanation as to why the $`u`$ and $`c`$ quarks are almost massless compared to the $`t`$ quark,
2. the Dirac mass scale must be about 100 GeV to produce the correct $`t`$ mass, and
3. the heavy sterile states must have mass of a few TeV to produce the correct suppression parameter $`\epsilon `$.

This last observation stands in stark contrast to Grand Unified Theory (GUT) scenarios in which the masses of the sterile states are at the GUT scale, far exceeding experimental reach. If the heavy states are indeed in the few TeV range, they could be observed at the LHC. Of course, this estimation is from one particular model, and we cannot discount the possibility that the mass is actually much lower and could be observed at the Tevatron Run II at Fermilab.

Enforcing generation universality among the light neutrinos necessarily reduces the rank of our mass matrix leading to zero mass eigenvalues for the light states. To obtain non-zero masses, one must raise the rank by breaking generation universality with small perturbations. The same is true of the quark sector also since neither the $`u`$ nor the $`c`$ is massless. It is interesting to ask whether the same universality breaking perturbation could generate the correct mass ratios and mixings for both the neutrino and up–quark sectors.

Our model comfortably accomodates the introduction of down quark-charged lepton mass unification in addition to the up quark-neutrino unification utilized above. The constraints from $`\mu e\gamma `$ mandate large mixing angles among the charged leptons in our model. In models with down quark-charged lepton unification (e.g., the Pati-Salam model) this implies large mixing angles in the down quark mass matrix. For the CKM matrix to have the small mixings that are observed, the large down quark mixings must be cancelled by large up quark mixings. The presence of large mixings in the neutrino Dirac mass matrix then follows from up quark-neutrino unification and is consistent with the texture of Eq. (125). Models of quark-lepton unification which generate the large mixing angles of the MNS matrix using large angles in the charged lepton mass matrix have been explored extensively largeangle . Such models generically relate the MNS and CKM matrices. In the present context, however, explicit prediction of the CKM and MNS matrices would require a detailed delineation of the small perturbations of the mass matrices, which is beyond the scope of this paper.

Due to the typically large mixings between the light and heavy states in our model, the process $`\mu e\gamma `$ provides a strong constraint. We have found that the product $`\epsilon _e\epsilon _\mu `$ must be strongly suppressed. Since either $`\epsilon _e`$ or $`\epsilon _\mu `$ must remain sizable to account for the discrepancy between the $`Z`$-pole and NuTeV data, lepton flavor universality among the suppression parameters $`\epsilon _{\mathrm{}}`$, ($`\mathrm{}=e,\mu ,\tau `$), must be violated. Lepton universality is constrained experimentally at the $`1\%`$ level by $`W`$ decays pich ,

$`\left|{\displaystyle \frac{g_\mu }{g_e}}\right|`$ $`=`$ $`1.000\pm 0.011`$
$`\left|{\displaystyle \frac{g_\tau }{g_\mu }}\right|`$ $`=`$ $`1.026\pm 0.014`$
$`\left|{\displaystyle \frac{g_\tau }{g_e}}\right|`$ $`=`$ $`1.026\pm 0.014,`$
and at the $`0.2\%`$ level by $`\tau `$ decays pich ,

$`\left|{\displaystyle \frac{g_\mu }{g_e}}\right|`$ $`=`$ $`1.0006\pm 0.0021`$
$`\left|{\displaystyle \frac{g_\tau }{g_\mu }}\right|`$ $`=`$ $`0.9995\pm 0.0023`$
$`\left|{\displaystyle \frac{g_\tau }{g_e}}\right|`$ $`=`$ $`1.0001\pm 0.0023.`$
Whether lepton universality and $`\mu e\gamma `$ constraints can be satisfied simultaneously within our model while maintaining the fit to the $`Z`$-pole and NuTeV data will be addressed in a subsequent paper nextone .

The model analyzed in this paper should be contrasted with the recent analysis of neutrino masses and mixings by Glashow Glashow:2003wt . In his work, he assumes, as we do, that neutrino masses arise from a seesaw mass texture involving three active and three sterile neutrino states. He postulates a particular neutrino mass texture with the property that the mixings between light and heavy states are non-universal, leading to non-universal suppressions. In particular, the suppression of the neutrino weak gauge boson couplings occurs in a single light–mass eigenstate. The large mixing angles in the MNS matrix Maki:mu ; SK ; SNO ; K2K  spread the suppression among the three flavors and enhance the $`\mu e\gamma `$ rate, providing a limit on the size of the suppression parameter $`\epsilon `$. In fact, Glashow’s model is similar to ours, although he uses a different basis for the mass matrix. His analysis starts from a basis where the inter-generational mixing is minimal. In our analysis, we have taken the opposite approach. We start from a basis in which the inter-generational mixing is maximal and the suppression of light neutrino couplings to the weak gauge bosons are universal. To suppress the $`\mu e\gamma `$ rate, we must require large mixings among the charged lepton flavors leading to the violation of flavor universality of the suppression parameters. In both approaches, the violation of flavor universality is essential for obtaining agreement with current experimental data. It is amusing to note that lepton flavor universality violation is required for lepton flavor conservation.

Given the strong constraint on our model from the flavor changing process $`\mu e\gamma `$, one is lead to question what constraints will be provided by other flavor changing processes such as $`\mu `$$`e`$ conversion in nuclei. This is a particularly interesting process since the MECO (Muon to Electron COnversion) experiment at Brookhaven meco  proposes to improve upon the current limits on $`\mu `$$`e`$ conversion PDB:2002  by more than three orders of magnitude. The MEG (Mu-E-Gamma) experiment at PSI muegamma  also proposes to improve upon the MEGA limit on $`\mu e\gamma `$ by about two orders of magnitude. We will investigate the implications of the potential MECO and MEG limits in a future publication.

Finally, we propose that if heavy neutral particles such as those discussed in this paper are found to exist they be called “neutrissimos” instead of “heavy sterile neutrinos” or “heavy neutral leptons”, the terms commonly used in the literature. Those names are balky, and the first one is misleading, since the particles are neither completely “sterile” nor are they “-ino” compared to the neutrons.

## Acknowledgments

We would like to thank Sandor Benczik, Lay Nam Chang, Naoyuki Haba, Yasuhiro Okada, and Alexey Pronin for helpful discussions. We would also like to thank Massimiliano Di Ventra for coining the term ‘neutrissimo’. This research was supported in part by the U.S. Department of Energy, grants DE–FG05–92ER40709, Task A (T.T. and N.O.), and DE-FG02-84ER40153 (L.C.R.W.).


format: text

text:
# The Magnetar Fields

## Abstract

We discuss the nature of evolution of the magnetic field in magnetars.

There is growing evidence that soft gamma-ray repeaters (SGR) and anomalous X-ray pulsars (AXP) are isolated neutron stars with superstore magnetic fields ($`B>10^{14}`$ G), commonly known as magnetars. Recently, the detection of a 5 Kiev absorption feature from SGR 1806-20 has been identified as proton cyclotron resonance, offering a direct measurement of the magnetic field to be $`10^{15}`$ G (Ibrahim, Swank & Parke 2003). Decay of this strong magnetic field is supposed to power these magnetars. The strong fields affect the physical processes in the stellar interior and consequently influence the evolution of the magnetic field itself.

The ratio between the Landau level spacing and the Fermi energy for the electrons is,

$$\frac{E_F}{\mathrm{\Delta }E_{\mathrm{LL}}}=2.5\mu ^{1/3}\rho _{15}^{1/3}_{15}^1,(\rho =\rho _{15}10^{15}\mathrm{g}\mathrm{cm}^3\text{}=_{15}10^{15}\text{ G}).$$
(1)
As seen from Fig.1A, this ratio is smaller than unity for a large range of densities where the system goes into the quantum Hall regime. For neutron stars born with strong fields the electron fluid is already in this regime at birth. Then the outer regions of the star may not form a crystalline solid upon cooling, greatly affecting the transport properties.

As the crustal lattice yields under the pressure exerted by a $`10^{15}`$ G field, the currents need to be anchored in the deeper interior of the star, comprising of an n-p-e plasma. Assuming the plasma to be non-superfluid the evolution of the magnetic field would be governed by the induction equation (Goldreich & Reisenegger, 1992)

$$\frac{𝐁}{t}=c\times \left(\frac{𝐉}{\sigma _0}\right)+\times (𝐯\times 𝐁)\frac{m_p/\tau _{pn}m_e/\tau _{en}}{m_p/\tau _{pn}+m_e/\tau _{en}}\times \left(\frac{𝐉\times 𝐁}{n_ce}\right),$$
(2)
where $`\sigma _0`$ is the homogeneous electrical conductivity of the medium in absence of a magnetic field. Since, the magnetar fields are strongly quantizing the transport properties would be highly anisotropic as well as field-dependent (Yakovlev, 1984; Ghosh et.al, 2002). This renders the ohmic dissipation time-scale of the internal currents field dependent as shown in Fig.1B. However, we expect the field to decay faster for stronger fields and the dependence of $`\tau _{\mathrm{ohmic}}`$ should be more like the dashed curve in Fig.1B (obtained by using $`\sigma (B)=\sigma _0(1\beta (B/B_c)^\alpha )`$ where $`B_c`$ is the electron critical field) and contrary to that obtained using the electrical conductivity of either Yakovlev (1984) or Ghosh (2002).

Evidently, the terms corresponding to Hall drift and Ambipolar diffusion would be important for the magnetar field evolution. The anisotropy in the conductivity would give rise to strong Hall currents inducing rapid transfer of energy from dipolar component of the field to higher multi-poles. Since, currents of higher order multi-poles dissipate faster (Mitra, Konar & Bhattacharya 1999) the field would decay rapidly because of this. The details of this work can be found in Konar (2003).


text:
# Supersolid versus Phase Separation in Atomic Bose-Fermi Mixtures

## Abstract

We show that a two-dimensional atomic mixture of Bosons and Fermions cooled into their quantum degenerate states and subject to an optical lattice develops a supersolid phase characterized by the simultaneous presence of a non-trivial crystalline order and phase order. This transition is in competition with a phase separated ground state. We determine the phase diagram of the system and propose an experiment allowing for the observation of the supersolid phase.

Cooling atoms to the nK regime allows for the realization and study of new thermodynamic phase transitions and their associated phases, with an interesting synergy emerging between the fields of quantum atom optics and condensed matter physics. Recent trends are the study of the superfluid to Mott-insulator phase transition appearing in cold bosonic systems subject to an optical lattice Jaksch et al. (1998); Greiner et al. (2002) and the striving for the realization of a BCS-type condensate in a fermionic system Houbiers et al. (1997); Heiselberg et al. (2000). In this letter, we investigate the possibility to realize a non-trivial supersolid phase in a mixed boson-fermion system sympathetically cooled into their corresponding quantum degenerate states Truscott et al. (2001); Roati et al. (2002). We identify a promising system where this novel phase can be observed and determine the relevant phase diagram.

Supersolids simultaneously exhibit two types of order which usually appear in competition to each other — these are the diagonal long-range order (DLRO) associated with the periodic density modulation in a crystal and the off-diagonal long-range order (ODLRO) associated with the phase order in the condensate Penrose and Onsager (1956). Supersolids have been proposed to exist in the strongly interacting 4He system Andreev and Lifshitz (1967); Leggett (1970), where experimental results are still hotly debated Lengua and Goodkind (1990); Meisel (1992), and in various model systems describing interacting Bosons on a lattice and analyzed numerically Batrouni et al. (1995); frey et al. (1997); Góral et al. (2002). Here, we investigate the possibility to use a specifically tuned boson-fermion mixture to realize a supersolid phase in a controlled experiment.

The basic idea underlying our scheme is to share tasks between the fermions and the bosons: the fermions are tuned through a density wave instability establishing crystalline order (DLRO), while the condensate bosons provide the off-diagonal long range order (ODLRO). The interaction with the fermions imprints an additional density modulation also in the bosonic density field, hence resulting in a supersolid phase. In order to trigger a density wave instability in the fermions, we confine the mixed boson-fermion system to two dimensions and subject it to an optical lattice providing perfect Fermi surface nesting at half-filling Grüner (2000). Note, that the resulting crystalline order relevant for the DLRO component in the supersolid is not due to the density modulation enforced by the optical lattice but is the superstructure triggered by the density wave instability.

The supersolid transition triggered by the fermions competes with an instability towards phase separation in the boson system Mølmer (1998); Amoruso et al. (1998). Given the dimensionality and the lattice geometry of our system, the presence of van Hove singularities strongly enhances the tendency towards phase separation and produces new and interesting features in this transition: an arbitrary weak interaction between the bosons and the fermions is sufficient to drive the phase separation at low temperatures. However, proper tuning of the parameters allows to supersede this phase separation through the supersolid transition shown in Fig. 1. In the following, we first investigate the two instabilities towards phase separation and density wave formation and then analyze the supersolid phase within a mean-field scheme. We focus on the weak coupling limit between the bosons and the fermions, which excludes a demixing in a repulsive fermion-boson system along the lines discussed in Ref. Viverit et al. (2000).

The Hamiltonian for interacting bosons and fermions subject to an optical lattice takes the form $`H=H_\mathrm{B}+H_\mathrm{F}+H_{\mathrm{int}}`$ with ($`\alpha =\mathrm{F},\mathrm{B}`$)

$`H_\alpha `$ $`=`$ $`{\displaystyle 𝑑𝐱\psi _\alpha ^+\left(\frac{\mathrm{}^2}{2m_\alpha }\mathrm{}+V_\alpha (𝐱)\right)\psi _\alpha },`$ (1)
$`H_{\mathrm{int}}`$ $`=`$ $`{\displaystyle 𝑑𝐱\left(g_{\mathrm{FB}}\psi _\mathrm{B}^+\psi _\mathrm{B}\psi _\mathrm{F}^+\psi _\mathrm{F}+\frac{1}{2}g_\mathrm{B}\psi _\mathrm{B}^+\psi _\mathrm{B}^+\psi _\mathrm{B}\psi _\mathrm{B}\right)}.`$
Here, we assume a repulsive interaction $`g_\mathrm{B}=4\pi a_s\mathrm{}^2/m`$ between the bosons, with the scattering length $`a_s>0`$. The coupling $`g_{\mathrm{FB}}=2\pi a_{\mathrm{FB}}\mathrm{}^2/\mu `$ accounts for the interaction between the fermions and the bosons, with $`\mu `$ the relative mass and $`a_{\mathrm{FB}}`$ the boson-fermion scattering length. Furthermore, we restrict the analysis to spinless fermions; such a spinless fermionic atom gas is naturally achieved via spin polarization. The $`s`$-wave scattering length in the fermion system vanishes, while $`p`$-wave scattering is small in general and is neglected in the following analysis. The optical lattice with wave length $`\lambda `$ provides an $`a=\lambda /2`$-periodic potential for the bosons and fermions with $`V_{\mathrm{F},\mathrm{B}}(𝐱)=V_{\mathrm{F},\mathrm{B}}[\mathrm{sin}^2(\pi x/a)+\mathrm{sin}^2(\pi y/a)]`$. We expand the bosonic and fermionic field operators $`\psi _{\mathrm{B},\mathrm{F}}`$ in the Bloch wave functions $`v_{k,n}`$ and $`w_{k,n}`$ of the single particle problem in a periodic potential, (we restrict the analysis to the lowest Bloch band)

$$\psi _\mathrm{B}(𝐱)=\underset{𝐤K}{}b_𝐤w_𝐤(𝐱),\psi _\mathrm{F}(𝐱)=\underset{𝐤K}{}c_𝐤v_𝐤(𝐱).$$
(2)
Here, $`K`$ denotes the first Brillouin zone, while $`b_𝐤`$ and $`c_𝐤`$ are the bosonic and fermionic annihilation operators. For a strong optical lattice $`V_{\mathrm{F},\mathrm{B}}>E_{\mathrm{F},\mathrm{B}}^r=2\mathrm{}^2\pi ^2/\lambda ^2m_{\mathrm{F},\mathrm{B}}`$, the restriction to the lowest Bloch band is justified, and the Hamiltonian simplifies to Jaksch et al. (1998)

$`H`$ $`=`$ $`{\displaystyle \underset{𝐤K}{}}ϵ_\mathrm{B}(𝐤)b_𝐤^+b_𝐤+{\displaystyle \frac{U_\mathrm{B}}{2N}}{\displaystyle \underset{\{𝐤,𝐤^{},𝐪,𝐪^{}\}}{}}b_𝐤^+b_𝐤^{}b_𝐪^+b_𝐪^{}`$ (3)
$`+{\displaystyle \underset{𝐪K}{}}ϵ_\mathrm{F}(𝐪)c_𝐪^+c_𝐪+{\displaystyle \frac{U_{\mathrm{FB}}}{N}}{\displaystyle \underset{\{𝐤,𝐤^{},𝐪,𝐪^{}\}}{}}b_𝐤^+b_𝐤^{}c_𝐪^+c_𝐪^{},`$
with quantization volume $`V=Na^2`$ and $`N`$ the number of unit cells (below, $`n_{\mathrm{F},\mathrm{B}}`$ denote the number of particles per unit cell). The summation $`\{𝐤,𝐤^{},𝐪,𝐪^{}\}`$ is restricted to $`𝐤,𝐤^{},𝐪,𝐪^{}K`$ with the momentum conservation $`𝐤𝐤^{}+𝐪𝐪^{}=𝐊_m`$; the reciprocal lattice vector $`𝐊_m`$ accounts for Umklapp processes. Here, $`U_{\mathrm{FB}}=g_{\mathrm{FB}}𝑑𝐱|\stackrel{~}{w}|^2|\stackrel{~}{v}|^2`$ and $`U_\mathrm{B}=g_\mathrm{B}𝑑𝐱|\stackrel{~}{w}|^4`$, with $`\stackrel{~}{w}(𝐱)`$ and $`\stackrel{~}{v}(𝐱)`$ the Wannier functions associated with the Bloch band $`w_𝐤`$ and $`v_𝐤`$ Jaksch et al. (1998), while $`ϵ_{\mathrm{F},\mathrm{B}}(𝐤)`$ denote the energy dispersion of the fermions and bosons, respectively. For a strong optical lattice only nearest neighbor hopping survives and the dispersion relation takes the form

$$ϵ_\mathrm{F}(𝐪)=2J_\mathrm{F}\left[\mathrm{cos}\left(q_xa\right)+\mathrm{cos}\left(q_ya\right)\right]$$
(4)
with $`J_\mathrm{F}`$ the hopping energy. The Fermi surface at half-filling $`n_\mathrm{F}=1/2`$ is shown in Fig. 2 and exhibits perfect nesting for $`𝐤_{\mathrm{DW}}=(\pi /a,\pi /a)`$ and van Hove singularities at $`𝐤=(0,\pm \pi /a),(\pm \pi /a,0)`$. The bosonic energies are $`ϵ_\mathrm{B}(𝐪)=2J_\mathrm{B}\left[2\mathrm{cos}\left(q_xa\right)\mathrm{cos}\left(q_ya\right)\right]`$ with $`ϵ_\mathrm{B}(0)=0`$.

Integrating out the fermions provides an effective interaction for the bosons which depends on the temperature $`T`$ of the fermionic atom gas. Within linear response theory, the boson density operator $`n_\mathrm{B}(𝐪)`$ drives the fermionic density $`n_\mathrm{F}(𝐪)=U_{\mathrm{FB}}\chi (T,𝐪)n_\mathrm{B}(𝐪)`$ with $`\chi (T,𝐪)`$ the fermionic response function. A perturbed fermionic density in turn acts as a drive for the bosons, leading to an effective boson-boson interaction

$$H_{\mathrm{int}}=\frac{1}{2N}\underset{\{𝐤,𝐤^{},𝐪,𝐪^{}\}}{}\left[U_\mathrm{B}+U_{\mathrm{FB}}^2\chi (T,𝐪𝐪^{})\right]b_𝐤^+b_𝐤^{}b_𝐪^+b_𝐪^{}.$$

The fermionic response is given by the Lindhard function

$$\chi (T,𝐪)=_K\frac{d𝐤}{v_0}\frac{f[ϵ_\mathrm{F}(𝐤)]f[ϵ_\mathrm{F}(𝐤+𝐪)]}{ϵ_\mathrm{F}(𝐤)ϵ_\mathrm{F}(𝐤+𝐪)+i\eta }$$
(5)
with $`v_0=(2\pi /a)^2`$ the volume of the first Brillouin zone. The temperature $`T`$ enters via the Fermi distribution function $`f(ϵ)=1/[1+\mathrm{exp}(ϵ/T)]`$ ($`\mu _\mathrm{F}=0`$ at half filling); in our weak coupling analysis we are interested in temperatures well below the superfluid transition temperature $`T_{\mathrm{KT}}`$ of the bosons. Using the fermionic dispersion relation (4), the Lindhard function exhibits two logarithmic singularities. These singularities give rise to instabilities in the system towards two new ground states: the singularity at $`𝐪=0`$ induces an instability towards a phase separated state, while the singularity at $`𝐤_{\mathrm{DW}}`$ induces an instability towards density wave formation and provides a supersolid phase. In the following, we discuss these two instabilities in detail.

For a fermionic system with a regular density of states, the Lindhard function at $`𝐪=0`$ reduces to $`\chi (T0,0)=N(0)`$ at low temperatures, with $`N(0)`$ the fermionic density of states. However, fermions on a square lattice within a tight-binding scheme exhibit a logarithmic van Hove singularity in the density of states, $`N(ϵ)=N_0K[\sqrt{1ϵ^2/16J_\mathrm{F}^2}]N_0\mathrm{ln}|16J_\mathrm{F}/ϵ|`$ with $`N_0=1/(2\pi ^2J_\mathrm{F})`$ and $`K[k]`$ the complete elliptic integral of the first kind Gradshteyn and Ryshik (1980), see Fig. 2. The response at half-filling $`n_\mathrm{F}=1/2`$ then behaves as

$$\chi (T0,0)=𝑑ϵN(ϵ)_ϵf(ϵ)N_0\mathrm{ln}\frac{16c_1J_\mathrm{F}}{T}$$
(6)
with $`c_1=2\mathrm{exp}(C)/\pi 1.13`$.

The coupling between the bosons and the fermions induces an attraction between the bosons, and the effective long distance scattering parameter takes the form $`U_{\mathrm{eff}}=U_\mathrm{B}+U_{\mathrm{FB}}^2\chi (T,0)`$. The thermodynamic stability of a superfluid condensate at low temperatures requires a positive effective interaction $`U_{\mathrm{eff}}>0`$ and the condition $`U_{\mathrm{eff}}(T_{\mathrm{PS}})=0`$ defines the critical temperature $`T_{\mathrm{PS}}`$ for phase separation. Using Eqs. (6), we find for the critical temperature

$$T_{\mathrm{PS}}=16c_1J_\mathrm{F}\mathrm{exp}\left[1/\lambda _{\mathrm{FB}}\right]$$
(7)
with $`\lambda _{\mathrm{FB}}=U_{\mathrm{FB}}^2N_0/U_\mathrm{B}1`$ the ratio between the induced attraction and the intrinsic repulsion between the bosons. Below the critical temperature $`T_{\mathrm{PS}}`$, the effective interaction $`U_{\mathrm{eff}}`$ turns negative, providing a negative compressibility and rendering the Bose system unstable. The new ground state with fixed averaged densities $`n_\mathrm{B}`$ and $`n_\mathrm{F}`$ exhibits phase separation with areas of increased and decreased local densities coexisting, cf. Fig. 1.

Note, that this transition towards phase separation exhibits two major differences as compared to the phase separation discussed in Refs. Mølmer, 1998; Viverit et al., 2000. First, our phase separation appears as an instability, i.e., for arbitrary small coupling $`U_{\mathrm{FB}}`$ between the bosons and fermions. Second, the increase/decrease in the bosonic density drives the fermionic density away from half-filling, providing a regular $`\chi (T,0)`$ which stabilizes the system.

This phase separation is in competition with a second instability in the system triggered by the singularity in the Lindhard function at $`𝐤_{\mathrm{DW}}`$. Using (5) and the symmetry $`ϵ_\mathrm{F}(𝐪+𝐤_{\mathrm{DW}})=ϵ_\mathrm{F}(𝐪)`$, the Lindhard function becomes

$$\chi (T,𝐤_{\mathrm{DW}})=𝑑ϵN(ϵ)\frac{\mathrm{tanh}\left(ϵ/2T\right)}{2ϵ}\frac{N_0}{2}\left[\mathrm{ln}\frac{16c_1J_\mathrm{F}}{T}\right]^2.$$

The combination of van Hove singularities and perfect nesting produces the $`[\mathrm{ln}T]^2`$ singular behavior known to produce a $`T_c`$-enhancement in superconductivity Hirsch and Scalapino (1986). Within Bogoliubov theory, the bosonic quasi-particle spectrum becomes

$$E_\mathrm{B}(𝐪)=\sqrt{ϵ_\mathrm{B}^2(𝐪)+2n_\mathrm{B}ϵ_\mathrm{B}(𝐪)\left[U_\mathrm{B}+U_{\mathrm{FB}}^2\chi (T,𝐪)\right]}.$$
(8)
The induced attraction between the bosons reduces the energy of quasi-particles at $`𝐤_{\mathrm{DW}}`$ which vanishes ($`E_\mathrm{B}(𝐤_{\mathrm{DW}})=0`$) at the critical temperature

$$T_{\mathrm{DW}}=16c_1J_\mathrm{F}\mathrm{exp}\left[\sqrt{(2+t_\mathrm{B})/\lambda _{\mathrm{FB}}}\right]$$
(9)
with $`t_\mathrm{B}=8J_\mathrm{B}/n_\mathrm{B}U_\mathrm{B}`$ the ratio between the kinetic and the interaction energy of the bosons (weak coupling requires $`t_\mathrm{B}\lambda _{\mathrm{FB}}`$). Below this critical temperature, the boson mode $`b_{𝐤_{\mathrm{DW}}}`$ becomes macroscopically occupied and its interference with the condensate produces a bosonic density wave, see below. In this new phase, a (quasi-)condensate characterized by an off-diagonal (quasi-)long-range order with a finite superfluid stiffness coexists with a density wave providing diagonal long-range order, thus establishing a supersolid phase. Note, that decreasing the hopping $`t_\mathrm{B}`$ drives a superfluid to Mott-insulator transition for commensurate densities Jaksch et al. (1998) below $`t_\mathrm{B}<t_{\mathrm{SF}\mathrm{MI}}1/3`$, providing an additional competing phase at strong coupling $`t_\mathrm{B}<1`$.

Next, we study the supersolid phase within a mean field description. We introduce the mean fields $`b_0=\sqrt{n_0N}\mathrm{exp}(i\phi _0)`$ and $`b_{𝐤_{\mathrm{DW}}}=(\mathrm{\Delta }/2U_{\mathrm{FB}})\sqrt{N/n_0}\mathrm{exp}(i\phi )`$ and neglect thermal excitations of bosonic quasi-particles at $`TT_{\mathrm{KT}}`$, implying the constraint $`n_\mathrm{B}=n_0+\mathrm{\Delta }^2/(4n_0U_{\mathrm{FB}}^2)`$. The bosonic density takes the form (to be evaluated at lattice sites)

$$n_\mathrm{B}(x,y)=n_\mathrm{B}+\frac{\mathrm{\Delta }\mathrm{cos}\theta }{U_{\mathrm{FB}}}\left[\mathrm{cos}\frac{\pi x}{a}\mathrm{cos}\frac{\pi y}{a}\right]$$
(10)
with $`\theta =\phi _0\phi `$; this bosonic density wave appears as a result of the interference between the two condensates $`b_0`$ and $`b_{𝐤_{\mathrm{DW}}}`$. Neglecting terms independent on $`\mathrm{\Delta }`$, the Hamiltonian (3) per unit cell reduces to

$$\frac{H}{N}=2J_\mathrm{B}\frac{\mathrm{\Delta }^2}{n_\mathrm{B}U_{\mathrm{FB}}^2}+\frac{U_\mathrm{B}\mathrm{\Delta }^2\mathrm{cos}^2\theta }{2U_{\mathrm{FB}}^2}+\frac{H_\mathrm{F}}{N}+o(\mathrm{\Delta }^4).$$
(11)
The first and second terms describe the increase in the kinetic and interaction energies of the bosons, while $`H_\mathrm{F}`$ accounts for the nesting of fermions with $`𝐪K`$ and $`𝐪^{}=𝐪𝐤_{\mathrm{DW}}+𝐊_m`$ (the reciprocal lattice vector $`𝐊_m`$ ensures the constraint $`𝐪^{}K`$)

$$H_\mathrm{F}=\frac{1}{2}\underset{𝐪K}{}(\begin{array}{cc}c_𝐪^+,& c_𝐪^{}^+\end{array})(\begin{array}{cc}ϵ_\mathrm{F}(𝐪)& \mathrm{\Delta }\mathrm{cos}\theta \\ \mathrm{\Delta }\mathrm{cos}\theta & ϵ_\mathrm{F}(𝐪^{})\end{array})(\begin{array}{c}c_𝐪\\ c_𝐪^{}\end{array}).$$
(12)
Diagonalizing, we obtain the fermionic quasi-particle excitation spectrum $`\stackrel{~}{ϵ}_\mathrm{F}(𝐤,\mathrm{\Delta })=\pm [ϵ_\mathrm{F}^2(𝐤)+\mathrm{cos}^2\theta \mathrm{\Delta }^2]^{1/2}`$. Minimizing the thermodynamic potential $`\mathrm{\Omega }(T,\mathrm{\Delta },\theta )`$ provides the constraint $`\theta =s\pi `$, with $`s`$ an integer, and the self-consistency relation ($`_\mathrm{\Delta }\mathrm{\Omega }=0`$)

$$\frac{1}{\lambda _{\mathrm{FB}}}\left(2+t_\mathrm{B}\right)=\frac{1}{N_0}_K\frac{d𝐤}{v_0}\frac{\mathrm{tanh}\left[\stackrel{~}{ϵ}_\mathrm{F}(𝐤,\mathrm{\Delta })/2T\right]}{\stackrel{~}{ϵ}_\mathrm{F}(𝐤,\mathrm{\Delta })}.$$
(13)
Setting $`\mathrm{\Delta }=0`$, we reproduce the critical temperature (9). Using the density of states $`N_\mathrm{\Delta }(ϵ)=N(\sqrt{ϵ^2\mathrm{\Delta }^2})`$ $`|ϵ|/\sqrt{ϵ^2+\mathrm{\Delta }^2}`$, the gap at $`T=0`$ becomes

$$\mathrm{\Delta }(0)=32J_\mathrm{F}\mathrm{exp}\left[\sqrt{(2+t_\mathrm{B})/\lambda _{\mathrm{FB}}}\right]$$
(14)
and we obtain the standard BCS relation $`2\mathrm{\Delta }(0)/T_{\mathrm{DW}}=2\pi /e^C3.58`$. The mean field $`b_0=\sqrt{n_0N}\mathrm{exp}[i\phi _0]`$ breaks the continuous $`U(1)`$ symmetry of the system and describes the off-diagonal quasi-long-range order. The excitation spectrum exhibits a linear dispersion around $`q=0`$ and the quasi-long-range order is sufficient to provide a finite superfluid stiffness Leggett (1970). On the other hand, the mean field $`b_{𝐤_{\mathrm{DW}}}`$ characterizes a commensurate density wave. Its phase is locked to the condensate phase via the constraint $`\theta =s\pi `$ and the excitation spectrum around $`𝐪=𝐤_{\mathrm{DW}}`$ is gapped. The transition breaks the discrete translation symmetry and establishes diagonal long-range order; breaking a discrete symmetry, diagonal long-range order can exist even at finite temperatures.

The competition between the two instabilities at $`𝐪=0`$ and $`𝐤_{\mathrm{DW}}`$ provides the phase diagram as shown in Fig. 1. For $`T_{\mathrm{DW}}>T_{\mathrm{PS}}`$ the system undergoes a transition into a supersolid phase at $`T_{\mathrm{DW}}`$; the gap in the fermionic excitation spectrum then removes the instability towards phase separation luring at lower temperatures. In turn, for $`T_{\mathrm{PS}}>T_{\mathrm{DW}}`$ the instability towards phase separation wins over the density wave formation and drives the fermionic density away from half-filling; the nesting at $`𝐤_{\mathrm{DW}}`$ is quenched and the instability towards density wave formation disappears. The projection of the critical line $`T_{\mathrm{PS}}=T_{\mathrm{DW}}`$ onto the $`\lambda _{\mathrm{FB}}`$-$`1/t_\mathrm{B}`$-plane satisfies the relation $`1/t_\mathrm{B}=\lambda _{\mathrm{FB}}/(12\lambda _{\mathrm{FB}})`$; with decreasing coupling $`\lambda _{\mathrm{FB}}`$ we enter the supersolid phase, while increasing the hopping $`t_\mathrm{B}`$ drives the system towards phase separation.

Finally, we estimate the relevant experimental parameters for an atomic mixture of fermionic 40K and bosonic 87Rb with scattering lengths $`a_\mathrm{B}=5.77\mathrm{nm}`$ and $`a_{\mathrm{FB}}15\mathrm{nm}`$ Roati et al. (2002). The 2D setup is realized through application of an anisotropic 3D optical lattice ($`\lambda =830\mathrm{nm}`$ and $`V_\mathrm{F}/V_\mathrm{B}3/7`$), with $`V_\mathrm{F}^zV_\mathrm{F}`$ and $`V_\mathrm{B}^zV_\mathrm{B}`$ quenching the inter-plane hopping (we express $`V_{\mathrm{F},\mathrm{B}}`$ and $`V_{\mathrm{F},\mathrm{B}}^z`$ via the recoil energies $`E_{\mathrm{F},\mathrm{B}}^r=2\pi ^2\mathrm{}^2/\lambda ^2m_{\mathrm{F},\mathrm{B}}`$). The hopping amplitudes $`J_{\mathrm{F},\mathrm{B}}`$ derive from the 1D Mathieu equation,

$$J_{\mathrm{F},\mathrm{B}}=(4/\sqrt{\pi })E_{\mathrm{F},\mathrm{B}}^rV_{\mathrm{F},\mathrm{B}}^{3/4}\mathrm{exp}\left(2\sqrt{V_{\mathrm{F},\mathrm{B}}}\right),$$
(15)
while the interactions $`U_{\mathrm{FB}}`$ and $`U_\mathrm{B}`$ are given as Jaksch et al. (1998)

$$\frac{U_{\mathrm{FB}}}{E_\mathrm{F}^r}=8\sqrt{\pi }\frac{1+m_\mathrm{F}/m_\mathrm{B}}{(1+\sqrt{V_\mathrm{F}/V_\mathrm{B}})^{3/2}}\frac{a_{\mathrm{FB}}}{\lambda \gamma }\left(V_\mathrm{F}^z\right)^{1/4}V_\mathrm{F}^{1/2}$$
(16)
and $`U_\mathrm{B}/E_\mathrm{B}^r=4\sqrt{2\pi }(a_\mathrm{B}/\lambda \gamma )\left(V_\mathrm{B}^z\right)^{1/4}V_\mathrm{B}^{1/2}`$. Using a finite angle between the laser beams producing the standing light waves, we allow to change the relative size of the in- and out-of-plane lattice constants $`a`$ and $`a_z`$. The parameter $`\gamma =2a_z/\lambda `$ then denotes the increase in the unit cell volume and allows to tune the interaction strengths $`U_\mathrm{B}`$ and $`U_{\mathrm{FB}}`$ independent on $`J_{\mathrm{F},\mathrm{B}}`$ (alternatively, Feshbach resonances allow to tune $`a_{\mathrm{FB}}`$). Fixing $`V_\mathrm{F}^z=20`$, $`n_\mathrm{F}=1/2`$, and $`n_\mathrm{B}=3/2`$, we obtain for $`\gamma =3`$ and $`V_\mathrm{F}=4.5`$ the coupling parameters $`\lambda _{\mathrm{FB}}0.39`$, $`t_\mathrm{B}0.55`$ ($`T_{\mathrm{DW}}48\mathrm{nK}`$); we enter a regime at the border of validity of our weak coupling analysis. Using the above estimates, the $`V_\mathrm{F}`$-$`\gamma `$ phase diagram is shown in Fig. 3; we find that changing the strength of the optical lattice $`V_\mathrm{F}`$ allows to drive the transition from the supersolid to a phase separated state. The supersolid state is easily detected via the usual coherence peak of a bosonic condensate in an optical lattice and the additional appearance of coherence peaks at $`𝐤_{\mathrm{DW}}`$; using the above parameters, the weight of these additional peaks involve $`15\%`$ of the total particle number.

In conclusion, we have identified a new supersolid phase in a 2D Fermi-Bose gas mixture subject to an optical lattice; the bosons then play the role of the phonons in a condensed matter system. The perfect Fermi-surface nesting leads to the appearance of a fermionic density wave and the condensation of the bosons at $`𝐤_{\mathrm{DW}}`$. The interference of this $`𝐤_{\mathrm{DW}}`$-condensate with the usual $`𝐪=0`$-condensate establishes the bosonic density wave characteristic of the supersolid phase.

###### Acknowledgements.

We thank T. M. Rice, T. Esslinger, M. Köhl, and M. Troyer for stimulating discussions.