text:
# 1 Introduction

## 1 Introduction

In the early sixties, Dyson predicted that the local correlations between the eigenvalues of ensembles of random matrices, when their size tends to infinity, have universal behavior in the bulk of the spectrum. He expected that this universal behavior depends only on the type of the ensemble: orthogonal, unitary or symplectic. This constitutes the famous conjecture of universality in the theory of random matrices. For the classical ensembles (Hermite, Laguerre and Jacobi), this conjecture has been proven, see for example . For the unitary ensembles much more is known due to the connection with orthogonal polynomials, and the universality conjecture in the bulk of the spectrum is proved for a wide class of unitary ensembles, see .

At the edge of the spectrum this universal behavior breaks down. For Hermite ensembles, it is known that the local correlations (at the soft edge) can be expressed in terms of Airy functions , and for Jacobi and Laguerre ensembles (at the hard edge) in terms of Bessel functions . For example, for the Jacobi Unitary Ensemble

$$\frac{1}{\stackrel{~}{Z}_n}e^{\mathrm{tr}\mathrm{log}w(M)}dM,$$
(1.1)
where $`w(x)=(1x)^\alpha (1+x)^\beta `$ is the Jacobi weight, the eigenvalue correlations near $`1`$ are expressed in terms of the Bessel kernel

$$𝕁_\alpha (u,v)=\frac{J_\alpha (\sqrt{u})\sqrt{v}J_\alpha ^{}(\sqrt{v})J_\alpha (\sqrt{v})\sqrt{u}J_\alpha ^{}(\sqrt{u})}{2(uv)}$$
(1.2)
as $`n\mathrm{}`$. $`J_\alpha `$ is the usual Bessel function of the first kind and order $`\alpha `$. The order agrees with the exponent of $`1x`$ in the Jacobi weight.

Nagao and Wadati \[20, $`\mathrm{\S }6`$\] expect that a universality result persists for more general Jacobi-like ensembles, in the sense that the local form of the weight function near $`1`$ determines the eigenvalue correlation near $`1`$. It is the aim of this paper to prove this universal behavior for a generalization of the Jacobi Unitary Ensemble, which we call the Modified Jacobi Unitary Ensemble (MJUE). The MJUE is given by (1.1) with modified Jacobi weight

$$w(x)=(1x)^\alpha (1+x)^\beta h(x),\text{for }x(1,1),$$
(1.3)
where $`\alpha ,\beta >1`$ and the extra factor $`h`$ is real analytic and strictly positive on $`[1,1]`$. The Modified Jacobi Ensemble is a probability measure on the space of $`n\times n`$ Hermitian matrices with all eigenvalues in $`(1,1)`$. The MJUE gives rise to a probability density function of the $`n`$ eigenvalues $`x_1,x_2,\mathrm{},x_n`$ given by

$$P^{(n)}(x_1,x_2,\mathrm{},x_n)=\frac{1}{Z_n}\underset{j=1}{\overset{n}{}}w(x_j)\underset{i<j}{}|x_ix_j|^2,$$
(1.4)
with $`x_1,x_2,\mathrm{},x_n(1,1)`$ and $`Z_n`$ a normalizing constant.

Dyson  showed, see also , that we can express the correlation functions $`_{n,m}`$

$$_{n,m}(x_1,\mathrm{}x_m)=\frac{n!}{(nm)!}\underset{nm}{\underset{}{\mathrm{}}}P^{(n)}(x_1,\mathrm{},x_m,x_{m+1},\mathrm{},x_n)dx_{m+1}\mathrm{}dx_n$$

in terms of orthogonal polynomials. Denote the $`n`$th degree orthonormal polynomial with respect to the modified Jacobi weight $`w`$ by $`p_n(z)=p_n(z;w)=\gamma _nz^n+\mathrm{}`$, $`\gamma _n>0`$. Then $`_{n,m}(x_1,\mathrm{}x_m)=det\left(K_n(x_i,x_j)\right)_{1i,jm}`$, where

$$K_n(x,y)=\sqrt{w(x)}\sqrt{w(y)}\underset{j=0}{\overset{n1}{}}p_j(x)p_j(y).$$
(1.5)
By the Christoffel-Darboux formula, we have

$$K_n(x,y)=\sqrt{w(x)}\sqrt{w(y)}\frac{\gamma _{n1}}{\gamma _n}\frac{p_n(x)p_{n1}(y)p_{n1}(x)p_n(y)}{xy},$$
(1.6)
which shows that asymptotic properties of $`K_n`$ are intimately related with asymptotics of the orthogonal polynomials $`p_n`$ as $`n\mathrm{}`$.

In a previous paper with K.T-R McLaughlin and W. Van Assche , we studied the asymptotics of the polynomials that are orthogonal with respect to the modified Jacobi weight. We used the Riemann-Hilbert formulation for orthogonal polynomials of Fokas, Its, and Kitaev  and the steepest descent method for Riemann-Hilbert problems of Deift and Zhou . In  we concentrated on the asymptotics of the polynomials away from the interval $`[1,1]`$, but the Riemann-Hilbert method gives uniform asymptotics in all regions in the complex plane. Here we are interested in the behavior on $`[1,1]`$, and in particular near the endpoints $`\pm 1`$. The Riemann-Hilbert method was applied before to orthogonal polynomials by Deift and co-authors . They studied orthogonal polynomials on the real line with varying weights, and used the asymptotics to prove the universality in the bulk of the spectrum for the associated unitary ensembles. We apply the same method to prove the universality at the edge of the spectrum for the MJUE. Our main result is the following.

###### Theorem 1.1

Let $`w`$ be the modified Jacobi weight (1.3) and let $`K_n`$ be the kernel (1.5) associated with $`w`$. Then the following holds.

1. For $`x(1,1)`$, we have as $`n\mathrm{}`$,

$$\frac{1}{n}K_n(x,x)=\frac{1}{\pi \sqrt{1x^2}}+O\left(\frac{1}{n}\right).$$
(1.7)
The error term is uniform for $`x`$ in compact subsets of $`(1,1)`$.
2. Let $`\xi (x)=\frac{1}{\pi \sqrt{1x^2}}`$. Then for $`x(1,1)`$ and $`u,v`$, we have as $`n\mathrm{}`$,

$$\frac{1}{n\xi (x)}K_n(x+\frac{u}{n\xi (x)},x+\frac{v}{n\xi (x)})=\frac{\mathrm{sin}\pi (uv)}{\pi (uv)}+O\left(\frac{1}{n}\right).$$
(1.8)
The error term is uniform for $`x`$ in compact subsets of $`(1,1)`$ and for $`u,v`$ in compact subsets of $``$.
3. For $`u,v(0,\mathrm{})`$, we have as $`n\mathrm{}`$,

$$\frac{1}{2n^2}K_n(1\frac{u}{2n^2},1\frac{v}{2n^2})=𝕁_\alpha (u,v)+O\left(\frac{u^{\frac{\alpha }{2}}v^{\frac{\alpha }{2}}}{n}\right),$$
(1.9)
where $`𝕁_\alpha `$ is the Bessel kernel given by (1.2). The error term is uniform for $`u,v`$ in bounded subsets of $`(0,\mathrm{})`$.

Note that the error term in (1.9) holds uniformly for $`u,v`$ in bounded subsets of $`(0,\mathrm{})`$, not just in compact subsets. By symmetry, there is a corresponding universality result near $`1`$.

The eigenvalue density is the 1–point correlation function $`_{n,1}(x)=K_n(x,x)`$, see for example . Therefore, part (a) of the theorem yields the asymptotic eigenvalue density $`_{n,1}(x)n\xi (x)`$ as $`n\mathrm{}`$. This result is in agreement with . The scaling in (1.8) has the effect that $`x`$ is the new origin and that the asymptotic eigenvalue density at $`x`$ is 1. At the endpoints, (1.7) breaks down, and the eigenvalue density is $`O(n^2)`$ as $`n\mathrm{}`$, near the endpoints, see for example . This explains the scaling in (1.9).

Part (b) of the theorem states the universality (independent of the choice of $`\alpha ,\beta ,h`$ and $`x`$) for $`K_n`$ in the bulk of the spectrum. It extends the result of Nagao and Wadati \[19, (4.19)\] for the case that $`h1`$. At the edge 1 of the spectrum we have a universality class for $`K_n`$ (independent of the choice of $`\beta `$ and $`h`$) which is only affected by the local form of the modified Jacobi weight near 1, see part (c).

Using Theorem 1.1 we can answer local statistical quantities concerning the eigenvalues. Here we follow . The probability $`P_n(a,b)`$ that there are no eigenvalues in the interval $`(a,b)(1,1)`$ is given by

$$P_n(a,b)=det(IK_n),$$

where $`K_n`$ is the trace class operator with integral kernel $`K_n(x,y)`$ acting on $`L^2(a,b)`$, and where $`det(IK_n)`$ is the Fredholm determinant. For a fixed interval $`(a,b)`$ we have that $`P_n(a,b)0`$, as $`n\mathrm{}`$. So, to understand the asymptotic behavior of $`P_n`$ at the edge of the spectrum, we will look at intervals near the edges which shrink with $`n`$, and we are led to consider the asymptotic behavior of $`P_n(1\frac{s}{2n^2},1)`$ as $`n\mathrm{}`$, where $`s>0`$. We have the following universality for $`P_n`$ at the edge 1 of the spectrum, depending on the parameter $`\alpha `$ but independent of the choice of $`\beta `$ and $`h`$.

###### Corollary 1.2

For $`s>0`$, we have

$$\underset{n\mathrm{}}{lim}P_n(1\frac{s}{2n^2},1)=det(I𝕁_{\alpha ,s}),$$
(1.10)
where $`𝕁_{\alpha ,s}`$ is the integral operator with kernel $`𝕁_\alpha (u,v)`$ acting on $`L^2(0,s)`$, and $`det(I𝕁_{\alpha ,s})`$ is the Fredholm determinant.

As mentioned before, our main tool in proving Theorem 1.1 is the asymptotic analysis of the Riemann-Hilbert problem for the orthogonal polynomials with respect to the modified Jacobi weight, as developed in . We give an overview of this work in section 2. This approach is able to give strong and uniform asymptotics for the orthogonal polynomials in every region in the complex plane, which we also review in section 2. The proofs of Theorem 1.1 and Corollary 1.2 are given in section 3.

## 2 Riemann–Hilbert problem for orthogonal polynomials

In this section we will recall the Riemann–Hilbert problem (RH problem) from  for the orthogonal polynomials for the modified Jacobi weight, and the steps in the steepest descent method that are used to obtain the asymptotics on the interval.

The analysis starts from a characterization of the monic orthogonal polynomials $`\pi _n`$ for the modified Jacobi weight $`w`$ given by (1.3) as a solution of a RH problem for a $`2\times 2`$ matrix valued function $`Y(z)=Y(z;n,w)`$. This characterization of orthogonal polynomials is due to Fokas, Its, and Kitaev . The conditions (2.3) and (2.4) are needed to control the behavior near the endpoints, see  for discussion.

1. $`Y(z)`$ is analytic for $`z[1,1]`$.
2. $`Y`$ possesses continuous boundary values for $`x(1,1)`$ denoted by $`Y_+(x)`$ and $`Y_{}(x)`$, where $`Y_+(x)`$ and $`Y_{}(x)`$ denote the limiting values of $`Y(z)`$ as $`z`$ approaches $`x`$ from above and below, respectively, and

$$Y_+(x)=Y_{}(x)\left(\begin{array}{cc}1& w(x)\\ 0& 1\end{array}\right),\text{for }x(1,1)\text{.}$$
(2.1)
3. $`Y(z)`$ has the following asymptotic behavior at infinity:

$$Y(z)=\left(I+O\left(\frac{1}{z}\right)\right)\left(\begin{array}{cc}z^n& 0\\ 0& z^n\end{array}\right),\text{as }z\mathrm{}\text{.}$$
(2.2)
4. $`Y(z)`$ has the following behavior near $`z=1`$:

$$Y(z)=\{\begin{array}{cc}O\left(\begin{array}{cc}1& |z1|^\alpha \\ 1& |z1|^\alpha \end{array}\right),& \text{if }\alpha <0\text{,}\hfill \\ O\left(\begin{array}{cc}1& \mathrm{log}|z1|\\ 1& \mathrm{log}|z1|\end{array}\right),& \text{if }\alpha =0\text{,}\hfill \\ O\left(\begin{array}{cc}1& 1\\ 1& 1\end{array}\right),& \text{if }\alpha >0\text{,}\hfill \end{array}$$
(2.3)
as $`z1`$, $`z[1,1]`$.
5. $`Y(z)`$ has the following behavior near $`z=1`$:

$$Y(z)=\{\begin{array}{cc}O\left(\begin{array}{cc}1& |z+1|^\beta \\ 1& |z+1|^\beta \end{array}\right),& \text{if }\beta <0\text{,}\hfill \\ O\left(\begin{array}{cc}1& \mathrm{log}|z+1|\\ 1& \mathrm{log}|z+1|\end{array}\right),& \text{if }\beta =0\text{,}\hfill \\ O\left(\begin{array}{cc}1& 1\\ 1& 1\end{array}\right),& \text{if }\beta >0\text{,}\hfill \end{array}$$
(2.4)
as $`z1`$, $`z[1,1]`$.

The unique solution of this RH Problem is given by

$$Y(z)=\left(\begin{array}{cc}\pi _n(z)& \frac{1}{2\pi i}_1^1\frac{\pi _n(x)w(x)}{xz}𝑑x\\ 2\pi i\gamma _{n1}^2\pi _{n1}(z)& \gamma _{n1}^2_1^1\frac{\pi _{n1}(x)w(x)}{xz}𝑑x\end{array}\right),$$
(2.5)
where $`\pi _n`$ is the monic polynomial of degree $`n`$ orthogonal with respect to the weight $`w`$ and with $`\gamma _n`$ the leading coefficient of the orthonormal polynomial $`p_n`$.

We apply a number of transformations $`YTSR`$ to the original RH problem in order to arrive at a RH problem for $`R`$, which is normalized at infinity, and whose jump matrices are uniformly close to the identity matrix. Then, $`R`$ is uniformly close to the identity matrix, and by tracing back the steps we deduce the asymptotic behavior of $`Y`$.

In the first transformation we turn the original RH problem into an equivalent RH problem for $`T`$, which is normalized at infinity and with a jump matrix whose diagonal elements are oscillatory. Let

$$\sigma _3=\left(\begin{array}{cc}1& 0\\ 0& 1\end{array}\right)$$

be the Pauli matrix, and let $`T`$ be given by

$$T(z)=2^{n\sigma _3}Y(z)\phi (z)^{n\sigma _3},$$
(2.6)
where $`\phi (z)=z+(z^21)^{1/2}`$ for $`z[1,1]`$, so that $`\phi `$ is the conformal map from $`[1,1]`$ onto the exterior of the unit circle. Then $`T`$ satisfies a RH problem which is normalized at infinity (i.e., $`T(z)I`$ as $`z\mathrm{}`$), and

$$T_+(x)=T_{}(x)\left(\begin{array}{cc}\phi _+(x)^{2n}& w(x)\\ 0& \phi _{}(x)^{2n}\end{array}\right),\text{for }x(1,1).$$

The jump matrix for $`T`$ factors into a product of three matrices. Using this factorization, we transform the RH problem for $`T`$ into an equivalent RH problem for $`S`$, with jumps on a lens shaped contour $`\mathrm{\Sigma }`$, as in Figure 1. $`S`$ is defined in terms of $`T`$ by

$$S(z)=\{\begin{array}{cc}T(z),& \text{for }z\text{ outside the lens,}\hfill \\ T(z)\left(\begin{array}{cc}1& 0\\ w(z)^1\phi (z)^{2n}& 1\end{array}\right),& \text{for }z\text{ in the upper part of the lens,}\hfill \\ T(z)\left(\begin{array}{cc}1& 0\\ w(z)^1\phi (z)^{2n}& 1\end{array}\right),& \text{for }z\text{ in the lower part of the lens.}\hfill \end{array}$$
(2.7)
For the third transformation, we need to construct parametrices in the outside region and near the endpoints $`\pm 1`$. Constructing the parametrix in the outside region, we need the Szegő function $`D(z)=D(z;w)`$ associated with the weight $`w`$ given by

$$D(z)=\mathrm{exp}\left(\frac{(z^21)^{1/2}}{2\pi }_1^1\frac{\mathrm{log}w(x)}{\sqrt{1x^2}}\frac{dx}{zx}\right),\text{for }z[1,1].$$
(2.8)
The Szegő function is a non-zero analytic function on $`[1,1]`$ such that $`D_+(x)D_{}(x)=w(x)`$ for $`x(1,1)`$. The parametrix $`N`$ in the outside region is given by

$$N(z)=D_{\mathrm{}}^{\sigma _3}\left(\begin{array}{cc}\frac{a(z)+a(z)^1}{2}& \frac{a(z)a(z)^1}{2i}\\ \frac{a(z)a(z)^1}{2i}& \frac{a(z)+a(z)^1}{2}\end{array}\right)D(z)^{\sigma _3},$$
(2.9)
where $`D_{\mathrm{}}=lim_z\mathrm{}D(z)`$ and $`a(z)=(z1)^{1/4}(z+1)^{1/4}`$.

Next, we define a parametrix $`P`$ in $`U_\delta `$, which is the disk with radius $`\delta `$ and center $`1`$, where $`\delta >0`$ is sufficiently small, by

$$P(z)=E_n(z)\mathrm{\Psi }(n^2f(z))W(z)^{\sigma _3}\phi (z)^{n\sigma _3},$$
(2.10)
where $`f(z)`$ and $`W(z)`$ are scalar functions given by

$$f(z)=\frac{1}{4}\left(\mathrm{log}\phi (z)\right)^2,$$
(2.11)
and

$$W(z)=(z1)^{\alpha /2}(z+1)^{\beta /2}h^{1/2}(z).$$
(2.12)
In (2.10) $`\mathrm{\Psi }(\zeta )`$ is a $`2\times 2`$ matrix valued function defined for $`\zeta \mathrm{\Sigma }_\mathrm{\Psi }`$, where $`\mathrm{\Sigma }_\mathrm{\Psi }`$ is the contour shown in Figure 2.

For our purpose here, it suffices to know the expression of $`\mathrm{\Psi }(\zeta )`$ for $`2\pi /3<\mathrm{arg}\zeta <\pi `$. This is given in terms of the Hankel functions $`H_\alpha ^{(1)}`$ and $`H_\alpha ^{(2)}`$. For $`2\pi /3<\mathrm{arg}\zeta <\pi `$, we have

$$\mathrm{\Psi }(\zeta )=\left(\begin{array}{cc}\frac{1}{2}H_\alpha ^{(1)}(2(\zeta )^{1/2})& \frac{1}{2}H_\alpha ^{(2)}(2(\zeta )^{1/2})\\ \pi \zeta ^{1/2}\left(H_\alpha ^{(1)}\right)^{}(2(\zeta )^{1/2})& \pi \zeta ^{1/2}\left(H_\alpha ^{(2)}\right)^{}(2(\zeta )^{1/2})\end{array}\right)e^{\frac{1}{2}\alpha \pi i\sigma _3}.$$
(2.13)
The factor $`E_n(z)`$ in (2.10) is given by

$$E_n(z)=N(z)W(z)^{\sigma _3}\frac{1}{\sqrt{2}}\left(\begin{array}{cc}1& i\\ i& 1\end{array}\right)f(z)^{\sigma _3/4}\left(2\pi n\right)^{\sigma _3/2}.$$
(2.14)
$`E_n(z)`$ is analytic in a full neighborhood of $`U_\delta `$.

There is a similar definition for the parametrix $`\stackrel{~}{P}`$ in a $`\delta `$ neighborhood $`\stackrel{~}{U}_\delta `$ of $`1`$, see  for details.

We then have all the ingredients for the third transformation. We define

$`R(z)`$ $`=`$ $`S(z)N^1(z),\text{for }z(\overline{U}_\delta \overline{\stackrel{~}{U}}_\delta \mathrm{\Sigma }),`$ (2.15)
$`R(z)`$ $`=`$ $`S(z)P^1(z),\text{for }zU_\delta \mathrm{\Sigma },`$ (2.16)
$`R(z)`$ $`=`$ $`S(z)\stackrel{~}{P}^1(z),\text{for }z\stackrel{~}{U}_\delta \mathrm{\Sigma }.`$ (2.17)
Then $`R`$ satisfies a RH problem with jumps on the system of contours $`\mathrm{\Sigma }_R`$ shown in Figure 3.

Note that $`R`$ depends on $`n`$, but the contour $`\mathrm{\Sigma }_R`$ does not depend on $`n`$. The jump matrices for the RH problem for $`R`$ turn out to be uniformly close to the identity matrix with error term $`O(\frac{1}{n})`$. Then it follows that

$$R(z)=I+O\left(\frac{1}{n}\right),\text{as }n\mathrm{},$$
(2.18)
uniformly for $`z\mathrm{\Sigma }_R`$, and also, see \[3, Section 8.1\] in particular formulas (8.19) and (8.20),

$$\frac{d}{dz}R(z)=O\left(1\right),\text{as }n\mathrm{},$$
(2.19)
uniformly for $`z\mathrm{\Sigma }_R`$. In the following we will also use that

$$detR(z)1.$$
(2.20)
###### Remark 2.1

The asymptotic result (2.18) for $`R`$ may be refined and it is possible to obtain a full asymptotic expansion

$$R(z)I+\underset{k=1}{\overset{\mathrm{}}{}}\frac{R_k(z)}{n^k},\text{as }n\mathrm{}.$$

The matrix coefficients $`R_k(z)`$ are explicitly computable in function of $`\alpha `$, $`\beta `$, and the analytic factor $`h`$ in the modified Jacobi weight. The first two coefficients are computed in .

###### Remark 2.2

In  we also derived asymptotic expansions for the polynomials $`p_n`$ and $`\pi _n`$ uniformly valid in compact subsets of $`\overline{}[1,1]`$, as well as asymptotic expansions for the leading coefficient $`\gamma _n`$ and for the coefficients in the recurrence relation satisfied by the orthonormal polynomials. For example, we have by \[12, Theorem 1.4\] uniformly for $`z[1,1]`$,

$$\frac{2^n\pi _n(z)}{\phi (z)}=\frac{D_{\mathrm{}}}{D(z)}\frac{\phi (z)^{1/2}}{\sqrt{2}(z^21)^{1/4}}\left[1+O\left(\frac{1}{n}\right)\right],\text{as }n\mathrm{}.$$

The $`O(\frac{1}{n})`$ term can be developed into a complete asymptotic expansion in powers of $`n^1`$.

The Riemann-Hilbert method also leads to strong asymptotics on the interval $`(1,1)`$ and near the endpoints $`\pm 1`$. While these results are closely related to the asymptotics of $`K_n`$ as given in Theorem 1.1, we do not actually rely on them in the proof of Theorem 1.1. Therefore we state here the asymptotics of the orthogonal polynomials without proof. The proof is similar to the proof of Theorem 1.1, and in fact somewhat simpler. See also .

As before, we let $`\delta >0`$ be the radius of the disks $`U_\delta `$ and $`\stackrel{~}{U}_\delta `$. Then we have for $`x(1+\delta ,1\delta )`$,

$`\pi _n(x)`$ $`=`$ $`{\displaystyle \frac{\sqrt{2}D_{\mathrm{}}}{2^n\sqrt{w(x)}(1x^2)^{1/4}}}[R_{11}(x)\mathrm{cos}((n+1/2)\mathrm{arccos}x+\psi (x){\displaystyle \frac{\pi }{4}})`$ (2.21)
$`{\displaystyle \frac{i}{D_{\mathrm{}}^2}}R_{12}(x)\mathrm{cos}((n1/2)\mathrm{arccos}x+\psi (x){\displaystyle \frac{\pi }{4}})],`$
where $`R_{11}(x)=1+O(\frac{1}{n})`$ and $`R_{12}(x)=O(\frac{1}{n})`$ as $`n\mathrm{}`$, uniformly for $`x(1+\delta ,1\delta )`$. The $`O(\frac{1}{n})`$ terms have a complete asymptotic expansion, see . The function $`\psi (x)`$ in (2.21) is given by

$`\psi (x)`$ $`=`$ $`{\displaystyle \frac{\sqrt{1x^2}}{2\pi }}{\displaystyle _1^1}{\displaystyle \frac{\mathrm{log}w(t)}{\sqrt{1t^2}}}{\displaystyle \frac{dt}{tx}}`$ (2.22)
$`=`$ $`{\displaystyle \frac{1}{2}}\left(\alpha (\mathrm{arccos}x\pi )+\beta \mathrm{arccos}x\right)+{\displaystyle \frac{\sqrt{1x^2}}{2\pi }}{\displaystyle _1^1}{\displaystyle \frac{\mathrm{log}h(t)}{\sqrt{1t^2}}}{\displaystyle \frac{dt}{tx}},`$
where the integral is a Cauchy principal value integral. We remark that, under various assumptions, asymptotic results on the interval of orthogonality have been established by many authors, see for example .

For $`x(1\delta ,1)`$, the following is valid

$`\pi _n(x)`$ $`=`$ $`{\displaystyle \frac{\sqrt{\pi }D_{\mathrm{}}}{2^n\sqrt{w(x)}}}{\displaystyle \frac{(n\mathrm{arccos}x)^{1/2}}{(1x^2)^{1/4}}}\times `$ (2.23)
$`[R_{11}(x)(\mathrm{cos}\zeta _1(x)J_\alpha (n\mathrm{arccos}x)+\mathrm{sin}\zeta _1(x)J_\alpha ^{}(n\mathrm{arccos}x))`$
$`{\displaystyle \frac{i}{D_{\mathrm{}}^2}}R_{12}(x)(\mathrm{cos}\zeta _2(x)J_\alpha (n\mathrm{arccos}x)+\mathrm{sin}\zeta _2(x)J_\alpha ^{}(n\mathrm{arccos}x))],`$
where $`R_{11}(x)=1+O(\frac{1}{n})`$ and $`R_{12}(x)=O(\frac{1}{n})`$ as $`n\mathrm{}`$, uniformly for $`x(1\delta ,1)`$, and where

$$\zeta _1(x)=\frac{1}{2}\mathrm{arccos}x+\psi (x)+\frac{\alpha \pi }{2},\zeta _2(x)=\frac{1}{2}\mathrm{arccos}x+\psi (x)+\frac{\alpha \pi }{2},$$

with $`\psi `$ given by (2.22). There is an analogous expression for $`x`$ in the interval $`(1,1+\delta )`$.

## 3 Proof of Theorem 1.1 and Corollary 1.2

In this section we prove Theorem 1.1 and Corollary 1.2. We follow the work of Deift et al. . Recall that

$`K_n(x,y)`$ $`=`$ $`\sqrt{w(x)}\sqrt{w(y)}{\displaystyle \frac{\gamma _{n1}}{\gamma _n}}{\displaystyle \frac{p_n(x)p_{n1}(y)p_{n1}(x)p_n(y)}{xy}}`$
$`=`$ $`\sqrt{w(x)}\sqrt{w(y)}\gamma _{n1}^2{\displaystyle \frac{\pi _n(x)\pi _{n1}(y)\pi _{n1}(x)\pi _n(y)}{xy}},`$
where $`\pi _n`$ is the monic orthogonal polynomial of degree $`n`$ with respect to the modified Jacobi weight $`w`$. As in  we replace the polynomials $`\pi _{n1}`$ and $`\pi _n`$ by the appropriate entries of $`Y`$, see (2.5), to obtain

$$K_n(x,y)=\frac{1}{2\pi i}\sqrt{w(x)}\sqrt{w(y)}\frac{Y_{11}(x)Y_{21}(y)Y_{21}(x)Y_{11}(y)}{xy}.$$
(3.1)
Thus, $`K_n`$ can be expressed in terms of the first column of $`Y`$. The asymptotic behavior of $`Y`$ follows from the transformations $`YTSR`$ described in section 2, and the behavior (2.18)–(2.20) of $`R`$.

### 3.1 Proof of Theorem 1.1 (a)

We will first express $`Y_{11}`$ and $`Y_{21}`$ in terms of $`R`$. In the following, $`\delta >0`$ will be a small but fixed number. This number is the radius of the disks $`U_\delta `$ and $`\stackrel{~}{U}_\delta `$ used in the local RH analysis around $`\pm 1`$.

###### Lemma 3.1

We have for $`x(1+\delta ,1\delta )`$,

$$\left(\begin{array}{c}Y_{11}(x)\\ Y_{21}(x)\end{array}\right)=\frac{1}{\sqrt{w(x)}}\mathrm{\hspace{0.17em}2}^{n\sigma _3}L_+(x)\left(\begin{array}{c}e^{in\mathrm{arccos}x}\\ e^{in\mathrm{arccos}x}\end{array}\right),$$
(3.2)
with $`L(z)`$ given by

$$L(z)=R(z)D_{\mathrm{}}^{\sigma _3}\left(\begin{array}{cc}\frac{a(z)+a(z)^1}{2}& \frac{a(z)a(z)^1}{2i}\\ \frac{a(z)a(z)^1}{2i}& \frac{a(z)+a(z)^1}{2}\end{array}\right)e^{i\psi (z)\sigma _3},$$
(3.3)
where

$$\psi (x)=\frac{1}{2}\left(\alpha (\mathrm{arccos}x\pi )+\beta \mathrm{arccos}x\right)+\frac{\sqrt{1x^2}}{2\pi }\text{ }_1^1\frac{\mathrm{log}h(t)}{\sqrt{1t^2}}\frac{dt}{tx}.$$
(3.4)
The matrices $`L_+(x)`$ and $`\frac{d}{dx}L_+(x)`$ are uniformly bounded for $`x(1+\delta ,1\delta )`$ as $`n\mathrm{}`$, and

$$detL_+(x)1,\text{for }x(1+\delta ,1\delta ).$$
(3.5)
###### Remark 3.2

The singular integral (3.4) is being understood in the sense of the principal value. It may be shown that $`D_+(x)=\sqrt{w(x)}\mathrm{exp}(i\psi (x))`$ so that $`\psi (x)`$ is the argument of the Szegő function on the interval.

* We use the series of transformations $`YTSR`$ and we unfold them for $`z`$ in the upper part of the lens but outside the disks $`U_\delta `$ and $`\stackrel{~}{U}_\delta `$, and then take the limit to the interval $`(1+\delta ,1\delta )`$. Thus, let $`z`$ be in the upper part of the lens but outside the disks $`U_\delta `$ and $`\stackrel{~}{U}_\delta `$. We then have by (2.6), (2.7) and (2.15)

$$Y(z)=2^{n\sigma _3}R(z)N(z)\left(\begin{array}{cc}\phi (z)^n& 0\\ w(z)^1\phi (z)^n& \phi (z)^n\end{array}\right).$$
(3.6)
Inserting the expression (2.9) for $`N`$ into (3.6), we obtain for the first column of $`Y`$

$$\left(\begin{array}{c}Y_{11}(z)\\ Y_{21}(z)\end{array}\right)=2^{n\sigma _3}R(z)D_{\mathrm{}}^{\sigma _3}\left(\begin{array}{cc}\frac{a(z)+a(z)^1}{2}& \frac{a(z)a(z)^1}{2i}\\ \frac{a(z)a(z)^1}{2i}& \frac{a(z)+a(z)^1}{2}\end{array}\right)\left(\begin{array}{c}\frac{1}{D(z)}\phi (z)^n\\ \frac{D(z)}{w(z)}\phi (z)^n\end{array}\right).$$
(3.7)
We now take the limit $`zx(1+\delta ,1\delta )`$. Since $`D_+(x)=\sqrt{w(x)}e^{i\psi (x)}`$ and $`\phi _+(x)=\mathrm{exp}(i\mathrm{arccos}x)`$, (3.2) now follows from (3.7).

Note that $`R(x)`$ and $`\frac{d}{dx}R(x)`$ are uniformly bounded for $`x(1+\delta ,1\delta )`$ as $`n\mathrm{}`$. Let $`U`$ be a neighborhood of $`[1,1]`$ such that $`\mathrm{log}h`$ is defined and analytic in $`U`$, and let $`\gamma `$ be a closed contour in $`U[1,1]`$ encircling the interval $`[1,1]`$ once in the positive direction. Via contour deformation, we may write $`\psi `$ in the form

$$\psi (x)=\frac{1}{2}\left(\alpha (\mathrm{arccos}x\pi )+\beta \mathrm{arccos}x\right)+\frac{\sqrt{1x^2}}{4\pi i}_\gamma \frac{\mathrm{log}h(\zeta )}{(\zeta ^21)^{1/2}}\frac{d\zeta }{\zeta x}.$$

Thus $`\psi `$ has an analytic extension to a neighborhood of $`(1,1)`$. This implies that $`\psi `$ and its derivative are bounded on $`(1+\delta ,1\delta )`$. From the explicit form (3.3) of $`L_+`$ we then find that $`L_+(x)`$ and $`\frac{d}{dx}L_+(x)`$ are uniformly bounded for $`x(1+\delta ,1\delta )`$ as $`n\mathrm{}`$. Since $`detR(z)1`$ it is easy to see from (3.3) that $`detL_+(x)1`$. $`\mathrm{}`$

* of Theorem 1.1 (a) Letting $`yx`$ in (3.1) we get

$`K_n(x,x)`$ $`=`$ $`{\displaystyle \frac{1}{2\pi i}}w(x)\left(Y_{11}(x)Y_{21}^{}(x)Y_{21}(x)Y_{11}^{}(x)\right)`$ (3.8)
$`=`$ $`{\displaystyle \frac{1}{2\pi i}}det\left(\begin{array}{cc}\sqrt{w(x)}Y_{11}(x)& \sqrt{w(x)}Y_{11}^{}(x)\\ \sqrt{w(x)}Y_{21}(x)& \sqrt{w(x)}Y_{21}^{}(x)\end{array}\right)`$
$`=`$ $`{\displaystyle \frac{1}{2\pi i}}det\left(\begin{array}{cc}2^n\sqrt{w(x)}Y_{11}(x)& \frac{d}{dx}\left(2^n\sqrt{w(x)}Y_{11}(x)\right)\\ 2^n\sqrt{w(x)}Y_{21}(x)& \frac{d}{dx}\left(2^n\sqrt{w(x)}Y_{21}(x)\right)\end{array}\right).`$
By (3.2) the matrix in (3.8) is equal to

$`L_+(x)\left(\begin{array}{cc}e^{in\mathrm{arccos}x}& \frac{in}{\sqrt{1x^2}}e^{in\mathrm{arccos}x}\\ e^{in\mathrm{arccos}x}& \frac{in}{\sqrt{1x^2}}e^{in\mathrm{arccos}x}\end{array}\right)+\left({\displaystyle \frac{d}{dx}}L_+(x)\right)\left(\begin{array}{cc}0& e^{in\mathrm{arccos}x}\\ 0& e^{in\mathrm{arccos}x}\end{array}\right)`$
$`=L_+(x)e^{in\mathrm{arccos}x\sigma _3}\left(\begin{array}{cc}1& \frac{in}{\sqrt{1x^2}}\\ 1& \frac{in}{\sqrt{1x^2}}\end{array}\right)+\left({\displaystyle \frac{d}{dx}}L_+(x)\right)\left(\begin{array}{cc}0& e^{in\mathrm{arccos}x}\\ 0& e^{in\mathrm{arccos}x}\end{array}\right).`$
Since $`L_+(x)`$ and $`e^{in\mathrm{arccos}x\sigma _3}`$ have determinant one, it follows that

$$K_n(x,x)=\frac{1}{2\pi i}det\left[\left(\begin{array}{cc}1& \frac{in}{\sqrt{1x^2}}\\ 1& \frac{in}{\sqrt{1x^2}}\end{array}\right)+e^{in\mathrm{arccos}x\sigma _3}L_+^1(x)\left(\frac{d}{dx}L_+(x)\right)\left(\begin{array}{cc}0& e^{in\mathrm{arccos}x}\\ 0& e^{in\mathrm{arccos}x}\end{array}\right)\right].$$
(3.9)
The entries of $`L_+(x)`$ and $`\frac{d}{dx}L_+(x)`$ are uniformly bounded by Lemma 3.1. Since $`detL_+(x)=1`$, also the entries of $`L_+^1(x)`$ are uniformly bounded. Thus, we have uniformly for $`x(1+\delta ,1\delta )`$,

$`K_n(x,x)`$ $`=`$ $`{\displaystyle \frac{1}{2\pi i}}det\left(\begin{array}{cc}1& \frac{in}{\sqrt{1x^2}}+O(1)\\ 1& \frac{in}{\sqrt{1x^2}}+O(1)\end{array}\right)={\displaystyle \frac{1}{2\pi i}}\left({\displaystyle \frac{2in}{\sqrt{1x^2}}}+O(1)\right)`$
$`=`$ $`{\displaystyle \frac{n}{\pi \sqrt{1x^2}}}+O(1).`$
This proves part (a) of Theorem 1.1. $`\mathrm{}`$

### 3.2 Proof of Theorem 1.1 (b)

* of Theorem 1.1 (b) Let $`x(1,1)`$ and $`u,v`$. For the sake of brevity, we use $`u_{x,n}`$ and $`v_{x,n}`$ to denote $`x+\frac{u}{n\xi (x)}`$ and $`x+\frac{v}{n\xi (x)}`$, respectively. We write

$$\widehat{K}_n(u,v)=\frac{1}{n\xi (x)}K_n(u_{x,n},v_{x,n}).$$

From (3.1) we then have

$`\widehat{K}_n(u,v)`$ $`=`$ $`{\displaystyle \frac{1}{2\pi i(uv)}}\sqrt{w(u_{x,n})}\sqrt{w(v_{x,n})}det\left(\begin{array}{cc}Y_{11}(u_{x,n})& Y_{11}(v_{x,n})\\ Y_{21}(u_{x,n})& Y_{21}(v_{x,n})\end{array}\right)`$ (3.10)
$`=`$ $`{\displaystyle \frac{1}{2\pi i(uv)}}det\left(\begin{array}{cc}2^n\sqrt{w(u_{x,n})}Y_{11}(u_{x,n})& 2^n\sqrt{w(v_{x,n})}Y_{11}(v_{x,n})\\ 2^n\sqrt{w(u_{x,n})}Y_{21}(u_{x,n})& 2^n\sqrt{w(v_{x,n})}Y_{21}(v_{x,n})\end{array}\right).`$
Next, we use the expression (3.2) for $`Y_{11}`$ and $`Y_{21}`$ in (3.10) to obtain

$`\widehat{K}_n(u,v)`$ $`=`$ $`{\displaystyle \frac{1}{2\pi i(uv)}}det[L_+(u_{x,n})\left(\begin{array}{cc}e^{in\mathrm{arccos}u_{x,n}}& 0\\ e^{in\mathrm{arccos}u_{x,n}}& 0\end{array}\right)`$ (3.11)
$`+L_+(v_{x,n})\left(\begin{array}{cc}0& e^{in\mathrm{arccos}v_{x,n}}\\ 0& e^{in\mathrm{arccos}v_{x,n}}\end{array}\right)]`$
$`=`$ $`{\displaystyle \frac{1}{2\pi i(uv)}}det[L_+(v_{x,n})\left(\begin{array}{cc}e^{in\mathrm{arccos}u_{x,n}}& e^{in\mathrm{arccos}v_{x,n}}\\ e^{in\mathrm{arccos}u_{x,n}}& e^{in\mathrm{arccos}v_{x,n}}\end{array}\right)`$
$`+[L_+(u_{x,n})L_+(v_{x,n})]\left(\begin{array}{cc}e^{in\mathrm{arccos}u_{x,n}}& 0\\ e^{in\mathrm{arccos}u_{x,n}}& 0\end{array}\right)].`$
Since $`detL_+(v_{x,n})=1`$, we then get

$`\widehat{K}_n(u,v)`$ $`=`$ $`{\displaystyle \frac{1}{2\pi i(uv)}}det[\left(\begin{array}{cc}e^{in\mathrm{arccos}u_{x,n}}& e^{in\mathrm{arccos}v_{x,n}}\\ e^{in\mathrm{arccos}u_{x,n}}& e^{in\mathrm{arccos}v_{x,n}}\end{array}\right)`$ (3.12)
$`+L_+^1(v_{x,n})[L_+(u_{x,n})L_+(v_{x,n})]\left(\begin{array}{cc}e^{in\mathrm{arccos}u_{x,n}}& 0\\ e^{in\mathrm{arccos}u_{x,n}}& 0\end{array}\right)].`$
Since $`\frac{d}{dx}L_+(x)`$ is uniformly bounded as $`n\mathrm{}`$, we have by the mean value theorem

$$L_+(u_{x,n})L_+(v_{x,n})=O\left(u_{x,n}v_{x,n}\right)=O\left(\frac{uv}{n}\right),$$

uniformly for $`x(1+\delta ,1\delta )`$ and for $`u,v`$ in compact subsets of $``$. Since $`L_+(x)`$ is uniformly bounded, and $`detL_+(x)=1`$, we have that $`L_+^1(v_{x,n})`$ is uniformly bounded as well, so that

$$L_+^1(v_{x,n})[L_+(u_{x,n})L_+(v_{x,n})]\left(\begin{array}{cc}e^{in\mathrm{arccos}u_{x,n}}& 0\\ e^{in\mathrm{arccos}u_{x,n}}& 0\end{array}\right)=\left(\begin{array}{cc}O\left(\frac{uv}{n}\right)& 0\\ O\left(\frac{uv}{n}\right)& 0\end{array}\right).$$
(3.13)
Thus by (3.12) and (3.13)

$`\widehat{K}_n(u,v)`$ $`=`$ $`{\displaystyle \frac{1}{2\pi i(uv)}}det\left(\begin{array}{cc}e^{in\mathrm{arccos}u_{x,n}}+O\left(\frac{uv}{n}\right)& e^{in\mathrm{arccos}v_{x,n}}\\ e^{in\mathrm{arccos}u_{x,n}}+O\left(\frac{uv}{n}\right)& e^{in\mathrm{arccos}v_{x,n}}\end{array}\right)`$ (3.14)
$`=`$ $`{\displaystyle \frac{1}{2\pi i(uv)}}det\left(\begin{array}{cc}e^{in\mathrm{arccos}u_{x,n}}& e^{in\mathrm{arccos}v_{x,n}}\\ e^{in\mathrm{arccos}u_{x,n}}& e^{in\mathrm{arccos}v_{x,n}}\end{array}\right)+O\left({\displaystyle \frac{1}{n}}\right)`$
$`=`$ $`{\displaystyle \frac{1}{2\pi i(uv)}}\left(e^{in(\mathrm{arccos}u_{x,n}\mathrm{arccos}v_{x,n})}e^{in(\mathrm{arccos}u_{x,n}\mathrm{arccos}v_{x,n})}\right)+O\left({\displaystyle \frac{1}{n}}\right)`$
$`=`$ $`{\displaystyle \frac{\mathrm{sin}(n(\mathrm{arccos}u_{x,n}\mathrm{arccos}v_{x,n}))}{\pi (uv)}}+O\left({\displaystyle \frac{1}{n}}\right),`$
uniformly for $`x(1+\delta ,1\delta )`$ and $`u,v`$ in compact subsets of $``$. Since

$$n(\mathrm{arccos}u_{x,n}\mathrm{arccos}v_{x,n})=\pi (uv)(1+O(1/n)),\text{as }n\mathrm{}\text{,}$$

also uniformly for $`x(1+\delta ,1\delta )`$ and $`u,v`$ in compact subsets of $``$, part (b) of Theorem 1.1 follows. $`\mathrm{}`$

### 3.3 Proof of Theorem 1.1 (c)

To prove part (c) of Theorem 1.1, we start with a result similar to Lemma 3.1.

###### Lemma 3.3

For $`x(1\delta ,1)`$, we have

$$\left(\begin{array}{c}Y_{11}(x)\\ Y_{21}(x)\end{array}\right)=\sqrt{\frac{2\pi n}{w(x)}}\mathrm{\hspace{0.17em}2}^{n\sigma _3}M_+(x)\left(\begin{array}{c}J_\alpha (n\mathrm{arccos}x)\\ \frac{i}{2}\mathrm{arccos}xJ_\alpha ^{}(n\mathrm{arccos}x)\end{array}\right),$$
(3.15)
with $`M(z)`$ given by

$$M(z)=R(z)N(z)W(z)^{\sigma _3}\frac{1}{\sqrt{2}}\left(\begin{array}{cc}1& i\\ i& 1\end{array}\right)f(z)^{\sigma _3/4},$$
(3.16)
where $`R`$ is the result of the transformations $`YTSR`$ of the RH problem, the matrix valued function $`N`$ is given by (2.9), and the scalar functions $`f`$ and $`W`$ are given by (2.11) and (2.12), respectively.

$`M`$ is analytic in $`U_\delta `$ with $`M(z)`$ and $`\frac{d}{dz}M(z)`$ uniformly bounded for $`zU_\delta `$ as $`n\mathrm{}`$. Furthermore, we have

$$detM(z)1.$$

* As in the proof of Lemma 3.1 we unravel the transformations $`YTSR`$, but now for $`z`$ in the upper part of the lens and inside the disk $`U_\delta `$. We then have by (2.6), (2.7), (2.10) and (2.16)

$$Y(z)=2^{n\sigma _3}R(z)E_n(z)\mathrm{\Psi }\left(n^2f(z)\right)W(z)^{\sigma _3}\left(\begin{array}{cc}1& 0\\ w(z)^1& 1\end{array}\right).$$
(3.17)
Since $`\mathrm{Im}z>0`$, we have by (2.12) that $`W(z)=w^{1/2}(z)e^{i\pi \alpha /2}`$. Inserting this into (3.17) we get for the first column of $`Y`$

$$\left(\begin{array}{c}Y_{11}(z)\\ Y_{21}(z)\end{array}\right)=w^{1/2}(z)2^{n\sigma _3}R(z)E_n(z)\mathrm{\Psi }\left(n^2f(z)\right)\left(\begin{array}{c}e^{i\pi \alpha /2}\\ e^{i\pi \alpha /2}\end{array}\right).$$
(3.18)
Since $`z`$ is in the upper part of the lens and inside the disk $`U_\delta `$, we have $`2\pi /3<\mathrm{arg}n^2f(z)<\pi `$, see \[12, section 6\], and we thus use (2.13) to evaluate $`\mathrm{\Psi }(n^2f(z))`$. From formulas 9.1.3 and 9.1.4 of  we then have

$$\mathrm{\Psi }\left(n^2f(z)\right)\left(\begin{array}{c}e^{i\pi \alpha /2}\\ e^{i\pi \alpha /2}\end{array}\right)=\left(\begin{array}{c}J_\alpha (2n(f(z))^{1/2})\\ 2\pi nf^{1/2}(z)J_\alpha ^{}(2n(f(z))^{1/2})\end{array}\right).$$
(3.19)
By (2.14) and (3.16) we have $`R(z)E_n(z)=M(z)(2\pi n)^{\sigma _3/2}`$. Inserting this and (3.19) into (3.18) we get

$$\left(\begin{array}{c}Y_{11}(z)\\ Y_{21}(z)\end{array}\right)=\frac{\sqrt{2\pi n}}{w^{1/2}(z)}2^{n\sigma _3}M(z)\left(\begin{array}{c}J_\alpha (2n(f(z))^{1/2})\\ f^{1/2}(z)J_\alpha ^{}(2n(f(z))^{1/2})\end{array}\right).$$
(3.20)
We now take the limit $`zx(1\delta ,1)`$. By (2.11), and since $`\phi _+(x)=\mathrm{exp}(i\mathrm{arccos}x)`$ we have $`f_+^{1/2}(x)=\frac{i}{2}\mathrm{arccos}x`$, so that $`2(f_+(x))^{1/2}=\mathrm{arccos}x`$. Inserting this into (3.20), we obtain (3.15).

$`M`$ is analytic in $`U_\delta `$ since both $`E_n`$, see \[12, Proposition 6.5\], and $`R`$ are analytic in $`U_\delta `$. So, we may write $`M(x)`$ instead of $`M_+(x)`$ in (3.15).

By (2.18) and (2.19) we have that $`R(z)`$ and $`\frac{d}{dz}R(z)`$ are uniformly bounded for $`zU_\delta `$ as $`n\mathrm{}`$. Since

$$N(z)W(z)^{\sigma _3}\frac{1}{\sqrt{2}}\left(\begin{array}{cc}1& i\\ i& 1\end{array}\right)f(z)^{\sigma _3/4}$$

is analytic for $`zU_\delta `$ and does not depend on $`n`$, we have from (3.16) that also $`M(z)`$ and $`\frac{d}{dz}M(z)`$ are uniformly bounded for $`zU_\delta `$ as $`n\mathrm{}`$.

The fact that $`detM(z)1`$ follows easily from (2.9), (2.20), and (3.16). $`\mathrm{}`$

###### Remark 3.4

We check that (3.15) yields $`\left(\begin{array}{c}Y_{11}(x)\\ Y_{21}(x)\end{array}\right)=O(1)`$ as $`x1`$, which is in agreement with the fact that $`Y_{11}`$ and $`Y_{21}`$ are polynomials, see (2.5). Since $`\mathrm{arccos}x=O\left((1x)^{1/2}\right)`$ as $`x1`$, we have by formula 9.1.10 of

$$\left(\begin{array}{c}J_\alpha (n\mathrm{arccos}x)\\ \mathrm{arccos}xJ_\alpha ^{}(n\mathrm{arccos}x)\end{array}\right)=\left(\begin{array}{c}O\left((1x)^{\alpha /2}\right)\\ O\left((1x)^{\alpha /2}\right)\end{array}\right),\text{as }x1.$$
(3.21)
Since $`M`$ is analytic near 1, see Lemma 3.3, we have $`M(x)=O(1)`$ as $`x1`$. Inserting (3.21) into (3.15) and noting that $`w(x)^1=O((1x)^\alpha )`$ as $`x1`$, we then have indeed that $`Y_{11}(x)`$ and $`Y_{21}(x)`$ remain bounded as $`x1`$.

We also need the asymptotic behavior of $`J_\alpha (\stackrel{~}{u}_n)`$ and $`J_\alpha ^{}(\stackrel{~}{u}_n)`$ as $`n\mathrm{}`$, where we put $`u_n=1\frac{u}{2n^2}`$ and $`\stackrel{~}{u}_n=n\mathrm{arccos}u_n`$. These will be contained in the next lemma.

###### Lemma 3.5

Let $`u(0,\mathrm{})`$, $`u_n=1\frac{u}{2n^2}`$, and $`\stackrel{~}{u}_n=n\mathrm{arccos}u_n`$. We then have as $`n\mathrm{}`$,

$`\stackrel{~}{u}_n=\sqrt{u}+O\left({\displaystyle \frac{u^{\frac{3}{2}}}{n^2}}\right),`$ (3.22)
$`J_\alpha (\stackrel{~}{u}_n)=J_\alpha (\sqrt{u})+O\left({\displaystyle \frac{u^{\frac{\alpha }{2}+1}}{n^2}}\right),`$ (3.23)
$`J_\alpha ^{}(\stackrel{~}{u}_n)=J_\alpha ^{}(\sqrt{u})+O\left({\displaystyle \frac{u^{\frac{\alpha }{2}+\frac{1}{2}}}{n^2}}\right).`$ (3.24)
The error terms hold uniformly for $`u`$ in bounded subsets of $`(0,\mathrm{})`$.

* Since $`z^{1/2}\mathrm{arccos}(1z)`$ is analytic in a neighborhood of $`0`$ with expansion

$$z^{1/2}\mathrm{arccos}(1z)=\sqrt{2}+O(z),\text{as }z0\text{,}$$

we easily get (3.22).

By formula 9.1.10 of  we know that $`J_\alpha (z)=z^\alpha G(z)`$ with $`G`$ an entire function. From (3.22) and Taylor’s formula we then get uniformly for $`u`$ in bounded subsets of $`(0,\mathrm{})`$,

$`J_\alpha (n\mathrm{arccos}u_n)`$ $`=`$ $`u^{\frac{\alpha }{2}}\left(1+O\left({\displaystyle \frac{u}{n^2}}\right)\right)\left(G(\sqrt{u})+O\left({\displaystyle \frac{u^{\frac{3}{2}}}{n^2}}\right)\right)`$
$`=`$ $`J_\alpha (\sqrt{u})+O\left({\displaystyle \frac{u^{\frac{\alpha }{2}+1}}{n^2}}\right),`$
which is (3.23). The proof of (3.24) follows in a similar fashion, using

$$J_\alpha ^{}(z)=\alpha z^{\alpha 1}G(z)+z^\alpha G^{}(z),$$

and Taylor’s formula again. $`\mathrm{}`$

Now we are ready for the proof of our main result.

* of Theorem 1.1 (c) Let $`u,v(0,\mathrm{})`$ and define

$$u_n=1\frac{u}{2n^2},v_n=1\frac{v}{2n^2},\stackrel{~}{u}_n=n\mathrm{arccos}u_n,\stackrel{~}{v}_n=n\mathrm{arccos}v_n.$$
(3.25)
We put

$$D_n(u,v)=\frac{1}{2n^2}K_n(u_n,v_n).$$
(3.26)
From (3.1) we then have

$`D_n(u,v)`$ $`=`$ $`{\displaystyle \frac{1}{2\pi i(uv)}}\sqrt{w(u_n)}\sqrt{w(v_n)}det\left(\begin{array}{cc}Y_{11}(u_n)& Y_{11}(v_n)\\ Y_{21}(u_n)& Y_{21}(v_n)\end{array}\right)`$ (3.27)
$`=`$ $`{\displaystyle \frac{1}{2\pi i(uv)}}det\left(\begin{array}{cc}2^n\sqrt{w(u_n)}Y_{11}(u_n)& 2^n\sqrt{w(v_n)}Y_{11}(v_n)\\ 2^n\sqrt{w(u_n)}Y_{21}(u_n)& 2^n\sqrt{w(v_n)}Y_{21}(v_n)\end{array}\right).`$
Next, we replace the two columns in the determinant in (3.27) by the expression (3.15) we found in Lemma 3.3. It follows that

$$D_n(u,v)=\frac{2\pi n}{2\pi i(uv)}det\left[M(u_n)\left(\begin{array}{cc}J_\alpha (\stackrel{~}{u}_n)& 0\\ \frac{i}{2n}\stackrel{~}{u}_nJ_\alpha ^{}(\stackrel{~}{u}_n)& 0\end{array}\right)+M(v_n)\left(\begin{array}{cc}0& J_\alpha (\stackrel{~}{v}_n)\\ 0& \frac{i}{2n}\stackrel{~}{v}_nJ_\alpha ^{}(\stackrel{~}{v}_n)\end{array}\right)\right].$$
(3.28)
We rewrite the matrix appearing in the determinant in (3.28) as

$$M(v_n)\left[\left(\begin{array}{cc}J_\alpha (\stackrel{~}{u}_n)& J_\alpha (\stackrel{~}{v}_n)\\ \frac{i}{2n}\stackrel{~}{u}_nJ_\alpha ^{}(\stackrel{~}{u}_n)& \frac{i}{2n}\stackrel{~}{v}_nJ_\alpha ^{}(\stackrel{~}{v}_n)\end{array}\right)+M(v_n)^1\left[M(u_n)M(v_n)\right]\left(\begin{array}{cc}J_\alpha (\stackrel{~}{u}_n)& 0\\ \frac{i}{2n}\stackrel{~}{u}_nJ_\alpha ^{}(\stackrel{~}{u}_n)& 0\end{array}\right)\right].$$

Now we use $`detM(v_n)=1`$ and the fact that $`M(z)`$ is uniformly bounded for $`zU_\delta `$, see Lemma 3.3, to conclude that the entries of $`M(v_n)^1`$ are uniformly bounded. By Lemma 3.3, we also have that $`\frac{d}{dz}M(z)`$ is uniformly bounded so that $`M(u_n)M(v_n)=O\left(\frac{uv}{n^2}\right)`$. From Lemma 3.5 it follows that $`J_\alpha (\stackrel{~}{u}_n)=O(u^{\alpha /2})`$ and $`\stackrel{~}{u}_nJ_\alpha ^{}(\stackrel{~}{u}_n)=O(u^{\alpha /2})`$ uniformly for $`u`$ in bounded subsets of $`(0,\mathrm{})`$ as $`n\mathrm{}`$. Hence we have, uniformly for $`u,v`$ in bounded subsets of $`(0,\mathrm{})`$,

$$M(v_n)^1\left[M(u_n)M(v_n)\right]\left(\begin{array}{cc}J_\alpha (\stackrel{~}{u}_n)& 0\\ \frac{i}{2n}\stackrel{~}{u}_nJ_\alpha ^{}(\stackrel{~}{u}_n)& 0\end{array}\right)=\left(\begin{array}{cc}O\left(\frac{uv}{n^2}u^{\frac{\alpha }{2}}\right)& 0\\ O\left(\frac{uv}{n^2}u^{\frac{\alpha }{2}}\right)& 0\end{array}\right).$$
(3.29)
It now follows that (we use $`detM(v_n)=1`$)

$$D_n(u,v)=\frac{1}{2(uv)}det\left(\begin{array}{cc}J_\alpha (\stackrel{~}{u}_n)+O\left(\frac{uv}{n^2}u^{\frac{\alpha }{2}}\right)& J_\alpha (\stackrel{~}{v}_n)\\ \stackrel{~}{u}_nJ_\alpha ^{}(\stackrel{~}{u}_n)+O\left(\frac{uv}{n}u^{\frac{\alpha }{2}}\right)& \stackrel{~}{v}_nJ_\alpha ^{}(\stackrel{~}{v}_n)\end{array}\right).$$
(3.30)
Since $`J_\alpha (\stackrel{~}{v}_n)=O(v^{\alpha /2})`$ and $`\stackrel{~}{v}_nJ_\alpha ^{}(\stackrel{~}{v}_n)=O(v^{\alpha /2})`$ as $`n\mathrm{}`$, we then get uniformly for $`u,v`$ in bounded subsets of $`(0,\mathrm{})`$,

$$D_n(u,v)=\frac{1}{2(uv)}det\left(\begin{array}{cc}J_\alpha (\stackrel{~}{u}_n)& J_\alpha (\stackrel{~}{v}_n)\\ \stackrel{~}{u}_nJ_\alpha ^{}(\stackrel{~}{u}_n)& \stackrel{~}{v}_nJ_\alpha ^{}(\stackrel{~}{v}_n)\end{array}\right)+O\left(\frac{u^{\frac{\alpha }{2}}v^{\frac{\alpha }{2}}}{n}\right).$$
(3.31)
In the determinant in (3.31) we can replace $`\stackrel{~}{u}_n`$ and $`\stackrel{~}{v}_n`$ by $`\sqrt{u}`$ and $`\sqrt{v}`$ respectively, and make an error which we could estimate using Lemma 3.5. However, this estimate would not be uniform for $`uv`$ close to zero. So we will be more careful. We bring in a factor $`u^{\frac{\alpha }{2}}`$ into the first column of the determinant in (3.31) and a factor $`v^{\frac{\alpha }{2}}`$ into the second. Then we subtract the second column from the first to obtain

$$D_n(u,v)=\frac{u^{\frac{\alpha }{2}}v^{\frac{\alpha }{2}}}{2(uv)}det\left(\begin{array}{cc}u^{\frac{\alpha }{2}}J_\alpha (\stackrel{~}{u}_n)v^{\frac{\alpha }{2}}J_\alpha (\stackrel{~}{v}_n)& v^{\frac{\alpha }{2}}J_\alpha (\stackrel{~}{v}_n)\\ u^{\frac{\alpha }{2}}\stackrel{~}{u}_nJ_\alpha ^{}(\stackrel{~}{u}_n)v^{\frac{\alpha }{2}}\stackrel{~}{v}_nJ_\alpha ^{}(\stackrel{~}{v}_n)& v^{\frac{\alpha }{2}}\stackrel{~}{v}_nJ_\alpha ^{}(\stackrel{~}{v}_n)\end{array}\right)+O\left(\frac{u^{\frac{\alpha }{2}}v^{\frac{\alpha }{2}}}{n}\right).$$
(3.32)
From Lemma 3.5 it follows that uniformly for $`x`$ in bounded subsets of $`(0,\mathrm{})`$,

$$\frac{d}{dx}\left(x^{\frac{\alpha }{2}}J_\alpha (n\mathrm{arccos}x_n)x^{\frac{\alpha }{2}}J_\alpha (\sqrt{x})\right)=O\left(\frac{1}{n^2}\right).$$

Then it easily follows that the $`1,1`$-entry in the determinant in (3.32) is equal to

$$u^{\frac{\alpha }{2}}J_\alpha (\sqrt{u})v^{\frac{\alpha }{2}}J_\alpha (\sqrt{v})+O\left(\frac{uv}{n^2}\right).$$

Similarly, if we use

$$\frac{d}{dx}\left(x^{\frac{\alpha }{2}}n\mathrm{arccos}x_nJ_\alpha ^{}(n\mathrm{arccos}x_n)x^{\frac{\alpha }{2}}\sqrt{x}J_\alpha ^{}(\sqrt{x})\right)=O\left(\frac{1}{n^2}\right),$$

we find that the $`2,1`$-entry is

$$u^{\frac{\alpha }{2}}\sqrt{u}J_\alpha ^{}(\sqrt{u})v^{\frac{\alpha }{2}}\sqrt{v}J_\alpha ^{}(\sqrt{v})+O\left(\frac{uv}{n^2}\right).$$

From Lemma 3.5 it also follows that we may replace $`\stackrel{~}{v}_n`$ by $`\sqrt{v}`$ in the second column at the expense of an error term $`O(\frac{1}{n^2})`$. Therefore, uniformly for $`u,v`$ in bounded subsets of $`(0,\mathrm{})`$,

$`D_n(u,v)={\displaystyle \frac{u^{\frac{\alpha }{2}}v^{\frac{\alpha }{2}}}{2(uv)}}`$ (3.33)
$`\times det\left(\begin{array}{cc}u^{\frac{\alpha }{2}}J_\alpha (\sqrt{u})v^{\frac{\alpha }{2}}J_\alpha (\sqrt{v})+O\left(\frac{uv}{n^2}\right)& v^{\frac{\alpha }{2}}J_\alpha (\sqrt{v})+O\left(\frac{1}{n^2}\right)\\ u^{\frac{\alpha }{2}}\sqrt{u}J_\alpha ^{}(\sqrt{u})v^{\frac{\alpha }{2}}\sqrt{v}J_\alpha ^{}(\sqrt{v})+O\left(\frac{uv}{n^2}\right)& v^{\frac{\alpha }{2}}\sqrt{v}J_\alpha ^{}(\sqrt{v})+O\left(\frac{1}{n^2}\right)\end{array}\right)`$
$`+O\left({\displaystyle \frac{u^{\frac{\alpha }{2}}v^{\frac{\alpha }{2}}}{n}}\right)`$
$`=𝕁_\alpha (u,v)+{\displaystyle \frac{u^{\frac{\alpha }{2}}v^{\frac{\alpha }{2}}}{2(uv)}}det\left(\begin{array}{cc}u^{\frac{\alpha }{2}}J_\alpha (\sqrt{u})v^{\frac{\alpha }{2}}J_\alpha (\sqrt{v})& O\left(\frac{1}{n^2}\right)\\ u^{\frac{\alpha }{2}}\sqrt{u}J_\alpha ^{}(\sqrt{u})v^{\frac{\alpha }{2}}\sqrt{v}J_\alpha ^{}(\sqrt{v})& O\left(\frac{1}{n^2}\right)\end{array}\right)`$
$`+O\left({\displaystyle \frac{u^{\frac{\alpha }{2}}v^{\frac{\alpha }{2}}}{n}}\right).`$
Since $`z^{\alpha /2}J_\alpha (\sqrt{z})`$ is an entire function we get by the mean value theorem that

$$\frac{u^{\frac{\alpha }{2}}J_\alpha (\sqrt{u})v^{\frac{\alpha }{2}}J_\alpha (\sqrt{v})}{uv}$$

is bounded for $`u,v`$ in bounded subsets of $`(0,\mathrm{})`$, and similarly, that

$$\frac{u^{\frac{\alpha }{2}}\sqrt{u}J_\alpha ^{}(\sqrt{u})v^{\frac{\alpha }{2}}\sqrt{v}J_\alpha ^{}(\sqrt{v})}{uv}$$

is bounded for $`u,v`$ in bounded subsets of $`(0,\mathrm{})`$. Therefore, we have by (3.33)

$$D_n(u,v)=𝕁_\alpha (u,v)+O\left(\frac{u^{\frac{\alpha }{2}}v^{\frac{\alpha }{2}}}{n}\right),$$

uniformly for $`u,v`$ in bounded subsets of $`(0,\mathrm{})`$, which completes the proof of part (c) of Theorem 1.1. $`\mathrm{}`$

### 3.4 Proof of Corollary 1.2

To prove Corollary 1.2 we first need two lemmas.

###### Lemma 3.6

For $`u,v(0,\mathrm{})`$ we have as $`n\mathrm{}`$,

$$\frac{1}{2n^2}K_n(1\frac{u}{2n^2},1\frac{v}{2n^2})=O\left(u^{\frac{\alpha }{2}}v^{\frac{\alpha }{2}}\right).$$
(3.34)
The error term is uniform for $`u,v`$ in bounded subsets of $`(0,\mathrm{})`$.

* Since

$$𝕁_\alpha (u,v)=\frac{u^{\frac{\alpha }{2}}v^{\frac{\alpha }{2}}}{2(uv)}det\left(\begin{array}{cc}u^{\frac{\alpha }{2}}J_\alpha (\sqrt{u})v^{\frac{\alpha }{2}}J_\alpha (\sqrt{v})& v^{\frac{\alpha }{2}}J_\alpha (\sqrt{v})\\ u^{\frac{\alpha }{2}}\sqrt{u}J_\alpha ^{}(\sqrt{u})v^{\frac{\alpha }{2}}\sqrt{v}J_\alpha ^{}(\sqrt{v})& v^{\frac{\alpha }{2}}\sqrt{v}J_\alpha ^{}(\sqrt{v})\end{array}\right),$$

and since $`v^{\alpha /2}J_\alpha (\sqrt{v})`$ and $`v^{\alpha /2}\sqrt{v}J_\alpha ^{}(\sqrt{v})`$ are entire functions of $`v`$, it is easy to see, by the discussion following (3.33), that for every $`R>0`$, there exists a constant $`c>0`$ so that

$$𝕁_\alpha (u,v)cu^{\frac{\alpha }{2}}v^{\frac{\alpha }{2}},\text{for }u,v(0,R).$$
(3.35)
Therefore, (3.34) is an immediate consequence of Theorem 1.1. $`\mathrm{}`$

In the second lemma, we let $`D_{n,s}`$ and $`𝕁_{\alpha ,s}`$ be the integral operators with kernels $`D_n(u,v)=\frac{1}{2n^2}K_n(1\frac{u}{2n^2},1\frac{v}{2n^2})`$ and $`𝕁_\alpha (u,v)`$ respectively, acting on $`L^2(0,s)`$.

###### Lemma 3.7

$`D_{n,s}`$ and $`𝕁_{\alpha ,s}`$ are positive trace class operators on $`L^2(0,s)`$.

* Let $`u_n=1\frac{u}{2n^2}`$ and $`v_n=1\frac{v}{2n^2}`$. Since

$$D_n(u,v)=\frac{1}{2n^2}\sqrt{w(u_n)}\sqrt{w(v_n)}\underset{j=0}{\overset{n1}{}}p_j(u_n)p_j(v_n),$$
(3.36)
we have that $`D_{n,s}`$ is a finite rank operator and hence a trace class operator. Now, for every $`fL^2(0,s)`$ we have by (3.36)

$$_0^s_0^sD_n(u,v)f(u)f(v)𝑑u𝑑v=\frac{1}{2n^2}\underset{j=0}{\overset{n1}{}}\left(_0^s\sqrt{w(u_n)}p_j(u_n)f(u)𝑑u\right)^20,$$
(3.37)
so that $`D_{n,s}`$ is a positive operator. Letting $`n\mathrm{}`$, we have that $`D_n(u,v)𝕁_\alpha (u,v)`$ for every $`u,v`$ by Theorem 1.1(c). By (3.34), there is a constant $`c>0`$ independent of $`n`$ so that for $`u,v(0,s]`$,

$$|D_n(u,v)f(u)f(v)|cu^{\frac{\alpha }{2}}v^{\frac{\alpha }{2}}|f(u)||f(v)|.$$

Then by the dominated convergence theorem and (3.37),

$$_0^s_0^s𝕁_\alpha (u,v)f(u)f(v)𝑑u𝑑v=\underset{n\mathrm{}}{lim}_0^s_0^sD_n(u,v)f(u)f(v)𝑑u𝑑v0,$$
(3.38)
so that $`𝕁_\alpha `$ is positive. Since $`\left|𝕁_\alpha (u,v)\right|cu^{\alpha /2}v^{\alpha /2}`$ for $`u,v(0,s]`$ we also have

$$|\mathrm{tr}𝕁_{\alpha ,s}|=\left|_0^s𝕁_\alpha (u,u)𝑑u\right|c_0^su^\alpha 𝑑u<\mathrm{},$$

which implies that $`𝕁_{\alpha ,s}`$ is a trace class operator. $`\mathrm{}`$

* of Corollary 1.2 It is well known that, see for example ,

$$P_n(1\frac{s}{2n^2},1)=\underset{j=0}{\overset{n}{}}\frac{(1)^j}{j!}_0^s\mathrm{}_0^sdet\left(D_n(u_i,u_k)\right)_{1i,kj}du_1\mathrm{}du_j.$$

For $`\alpha 0`$, we have that $`D_n(u,v)`$ is continuous on $`[0,s]\times [0,s]`$. Then it follows as in  that $`P_n(1\frac{s}{2n^2},1)`$ is equal to the Fredholm determinant

$$P_n(1\frac{s}{2n^2},1)=det(ID_{n,s}).$$
(3.39)
For $`\alpha <0`$, the integral kernel $`D_n(u,v)`$ is not continuous, but satisfies an estimate $`|D_n(u,v)|cu^{\alpha /2}v^{\alpha /2}`$ for $`u,v(0,s]`$, for some constant $`c>0`$. This is enough to establish (3.39) also in this case. By Theorem 3.4 of  we have

$$\left|det(ID_{n,s})det(I𝕁_{\alpha ,s})\right|D_{n,s}𝕁_{\alpha ,s}_{\text{1}}\mathrm{exp}\left(D_{n,s}𝕁_{\alpha ,s}_{\text{1}}+2𝕁_{\alpha ,s}_{\text{1}}+1\right),$$

where $`_{\text{1}}`$ is the trace norm in $`L^2(0,s)`$. So, in order to prove that $`det(ID_{n,s})`$ converges to $`det(I𝕁_{\alpha ,s})`$ as $`n\mathrm{}`$, it is enough to show that $`D_{n,s}`$ tends to $`𝕁_{\alpha ,s}`$ in trace norm. By Theorem 2.20 of  and the positivity of $`D_{n,s}`$ and $`𝕁_{\alpha ,s}`$, it then suffices to prove that $`D_{n,s}𝕁_{\alpha ,s}`$ weakly, and that $`\mathrm{tr}D_{n,s}\mathrm{tr}𝕁_{\alpha ,s}`$ as $`n\mathrm{}`$. So we have to prove that for $`f,gL^2(0,s)`$,

$$\underset{n\mathrm{}}{lim}_0^s_0^sD_n(u,v)f(u)g(v)𝑑u𝑑v=_0^s_0^s𝕁_\alpha (u,v)f(u)g(v)𝑑u𝑑v,$$
(3.40)
and that

$$\underset{n\mathrm{}}{lim}_0^sD_n(u,u)𝑑u=_0^s𝕁_\alpha (u,u)𝑑u.$$
(3.41)
Both (3.40) and (3.41) follow easily from the pointwise convergence $`D_n(u,v)J_\alpha (u,v)`$, the uniform bound $`|D_n(u,v)|cu^{\alpha /2}v^{\alpha /2}`$ and the dominated convergence theorem. $`\mathrm{}`$

### Acknowledgements

We thank Ken McLaughlin and Walter Van Assche for useful discussions.


format: text

text:
# Scenery from the Top: Study of the Third Generation Squarks at CERN LHC

## Abstract

In the minimal supersymmetric standard model (MSSM) properties of the third generation sfermions are important from the viewpoint of discriminating the SUSY breaking models and in the determination of the Higgs boson mass. If gluinos are copiously produced at CERN LHC, gluino decays into $`tb`$ through stop and sbottom can be studied using hadronic decays of the top quark. The kinematical endpoint of the gluino decays can be evaluated using a $`W`$ sideband method to estimate combinatorial backgrounds. This implies that fundamental parameters related to the third generation squarks can be reliably measured. The top-quark polarization dependence in the decay process may also be extracted by looking at the $`b`$ jet distribution near the kinematical endpoint.

preprint: KEK-TH-818preprint: ICRR-Reprot-486-2002-4preprint: KOBE HEP 02-01preprint: YITP-02-05

The minimal supersymmetric standard model (MSSM) is one of the promising extensions of the standard model (SM). The model requires superpartners of ordinary particles (sparticles), and LHC at CERN might confirm the existence of these new particles TDR . Since the sparticle masses are related to the SUSY breaking mechanism, measurement of the masses provides a way to probe the origin of SUSY breaking in nature.

The masses and mixings of stops ($`\stackrel{~}{t}_{1,2}`$) and sbottoms ($`\stackrel{~}{b}_{1,2}`$) are sensitive to the flavor structure of scalar masses. First, the diagonal masses in the stop and sbottom mass matrices, $`m_{\stackrel{~}{Q}_3}`$, $`m_{\stackrel{~}{t}_R}`$, and $`m_{\stackrel{~}{b}_R}`$, are predicted to be less than those of other squarks in the minimal supergravity (MSUGRA) model SUGRA  because of the Yukawa RGE running effects. Some SUSY breaking models, such as the flavor U(2) model u2model  or the decoupling solution decoupling , and the superconformal model Nelson:2000sn  for the SUSY flavor problem, have also special imprints on these parameters. The $`\stackrel{~}{t}_L`$-$`\stackrel{~}{t}_R`$-Higgs trilinear coupling $`A_t`$ at the weak scale has a large coefficient proportional to $`m_t`$, resulting in a large left-right mixing of stops; $`m_{LR}^2=m_t(A_t\mu \mathrm{cot}\beta )`$. In the MSUGRA $`A_t`$ is proportional to the universal gaugino mass $`M`$ at the GUT scale $`M_{GUT}`$, and insensitive to the $`A`$ parameter at $`M_{GUT}`$ infrared . Indeed, this is one of the robust predictions of SUSY breaking at $`M_{GUT}`$. These relation is not guaranteed if the SUSY breaking mediation scale is much smaller than $`M_{GUT}`$. It should be stressed that the stop masses and the mixing are very important parameters to predict the light Higgs mass higgsmass .

It is possible to study the stop and sbottom at LHC through the gluino ($`\stackrel{~}{g}`$) decay modes listed below.

$`\mathrm{I})`$ $`\stackrel{~}{g}b\stackrel{~}{b}_1bb\stackrel{~}{\chi }_2^0bbl^+l^{}\stackrel{~}{\chi }_1^0,`$ (1)
$`\mathrm{II})`$ $`\stackrel{~}{g}t\stackrel{~}{t}_{1}^{}{}_{}{}^{}tt\stackrel{~}{\chi }_1^0,`$ (2)
$`\mathrm{III})`$ $`\stackrel{~}{g}t\stackrel{~}{t}_{1}^{}{}_{}{}^{}tb\stackrel{~}{\chi }_1^{},`$ (3)
$`\mathrm{IV})`$ $`\stackrel{~}{g}\overline{b}\stackrel{~}{b}_1tb\stackrel{~}{\chi }_1^{}.`$ (4)
In previous literatures TDR ; Hinchliffe:1998zj , the third generation sfermions are often studied using the mode I) ($`bbl^+l^{}`$ channels). This mode is important when $`\stackrel{~}{\chi }_2^0`$ has substantial branching ratios into leptons. Measurement of the kinematical endpoints of the signal distributions tells us the sparticle masses. Unfortunately, the branching ratios could be very small, and this mode is insensitive to the stop. We tried to study the mode II) in Eqs. (4), but the result was unsuccessful because of the small branching ratio in the MSUGRA.

In this paper we try to measure the endpoints of the modes III) and IV) in Eqs. (4). The decay modes are expected to be dominant in the MSUGRA since $`\stackrel{~}{\chi }_1^{}`$ is likely to be SU(2)L gaugino-like and $`Br(\stackrel{~}{b}_1,\stackrel{~}{t}_1\stackrel{~}{\chi }_1^{})`$ could be as large as 60%. We focus on the reconstruction of hadronic decay of the top quark, because the distribution of the $`tb`$ invariant mass $`m_{tb}`$ makes a clear endpoint in this case.

The parton level distribution of the $`tb`$ final state invariant mass is expressed as a function of $`m_{\stackrel{~}{g}}`$, $`m_{\stackrel{~}{t}_1}`$, $`m_{\stackrel{~}{b}_1}`$, and chargino mass $`m_{\stackrel{~}{\chi }_1^{}}`$: $`d\mathrm{\Gamma }/dm_{tb}m_{tb}`$, and the endpoints $`M_{tb}`$ for the modes III) and IV) are written as follows;

$`M_{tb}^2(\mathrm{III})=m_t^2+{\displaystyle \frac{m_{\stackrel{~}{t}_1}^2m_{\stackrel{~}{\chi }_1^{}}^2}{2m_{\stackrel{~}{t}_1}^2}}\{(m_{\stackrel{~}{g}}^2m_{\stackrel{~}{t}_1}^2m_t^2)`$
$`+\sqrt{(m_{\stackrel{~}{g}}^2(m_{\stackrel{~}{t}_1}m_t)^2)(m_{\stackrel{~}{g}}^2(m_{\stackrel{~}{t}_1}+m_t)^2)}\},`$
$`M_{tb}^2(\mathrm{IV})=m_t^2+{\displaystyle \frac{m_{\stackrel{~}{g}}^2m_{\stackrel{~}{b}_1}^2}{2m_{\stackrel{~}{b}_1}^2}}\{(m_{\stackrel{~}{b}_1}^2m_{\stackrel{~}{\chi }_1^{}}^2+m_t^2)`$
$`+\sqrt{(m_{\stackrel{~}{b}_1}^2(m_{\stackrel{~}{\chi }_1^{}}m_t)^2)(m_{\stackrel{~}{b}_1}^2(m_{\stackrel{~}{\chi }_1^{}}+m_t)^2)}\}.`$
Note that the endpoint of the final state $`tb\stackrel{~}{\chi }_1^{}`$ should be sensitive to both $`\stackrel{~}{t}_1`$ and $`\stackrel{~}{b}_1`$.

In order to demonstrate the endpoint reconstruction, we take an MSUGRA point with the scalar mass $`m=100`$GeV, the gaugino mass $`M=300`$GeV, the $`A`$ parameter $`300`$GeV at the GUT scale, $`\mathrm{tan}\beta =10`$ and $`\mu >0`$. This corresponds to the sample point A1 in Table 1. The masses and mixings of sparticles are calculated by ISASUSY/ISASUGRA ISAJET . The Monte Carlo SUSY events are generated by PYTHIA PYTHIA  using the masses and mixings, and then passed through a fast detector simulation program for the ATLAS experiment ATLFAST . Jets are reconstructed by a cone-based algorithm with $`\mathrm{\Delta }R=0.4`$. The $`b`$ and $`\tau `$ tagging efficiencies are set to be 60% and 50%, respectively.

In our study we apply the following selection for the $`tb`$ signal; 1) $`E_T^{\mathrm{miss}}>200`$ GeV, 2) $`m_{\mathrm{eff}}>1000`$ GeV ($`m_{\mathrm{eff}}=E_T^{\mathrm{miss}}+_{all}p_T^{\mathrm{jet}}`$), 3) two and only two $`b`$-jets with $`p_T>30`$ GeV, 4) $`4n_{\mathrm{jet}}6`$ ($`n_{\mathrm{jet}},`$ number of additional jets with $`p_T>30`$ GeV and $`|\eta |<3.0`$). In addition, events with leptons are removed to reduce the background from $`t\overline{t}`$ production. At this stage the number of remaining $`t\overline{t}`$ events is rather small, about 10% of the remaining SUSY events for the point A1.

To reconstruct the hadronic decay of the top quark, we first take a jet pair consistent with a hadronic $`W`$ boson decay with a cut on the jet pair invariant mass $`m_{jj}`$; $`|m_{jj}m_W|<15`$GeV. The invariant mass of the jet pair and one of the $`b`$ jets, $`m_{bjj}`$, is then calculated. All possible combinations are tried in an event, and the combination which minimizes $`|m_{bjj}m_t|`$ is chosen. The jet combination is regarded as a top candidate if $`|m_{bjj}m_t|<30`$GeV. The energy and momentum of the jet pair are then rescaled so that $`m_{jj}=m_W`$.

The expected $`tb`$ endpoint is not clearly visible in the $`m_{tb}`$ distribution shown in Fig. 1(a). As there are 7 to 8 jets on average in a selected event, events with a fake $`W`$ boson (and a fake top quark) dominate the distribution. The distribution of the fake $`W`$ events is estimated from the events that contain jet pairs with the invariant mass in the ranges A) $`|m_{jj}(m_W15\mathrm{G}\mathrm{e}\mathrm{V})|<15`$ GeV and B) $`|m_{jj}(m_W+15\mathrm{G}\mathrm{e}\mathrm{V})|<15`$ GeV; ‘the $`W`$ sidebands’. The energy and momentum of the jet pairs are then rescaled linearly to be in the range $`|m_{jj}m_W|<15`$GeV. The fake top candidates are reconstructed from the rescaled jet pairs and $`b`$ jets in the events.

The estimated background distribution is shown in Fig. 1(b), which is obtained by averaging distributions from the sidebands A) and B). The estimation is based on an assumption that most of the jets in the events do not have significant correlation with the $`b`$ jets in the events. The estimated background distribution is subtracted from the signal distribution in Fig. 1(c). The estimated correct signal distribution (c) shows the better endpoint compared to (a). Fig. 1(d) is the same distribution as Fig. 1(c) but for the events which contain the mode III), the mode IV), or a decay mode irreducible to the mode III); $`\stackrel{~}{g}\stackrel{~}{b}_1b\stackrel{~}{t}_1(Wb)b\stackrel{~}{\chi }_1^{}(bW)`$. Note that if $`(bW)`$ has an invariant mass consistent to a top, the decay is kinematically equivalent to the mode III). Fig. 1(d) shows the expected clear edge at the right place ($`M_{tb}(\mathrm{III})=476`$GeV and $`M_{tb}(\mathrm{IV})=420`$ GeV), demonstrating that the sideband method works well. Here, the number of the generated SUSY events is $`3\times 10^6`$, which corresponds to an integrated luminosity of $`120`$fb-1. The plots do not include the SM backgrounds.

Note that the signals from the modes III) and IV) in Eqs. (4) are significant in the total selected events. We fit the total distribution shown in Fig. 1(c) by a simple fitting function, which is described as a function of the endpoint $`M_{tb}^{\mathrm{fit}}`$, the edge height $`h`$, and the smearing parameter $`\sigma `$ from the jet energy resolution. We assume that the signal distribution is sitting on a linearly-decreasing background. The $`M_{tb}^{\mathrm{fit}}`$ is compared with the weighted endpoint $`M_{tb}^\mathrm{w}`$ defined by

$$M_{tb}^\mathrm{w}=\frac{Br(\mathrm{III})M_{tb}(\mathrm{III})+Br(\mathrm{IV})M_{tb}(\mathrm{IV})}{Br(\mathrm{III})+Br(\mathrm{IV})},$$

where $`Br(\mathrm{III})`$ and $`Br(\mathrm{IV})`$ are branching ratios for the modes III) and IV), respectively. The fit is shown in Fig. 2 (a). We obtain $`M_{tb}^{\mathrm{fit}}=443.2\pm 7.4`$ GeV, which is consistent with $`M_{tb}^\mathrm{w}=459`$GeV. The $`M_{tb}^{\mathrm{fit}}`$ changes moderately when one changes the $`m_{tb}`$ range used for the fit. We choose the range so that the significance of the height $`S=h/\mathrm{\Delta }h`$ is at maximum. For the fit shown in Fig. 2(a), $`S(\mathrm{max})=196.9/15.2=13.0`$ is obtained.

In order to check the availability of the endpoint measurement, we study twelve sample points in total, including the previous point, shown in Table 1, and compare $`M_{tb}^{\mathrm{fit}}`$ and $`M_{tb}^\mathrm{w}`$. We choose two reference MSUGRA points as A1 and A2, where $`m=100`$ GeV, $`M=300`$ GeV, $`\mathrm{tan}\beta =10`$, $`\mu >0`$ and $`A=300`$ GeV. We also study points with different mass spectrums from the MSUGRA predictions; two points with gluino masses heavier than the reference points, (G1, G2), and two points with modified stop masses (T1,T2). Furthermore, we include the MSUGRA points selected from ellis  (B,C,G,I) and two non-SUGRA points E1 and E2 where the gluino decays exclusively into $`\stackrel{~}{t}_1`$. The result is summarized in Fig. 2 (b)111 For the point C where SUSY decays contain many leptons, we omit the lepton veto and reduce the $`t\overline{t}`$ background by requiring $`m_{bl}>140`$ for any lepton with $`p_T>10`$ GeV. . The error bars represent the statistical errors for $`3\times 10^6`$ SUSY events at each point, and the systematic error of the jet energy scale (1%) is not included (see Table 1 for SUSY cross sections). This plot shows an impressive linearity between the expectation and the MC fits, although $`M_{tb}^{\mathrm{fit}}`$ is systematically lower than $`M_{tb}^\mathrm{w}`$. This is reasonable since some of particles are always out of the cone to define jets. This effect may be corrected by comparing distributions with different jet definitions.

Another uncertainty may come from the jet fragmentation. If events are generated by ISAJET for the point A1, the reconstructed endpoint is smaller by 10%, and the number of events after the sideband subtraction is smaller by a factor of 1.5. The difference comes from the different jet fragmentation schemes. ISAJET radiates more soft jets for a parton, resulting in more background and smeared endpoint distribution. The event generators must be tuned carefully to extract the kinematical information from the signal distribution.

We now discuss the physics that might be studied with the $`tb`$ endpoint measurement.

We cannot determine all of the relevant mass parameters from only the $`tb`$ endpoint measurement, since the endpoint $`M_{tb}`$ depends on $`m_{\stackrel{~}{g}}`$, $`m_{\stackrel{~}{b}_1}`$, and $`m_{\stackrel{~}{t}_1}`$. Thus, the study of $`bbl^+l^{}`$ final state would be important to single out the possible $`\stackrel{~}{t}`$ and $`\stackrel{~}{b}`$ contributions to the $`tb`$ final state and to proceed to a model-independent study. For example, let us assume that errors of the endpoints $`M_{bbll}`$ and $`M_{bll}`$ are 10 GeV and that of the endpoint $`M_{b_1l}`$ is 30 GeV in the measurement. Here, $`b_1`$ is one of the two $`b`$ jets for which the invariant mass $`m_{bll}`$ is larger. When we generate $`m_{\stackrel{~}{g}}`$, $`m_{\stackrel{~}{b}_1}`$, $`\stackrel{~}{\chi }_2^0`$, and $`\stackrel{~}{\chi }_1^0`$ randomly around the reference point A1 fixing the endpoint $`M_{ll}`$ (which is expected to have a very small error), and require $`\mathrm{\Delta }\chi ^2`$ $`_i(M_iM_i^{A1})^2/\mathrm{\Delta }M_i^2`$ $`<1(9)`$ ($`i`$ runs over the possible endpoints), the deviation $`M_{tb}(\mathrm{IV})M_{tb}^{A1}(\mathrm{IV})`$ is always less than 15(45)GeV. For the point A1, $`M_{tb}(\mathrm{IV})=420`$ GeV and $`M_{tb}^\mathrm{w}460`$ GeV. The difference of $`M_{tb}(\mathrm{IV})`$ and $`M_{tb}^\mathrm{w}`$ therefore may be statistically distinguishable.

The measurement of the SUSY breaking parameters in different sectors might reveal an overall inconsistency of the SUSY breaking mediation models. The distribution of the invariant mass formed by combining the highest $`P_T`$ jet and a same-flavor and opposite-sign lepton pair ($`jll`$ channels) is sensitive to $`m_{\stackrel{~}{q}}`$ and $`m_{\stackrel{~}{l}}`$, and this may lead to the determination of $`m`$ and $`M`$ in the MSUGRA TDR ; jll . Once $`M`$ and $`m`$ are fixed, they strongly constrain the endpoint $`M_{tb}^\mathrm{w}`$ in the MSUGRA – by comparing it with the measured value, one should be able to check if $`m_{\stackrel{~}{b}}`$ or $`m_{\stackrel{~}{t}}`$ are consistent to the MSUGRA predictions.

Note also that the formulae of $`M_{tb}`$ or $`M_{bbll}`$ involve $`m_{\stackrel{~}{g}}`$, while the first generation squark mass is lower bounded by $`m_{\stackrel{~}{g}}`$ in the model. If $`m_{\stackrel{~}{g}}m_{\stackrel{~}{q}}`$ is established by combining the squark mass scale determined through a $`jll`$ analysis TDR  and the analysis of the final state involving $`b`$ jets, we can show that some new physics should occur below the scale where $`m_{\stackrel{~}{q}}^2<0`$. Note that, in the points G1 and G2 in Table 1, where the gluino mass is increased by 100 GeV from the MSUGRA predictions, $`m^2`$ becomes negative at the GUT scale. We study the $`jll`$ distributions for the point G2 in a similar way as given in Ref. jll  and find that the $`jll`$ endpoints are successfully reconstructed. Therefore the information on $`m_{\stackrel{~}{q}}`$ should be obtained in this case.

In the framework of the MSUGRA, the measurement of $`M_{tb}`$ is sensitive to the GUT scale $`A`$ parameter. This effect is large when $`M\times A<0`$ as can be seen in Fig. 3. Here we take $`Mm`$ and $`m=230`$ GeV, and vary $`A`$ so that $`|A|<3m`$. The $`M_{tb}^\mathrm{w}`$ and $`m_{\stackrel{~}{t}_1}`$ vary by 50 GeV and 150 GeV, respectively, and the changes are again detectable. Note that the $`\stackrel{~}{\chi }_2^0`$ decay into $`\stackrel{~}{l}l`$ is closed in this case, therefore the information from the $`bbl^+l^{}`$ channels is not available. If $`m=100`$ GeV and $`A`$ is varied from $`300`$ GeV to 300 GeV, $`M_{tb}^\mathrm{w}`$ and $`m_{\stackrel{~}{t}_1}`$ change by 30 GeV and 70 GeV, respectively.

Finally, we discuss top-quark polarization dependence in the gluino decay. Naturally all top quarks coming from $`\stackrel{~}{g}`$, $`\stackrel{~}{t}_1`$, and $`\stackrel{~}{b}_1`$ decays are polarized, depending on the sfermion and chargino mixings. For example, when $`\stackrel{~}{t}_1`$ is mostly right(left)-handed stop, $`t`$ of $`\stackrel{~}{g}t\stackrel{~}{t}_1^{}`$ is $`t_{R(L)}`$. This propagates into the average top-quark helicity, if the top quark is relativistic enough in the gluino rest frame. If the $`tb`$ invariant mass is close to the endpoint $`M_{tb}`$ of the mode III) in Eqs. (4), the $`t`$ and $`b`$ go to the opposite direction in the gluino rest frame. Because the bottom quark from the top-quark decay tends to go to the opposite direction to the top-quark helicity, the distribution of the invariant mass $`m_{bb}`$ for events with $`m_{tb}`$ close to $`M_{tb}(\mathrm{III})`$ depends on the average top-quark helicity. The parton-level distributions of $`m_{bb}`$ are plotted in Fig. 4 for the point A1. The solid (dotted) curve shows the distribution for the events with $`M_{tb}(\mathrm{III})50\mathrm{G}\mathrm{e}\mathrm{V}<m_{tb}<M_{tb}(\mathrm{III})`$ in the case of $`\stackrel{~}{t}_1=\stackrel{~}{t}_R`$ ($`\stackrel{~}{t}_1=\stackrel{~}{t}_L`$). One can read from Fig. 1 that roughly 200 signal events are available near the endpoint. The statistical difference between the two distributions is 3 $`\sigma `$ for 100 signal events without taking care of the background 222 For events with $`m_{tb}<M_{tb}(III)50`$ GeV the gluino decay into $`\stackrel{~}{b}_1`$ starts to contribute. .

In this paper, we try to reconstruct final states which consisting of hadronic jets at LHC. This was considered difficult due to the large combinatorial background, but is overcome by a $`W`$ sideband method developed to estimate the background. We reconstruct the $`tb`$ final state from the event containing two $`b`$ jets. The reconstructed endpoint provides us an access to the gluino and the third generation sparticle masses without relying on the decay modes including leptons. The correct reconstruction of the events also allows us to consider the top-quark polarization dependence of the distribution. This information is important to determine the radiative correction to the Higgs mass, as well as the origin of SUSY breaking.

It is a pleasure to thank Dr. Kanzaki and Mr. Toya for the construction of a simulation environment. We acknowledge ICEPP, Univ. of Tokyo, for providing us computing resources. This work is supported in part by the Grant-in-Aid for Science Research, Ministry of Education, Science and Culture, Japan (No. 12047127 for MMN).