format: text

text:
# Increased Yield of 𝑡𝑡̄𝑏𝑏̄ at Hadron Colliders in Low-Energy Supersymmetry

## Abstract

Light bottom squarks and gluinos have been invoked to explain the $`b`$ quark pair production excess at the Tevatron. We investigate the associated production of $`t\overline{t}b\overline{b}`$ at hadron colliders in this scenario, and find that the rates for this process are enhanced over the Standard Model prediction. If light gluinos exist, it may be possible to detect them at the Tevatron, and they could easily be observed at the Large Hadron Collider.

preprint:   Fermi National Accelerator Laboratory  FERMILAB-Pub-02/028-T hep-ph/0202174

The bottom quark pair production cross section measured at the Fermilab Tevatron exceeds the theoretical prediction by about a factor of two . The state of the art theoretical prediction is currently next-to-leading (NLO) order in QCD. While the NLO corrections are large, it is possible that the measured excess is due to new physics .

Berger et al.  propose a solution to this puzzle based on low-energy supersymmetry (LESS) . In particular, they proposed the existence of a light gluino, $`\stackrel{~}{g}`$, with mass $`m_{\stackrel{~}{g}}1216`$ GeV, which decays to a bottom quark and light bottom squark, $`\stackrel{~}{b}_1`$, of mass $`m_{\stackrel{~}{b}_1}25.5`$ GeV. The bottom squark is either long lived or decays hadronically. It is argued that this scenario is not yet ruled out by other experiments .

One way to test the above scenario is via observation of the final state $`t\overline{t}b\overline{b}`$. With LESS, there is the new production channel $`t\overline{t}\stackrel{~}{g}\stackrel{~}{g}`$, with the gluinos decaying immediately to $`b\stackrel{~}{b}_1`$. If the bottom squark decays hadronically, the decay products are typically merged with those of the associated $`b`$ quark jet. If instead the bottom squarks are long-lived, the event signature is still two additional bottom quarks in top quark pair production. One would expect the rate for this new channel to be larger than the Standard Model (SM) $`t\overline{t}b\overline{b}`$ rate due to the large Casimir. Looking for LESS in this manner has many nice properties. Unlike for $`\mathrm{{\rm Y}}`$  or $`B`$  meson decays, we do not need to worry about non-perturbative physics. Furthermore, the scale dependence for top quark pair associated production is much smaller than for bottom quark pair production. Finally, unlike many supersymmetric searches, with this production channel there is almost no model dependence. As the production coupling involved is that of QCD, $`\alpha _s`$, to a first approximation the only LESS model parameter which enters at leading order is the gluino mass, $`m_{\stackrel{~}{g}}`$ (inclusion of top squarks leads to additional diagrams, but these are suppressed primarily due to the additional heavy propagators). One could also choose to examine other associated production processes, such as $`Z\stackrel{~}{g}\stackrel{~}{g}`$. The larger scale uncertainty could be compensated by the comparatively larger cross section, but the processes are not purely QCD, so we do not consider them here.

We perform leading order, parton level Monte Carlo calculations of the SM $`t\overline{t}b\overline{b}`$ and LESS $`t\overline{t}\stackrel{~}{g}\stackrel{~}{g}`$ production cross sections, for gluino masses in the range $`m_{\stackrel{~}{g}}=1216`$ GeV. Cross sections are calculated for both $`p\overline{p}`$ collisions relevant to Run II of the Fermilab Tevatron, $`\sqrt{s}=2.0`$ TeV, and for $`pp`$ collisions at the CERN Large Hadron Collider (LHC), $`\sqrt{s}=14.0`$ TeV. Decay of the top quarks is included at the matrix element level, to determine the efficiency of realistic kinematic cuts that would be imposed in such a search. We apply those efficiencies to the inclusive $`t\overline{t}\stackrel{~}{g}\stackrel{~}{g}`$ rate, but we treat the gluinos as final state particles, impose a minimum transverse momentum cut on the gluinos, and assume that their decay leads to an observable hadronic jet with vertex tag from the daughter bottom quark. Kinematic cuts used at the Tevatron (LHC) are as follows:

$`p_T(j)>15(20)\mathrm{GeV},`$ $`|\eta (j)|<3.0(4.0),`$ (1)
$`p_T(b)>20(20)\mathrm{GeV},`$ $`|\eta (b)|<2.0(2.5),`$ (2)
$`p_T(l)>15(15)\mathrm{GeV},`$ $`|\eta (l)|<2.0(2.5),`$ (3)
$`/p_T>30(30)\mathrm{GeV},`$ $`\mathrm{}R_{mn}>0.4(0.4).`$ (4)
where $`m,n`$ are leptons, bottom quarks, gluinos or light jets.

Matrix elements were constructed with a LESS-modified version of madgraph . We used CTEQ5L parton distribution functions  with factorization scale $`\mu _f=m_t+m_{jj}/2`$, where $`m_{jj}`$ is the invariant mass of the extra bottom quark or gluino pair. The renormalization scale was taken to be the same, $`\mu _r=\mu _f`$. We do not consider any additional contribution from $`t\overline{t}\stackrel{~}{b}_1\overline{\stackrel{~}{b}_1}`$ production, as first the rate is much lower than for gluinos, and second as this introduces additional model dependence: whether the bottom squarks are long-lived or not; and if not, whether they have sufficient mass to decay into bottom quarks.

In the LESS scenario, we take into account the altered running of the QCD coupling, $`\alpha _s`$, which occurs due to the presence of the light gluino and bottom squark contributions to the beta function. This causes $`\alpha _s`$ to be considerably larger at the top quark mass scale. Since the cross sections are proportional to $`\alpha _s^4`$, this effect increases the signal considerably. In calculating the signal cross section, we must also consider the effect of enhanced $`\alpha _s`$ on the SM $`t\overline{t}b\overline{b}`$ rate – an increase of $`30\%`$. We fix the value of the coupling to be $`\alpha _s(m_b)=0.205`$, and use two-loop running with the bottom squark mass set to the bottom quark mass, $`m_{\stackrel{~}{b}_1}=m_b`$ . We run the coupling from low-energy data, because if light gluinos and squarks exist, then the extraction of $`\alpha _s`$ from low energy data would be unaffected by the LESS particle content. The value of $`\alpha _s`$ obtained at the $`Z`$ mass scale is 0.127–0.128 for $`m_{\stackrel{~}{g}}=1512`$ GeV. Given the uncertainty on $`\alpha _s(m_b)`$, this result is within experimental errors, as discussed in Ref. .

We calculate the $`t\overline{t}b\overline{b}`$ cross section at the Tevatron to be 4.0 fb for $`p_T(b)>20`$ GeV (applied to the additional bottom quarks only; no top decays) ***Ref  imposed the same $`p_T(b)`$ cut but did not impose a cut on the rapidity of the $`b`$ quarks.. We find a $`25\%`$ efficiency for the kinematic cuts for both the semileptonic (brancyhing ratio (BR) = $`29\%`$) and all-hadronic (BR = $`46\%`$) decay modes of the top quarks. We do not consider the all-leptonic channel (BR = $`4.7\%`$) as the rate is much less than one expected event for reasonable Run II luminosity. The LESS cross section for $`t\overline{t}\stackrel{~}{g}\stackrel{~}{g}`$ production varies from 11.2 fb at $`m_{\stackrel{~}{g}}=12`$ GeV to 8.1 fb at 16 GeV, as shown in Fig. 1. The $`t\overline{t}b\overline{b}`$ rate is 5.8 fb with LESS $`\alpha _s`$ running. Assuming that the all-hadronic mode could be used (we don’t assume top quark reconstruction is necessary), which is somewhat optimistic, and with a bottom quark vertex tagging efficiency of $`ϵ_b=50\%`$ and demanding that all four bottom quarks are tagged, then with 30 fb-1 of integrated luminosity, we estimate between 4.5 and 3.5 signal events on a SM background Calculated with SM running of the coupling. of 1.4 events. Using Poisson statistics, this corresponds to a 2.6 to 2.0 sigma effect. Alternatively, one may interpret this as the Tevatron having some capability to place $`95\%`$ c.l. limits on this scenario.

We note that this analysis is different from the planned search for $`t\overline{t}H`$, which has the same final state signature, except for lack of a mass peak in the extra bottom quark pair spectrum. The fact that this final state arising from light gluinos produces essentially identical kinematic distributions to the SM case makes our proposed channel search more difficult, but is mitigated by the much larger overall rate from light gluinos that from a Higgs boson. We speculate that this search could be improved at the Tevatron by requiring only three vertex tags, as in the analysis of Ref. . This would increase the total sample by about a factor of five, but also approximately double the SM background by fake tags from $`t\overline{t}gg`$ events. Nevertheless, 22 signal events on a background of about 15 events is an $`4.5\sigma `$ effect. We feel more thorough investigation along these lines is warranted. It may also be useful to examine additional production channels at the Tevatron, such as $`Z\stackrel{~}{g}\stackrel{~}{g}`$, which will have larger cross sections but other complications in their analysis.

The situation is much better at the LHC. We calculate the SM $`t\overline{t}b\overline{b}`$ total cross section ($`p_T(b)>20`$ GeV, no top quark decays) to be 1.9 pb, a rate copious enough to allow one to examine the cleaner all-leptonic top quark decay channel as well as the semi-leptonic channel. We find cuts efficiencies of $`30\%`$ and $`20\%`$, respectively. The $`t\overline{t}\stackrel{~}{g}\stackrel{~}{g}`$ cross section varies from 9.0 pb for 12 GeV gluinos to 7.2 pb for 16 GeV gluinos, as shown in Fig. 1. The $`t\overline{t}b\overline{b}`$ rate is 2.8 pb with LESS $`\alpha _s`$ running. Again using $`ϵ_b=50\%`$ and demanding four tags, we estimate that each experiment would observe from 15 to 12 signal events in the all-leptonic channel alone, with only 2 fb-1 of data. Against the SM background of 3.3 events, this would yield a $`5\sigma `$ effect over the entire gluino mass range considered. In the semi-leptonic channel for the same amount of data, we estimate 61 to 54 signal events on a background of 18 events, potentially resulting in better than $`12\sigma `$ observation.

One caveat is that of the long-lived bottom squark scenario: if the daughter bottom quark jets coming from the low-$`p_T`$ portion of the gluino spectrum do not have sufficient energy to be identified as jets, this analysis could suffer. To compensate we also investigated the case where all $`b`$ partons and gluinos were instead required to have $`p_T>50`$ GeV. The rates fall by somewhat more than $`50\%`$, but the all-leptonic decay channel would need only 10 fb-1 and the semi-leptonic channel only 3 fb-1 each to reflect a $`5\sigma `$ observation of light gluinos. Thus, we feel this this analysis can easily be made model independent at the LHC.

A final point to consider is the scale uncertainty of the cross sections. We estimate this by first calculating the cross sections for $`\mu _{f,r}=2\mu _{f,r}`$, $`\mu _{f,r}/2`$ and find about $`+75\%/45\%`$ variation. We also find a $`75\%`$ enhancement if we use $`\mu _f`$ as originally stated, but apply two factors of $`\alpha _s(\mu _r=\mu _f)`$ and two factors $`\alpha _s(m_{jj})`$. At LHC energies this results in an increase of about $`75\%`$ in the rate. Since our signal cross sections are typically a factor four larger than the SM background, we are safe in assuming that theoretical uncertainties on the cross section cannot be a limiting factor in this analysis.

We have presented an alternative production channel for light gluinos in a particular LESS model. It has the advantages of extremely distinctive final state signature (four bottom quarks and two $`W`$ bosons), scale uncertainties much smaller than the increase in rate due to non-SM particles, and the only model dependence is the gluino mass. The observable rate at the Tevatron, after kinematical cuts and approximate efficiencies, is unfortunately useful only to place probably $`95\%`$ c.l. limits on the LESS scenario, and even then only with large integrated luminosity. However, the LHC can make a $`5\sigma `$ observation of light gluinos with only a few months of running at planned luminosity.

###### Acknowledgements.

We want to thank Alex Kagan for useful discussions. Fermilab is operated by URA under DOE contract No. DE-AC02-76CH03000.


text:
# References

A construction of finite-dimensional faithful representation of Lie algebra

Yurii A.Neretin

The Ado theorem is a fundamental fact, which has a reputation to be a ’strange theorem’. We give its natural proof.

1. Construction of faithful representation. Consider a finite-dimensional Lie algebra $`𝔤`$. Assume that $`𝔤`$ is a semidirect product $`𝔭𝔫`$ of a subalgebra $`𝔭`$ and a nilpotent ideal $`𝔫`$. Assume that the adjoint action of $`𝔭`$ on $`𝔫`$ is faithful, i.e., for any $`z𝔭`$, there exists $`x𝔫`$ such that $`[z,x]0`$.

Consider the minimal $`k`$ such that all the commutators

$$[\mathrm{}[[x_1,x_2],x_3],\mathrm{},x_k],x_j𝔫$$

are 0.

Denote by $`U(𝔫)`$ the enveloping algebra of $`𝔫`$. The algebra $`𝔫`$ acts on $`U(𝔫)`$ by the left multiplications. The algebra $`𝔭`$ acts on $`U(𝔫)`$ by the derivations

$$d_zx_1x_2x_3\mathrm{}x_l=[z,x_1]x_2x_3\mathrm{}x_l+x_1[z,x_2]x_3\mathrm{}x_l+\mathrm{},\text{where }z𝔭.$$

This defines the action of the semidirect product $`𝔭𝔫=𝔤`$ on $`U(𝔫)`$.

Denote by $`I`$ the subspace in $`U(𝔫)`$ spanned by all the products $`x_1x_2\mathrm{}x_N`$, where $`N>k+2`$. Obviously,

1. $`I`$ is the two-side ideal in $`U(𝔫)`$.

2. Consider the linear span $`𝒜`$ of all the elements having the form $`1`$, $`x`$, $`x_1x_2U(𝔫)`$. Obviously, $`I𝒜=0`$.

3. $`I`$ is invariant with respect to the derivations $`d_z`$.

Obviously, the module $`U(𝔫)/I`$ is a finite-dimensional faithful module over $`𝔤`$.

2. The Ado theorem.

Lemma 1. Any finite-dimensional Lie algebra $`𝔮`$ admits an embedding to an algebra $`𝔤`$ such that

a) $`𝔤`$ is a semidirect product of a reductive subalgebra $`𝔭`$ and a nilpotent ideal $`𝔫`$;

b) the action of $`𝔭`$ on $`𝔫`$ is completely reducible.

Obviously, Lemma 1 implies the Ado theorem. Indeed, $`𝔤`$ admits a decomposition

$$𝔤=𝔭^{}(𝔭^{\prime \prime }𝔫)$$

where $`𝔭^{}`$, $`𝔭^{\prime \prime }`$ are reductive subalgebras and the action of $`𝔭^{\prime \prime }`$ on $`𝔫`$ is faithfull. After this, it is sufficient to apply the construction of p.1.

Remark. The Ado theorem implies Lemma 1 modulo the Chevalley construction of algebraic envelope of a Lie algebra. But Lemma 1 itself can be easily proved directly.

3. Killing lemma. Let $`𝔤`$ be a Lie algebra, let $`d`$ be its derivation. For an eigenvalue $`\lambda `$, denote by $`𝔤_\lambda `$ its root subspace $`𝔤_\lambda =_k\mathrm{ker}(d\lambda )^k`$; we have $`𝔤=𝔤_\lambda `$. As it was observed by Killing, $`x𝔤_\lambda `$, $`y𝔤_\mu `$ implies $`[x,y]𝔤_{\lambda +\mu }`$.

Thus the Lie algebra $`𝔤`$ admits the gradation by the eigenvalues of $`d`$. Consider the gradation operator $`d_s:𝔤𝔤`$ defined by $`d_sv=\lambda v`$ if $`v𝔤_\lambda `$. Obviously, $`d_s`$ is a derivation, and $`dd_s=d_sd`$. We also consider the derivation $`d_n:=dd_s`$, this operator is nilpotent (the equality $`d=d_n+d_s`$ is called the Jordan–Chevalley decomposition). Clearly,

$`\mathrm{ker}d_s\mathrm{ker}d;\mathrm{ker}d_n\mathrm{ker}d;`$ (1)
$`\mathrm{im}d_s\mathrm{im}d_s;\mathrm{im}d_n\mathrm{im}d_s`$ (2)
4. Elementary expansions. Let $`𝔮`$ be a Lie algebra, let $`I`$ be an ideal of codimension 1. Let $`xI`$. Denote by $`d`$ the operator $`\mathrm{Ad}_x:II`$. Consider the corresponding pair of derivations $`d_s`$, $`d_n`$. Consider the space

$$𝔮^{}=y+z+I$$

where $`y`$, $`z`$ are formal vectors. We equip this space with a structure of a Lie algebra by the rule

$$[y,z]=0,[y,u]=d_su,[z,u]=d_nu,\text{for all }uI$$

and the commutator of $`u,vI`$ is the same as it was in $`I`$.

The subalgebra $`(y+z)I𝔮^{}`$ is isomorphic $`𝔮`$. We say that $`𝔮^{}`$ is an elementary expansion of $`𝔮`$.

Obviously, $`[𝔮^{},𝔮^{}]=[𝔮,𝔮]`$.

For a general Lie algebra, the required embedding to a semidirect product can be obtained by a sequence of elementary expansions.

5. Proof of Lemma 1. Let $`𝔮`$ be a Lie algebra. Let $`𝔥`$ be its Levi part, and $`𝔯`$ be the radical. Denote by $`𝔪`$ the nilradical of $`𝔮`$, i.e., $`𝔪=[𝔮,𝔯]`$; recall that $`𝔪`$ is a nilpotent ideal, and $`[𝔮,𝔮]=𝔥𝔪`$ (see , 1.4.9).

Consider a nilpotent ideal $`𝔫`$ of $`𝔮`$ containing the nilradical $`𝔪`$. Consider a subalgebra $`𝔭𝔥`$ such that the adjoint action of $`𝔭`$ on $`𝔮`$ is completely reducible and $`𝔭𝔫=0`$; for instance, the can choice $`𝔫=𝔪`$, $`𝔭=𝔥`$.

Obviously, the $`𝔮`$-module $`𝔮/(𝔭𝔫)`$ is trivial. Consider any subspace $`I`$ of codimension 1 containing $`𝔭𝔫`$, obviously $`I`$ is an ideal in $`𝔮`$. Since the action of $`𝔭`$ on $`𝔮`$ is completely reducible, there exists a $`𝔭`$-invariant complementary subspace for $`I`$. Let $`x`$ be an element of this subspace. Since the $`𝔭`$-module $`𝔮/I`$ is trivial, $`x`$ commutes with $`𝔭`$. We apply the elementary expansion to these data.

We obtain the new algebra $`𝔮^{}=y+z+I`$ with the nilpotent ideal $`𝔫^{}=z+𝔫`$ and with the reductive subagebra $`𝔭^{}=y𝔭`$ (by (1), $`y`$ commutes with $`𝔭`$).

It remains to notice that

$$dim𝔮^{}dim𝔭^{}dim𝔫^{}=dim𝔮dim𝔭dim𝔫1$$

and we can repeat the same construction.

Addres:

Math.Physics Group, ITEP, B.Cheremushkinskaya, 25, Moscow, Russia

&

Independent University of Moscow

&

ESI, Vienna, Austria (January, 2002).

neretin@main.mccme.rssi.ru


text:
# Mirror symmetry and deformation quantization

## 1 Introduction

In homological mirror symmetry (see \[Ko1\]) and in the theory of D-modules one meets similar objects. They are pairs $`(L,\rho )`$ where $`L`$ is a Lagrangian manifold, and $`\rho `$ is a flat bundle on $`L`$ (local system). In the framework of homological mirror symmetry such pairs are objects of the so-called Fukaya category, which is the principal mathematical structure of the genus zero part of the A-model. In the framework of D-modules they are holonomic D-modules (in $`C^{\mathrm{}}`$ category). It is natural to compare the categories themselves. This comparison is the main theme of present paper.

One can object any relationship between mirror symmetry and deformation quantization. Let us mention some of possible objections.

a) Theory of D-modules (or more generally modules over the ring of micro-differential operators) “lives” on the cotangent bundle $`T^{}X`$, while the Fukaya category is defined for any symplectic manifold, hence is more general.

b) Theory of D-modules works well in algebraic or complex analytic framework, while the Fukaya category (and in general Floer theory) exists in $`C^{\mathrm{}}`$-category only.

c) In mirror symmetry one considers series in exponentially small (with respect to the symplectic structure) parameter, while in deformation quantization the parameter “is of the size” of the symplectic structure.

From our point of view these are rather problems than objections. In regard to the point a) this means that one should work out a version of the theory of microdifferential operators for arbitary symplectic manifolds. This theory is known as deformation quantization of symplectic manifolds. In regard to the point b) we remark that finding of a complex analog of the Floer theory is an interesting problem. At this time we have only speculations in this directions.

The aim of present paper is to summarize our present understanding of the topic and make some conjectures. It is a part of an ongoing project. The details will appear elsewhere. Our main conjecture can be briefly formulated such as follows:

i) there is a category of holonomic modules over the quantized algebra of smooth functions on a symplectic manifold;

ii) it is possible to change morphisms in this category by a kind of integral transformation, so that it becomes equivalent (at least locally) to the Fukaya category of the same symplectic manifold.

Let us mention two possible applications of this idea. First, it can help in constructing of an algebraic model of the Fukaya category (analogy: de Rham complex is an algebraic model of Morse complex). Second, it might help to resolve some difficulties in the definition of the Fukaya category (non-transversality of supports, non-existence of identity morphisms, etc.) In our opinion current situation is not completely satisfactory despite the recent progress (cf. for ex. \[FOOO\]). Third, one can go beyond Lagrangian submanifolds by considering coisotropic submanifolds or even non-smooth varieties(cf. \[KO\]). Finally, one hopes to achieve a deeper understanding of the relationship at the level of chiral algebras (see \[BD\],\[MSV\], \[KaV\]). Hence the subject of this paper is just the first approximation to the full picture. We hope to discuss it elsewhere.

The paper is organized as follows. In Section 2 we briefly recall basics on the deformation theory and $`A_{\mathrm{}}`$-categories. The purpose of this section is to fix the language. Details of the formalism will be explained in \[KoSo2\].

In Section 3 we recall main facts about the Fukaya category, which is one of the main structures of homological mirror symmetry. Section 4 is a reminder on deformation quantization of symplectic manifolds. Section 5 is devoted to the comparison of the Fukaya category with the category of holonomic modules over the quantized algebra of functions.

Main idea of Section 3 goes back to the pioneering paper \[Ko1\]. As explained in loc. cit. homological mirror symmetry is not a statement about individual categories, but rather about families of $`A_{\mathrm{}}`$-categories over $`𝐙`$-graded formal schemes. Geometrically this structure is modelled by a “family of non-commutative differential-graded manifolds over a commutative differential-graded base”. The framework in which the latter phrase has precise meaning will be explained in \[KoSo2\]. The reader can keep in mind the example of the deformation theory of an associative algebra. In this case one has a family of associative algebras parametrized by the formal moduli space $``$ of deformations of the given algebra $`A`$. The moduli space $``$ is a formal pointed differential-graded manifold (dg-manifold for short) (see for ex. \[Ko2\], \[KoSo3\]). Then by definition $``$ is a base of a family of associative algebras $`A_\gamma ,\gamma `$. Each algebra gives rise to a “non-commutative scheme” $`Spec(A_\gamma )`$, hence one has the desired structure.

Acknowledgements. We thank to Dima Arinkin and especially to Maxim Kontsevich for useful discussions. Some of the background material is taken from the joint book of Kontsevich and second author (see \[KoSo2\]). We thank to Sergei Barannikov for comments on the paper. Y.S. thanks to the Clay Mathematics Institute for supporting him as a Fellow and to IHES for hospitality and excellent research conditions. He also thanks to the organizers of the workshop CATS-1 (Nice, November 2001) where the main ideas were reported.

## 2 Reminder on $`A_{\mathrm{}}`$-categories and deformation theory

In this section we recall some facts about homological algebra of mirror symmetry. Full details including the necessary language of non-commutative geometry will appear in \[KoSo2\]. Some of the material can be found in existing literature, for example in \[Ko2\]. We are not going to discuss in detail motivations for all of the definitions and notions below. Main purpose of this section is to fix the language.

Let $`A`$ be a free $`𝐙`$-graded $`k`$-module over a unital commutative ring $`k`$ of characteristic zero (main applications deal with the case when $`k`$ is a field).

###### Definition 1

An $`A_{\mathrm{}}`$-algebra $`A`$ over $`k`$ is given by the following data:

a) A $`𝐙`$-graded free $`k`$-module $`A`$.

b) A codifferential $`d`$ on the cofree coalgebra $`T(A[1])=_{n1}A[1]^n`$, where $`A[1]`$ denotes the graded free $`k`$-module such that $`A[1]^i=A^{i+1}`$ ($`A`$ with shifted grading).

(We recall that a codifferential means a coderivation $`d`$ of the coalgebra satisfying the condition $`d^2=0`$).

Since $`d`$ is uniquely defined on cogenerators, it gives rise to “higher multiplications” $`m_n:A^nA,n1`$ of degrees $`2n`$ satisfying a system of quadratic equations which follows from the equation $`d^2=0`$.

###### Definition 2

An $`L_{\mathrm{}}`$-algebra on $`A`$ is given by the following data:

a) A $`𝐙`$-graded vector space $`A`$.

b) A codifferential on the cofree cocommutative coalgebra $`C(A[1])=_{n1}S^n(A[1])`$, where $`S^n(V)`$ denotes the $`n`$th symmetric power in the symmetric monoidal category of $`𝐙`$-graded $`k`$-modules.

The codifferential $`d`$ defines a sequence of “higher Lie brackets” $`m_n:A^nA,n1`$ of degrees $`2n`$ satisfying a system of quadratic equations which follows from the equality $`d^2=0`$.

It is useful to have in mind geometric picture for both algebraic structures defined above. We start with $`L_{\mathrm{}}`$-algebras.

An $`L_{\mathrm{}}`$-algebra gives rise to a formal pointed $`𝐙`$-graded manifold $`X`$, which carries a vector field $`d_X`$ of degree $`+1`$ such that $`d_X`$ vanishes at the marked point, and satisfies the condition $`[d_X,d_X]=0`$. This structure is called formal pointed differential-graded manifold in \[Ko2\], \[KoSo2\], \[KoSo3\] (it was introduced by A. Schwarz under the name of $`Q`$-manifold). Algebra of formal functions of $`X`$ is isomorphic to the graded dual to the coalgebra $`C(A[1])`$. Thus we can write $`X=Spf((C[1])^{})`$, where $`Spf`$ stands for the formal spectrum.

###### Remark 1

a) One should remember that formal $`𝐙`$-graded manifolds have only nilpotent points.

b) It is useful to interpret maps $`m_n`$ as Taylor coefficient of the vector field $`d_X`$ at the marked point.

Let us consider two illustrating examples.

###### Example 1

Let $`A`$ be an associative algebra. Then its truncated Hochschild complex $`C_+^{}(A,A)[1]=_{n1}Hom_k(A^n,A)[1]`$ carries a structure of differential-graded Lie algebra, hence defines a formal pointed dg-manifold (see \[Ko2\]).

One can interpret the DGLA from the example as a DGLA of coderivations of the tensor coalgebra cogenerated by $`A[1]`$. Equivalently, it is the DGLA of vector fields on the formal pointed graded manifold $`X`$ vanishing at the marked point. Then the DGLA structure is the natural one on vector fields.

One also has the notion of formal differential-graded manifold, where the condition of vanishing at the marked point is dropped. Algebraically this means that we allow the condition $`m_00`$.

###### Example 2

In the previous example we consider the full Hochschild complex $`C^{}(A,A)[1]=_{n0}Hom_k(A^n,A)[1]`$. It gives rise to a formal dg-manifold.

Importance of $`L_{\mathrm{}}`$-algebras and formal dg-manifolds in deformation theory is based on the fact that they define deformation functors (see for ex. \[Ko1\], \[KoSo2\]). Let us briefly recall the construction. If $`g=_{n0}g^n`$ is an $`L_{\mathrm{}}`$-algebra ($`g^n`$ is the $`n`$th graded component) then one has the deformation functor from commutative nilpotent algebras (possibly graded) to groupoids. Namely to a commutative nilpotent ring $`R`$ one assigns the groupoid $`Def_g(R)`$ consisting for $`\gamma g^1R`$ which satisfy the Maurer-Cartan equation

$$m_1(\gamma )+m_2(\gamma ,\gamma )/2!+\mathrm{}+m_n(\gamma ,\mathrm{},\gamma )/n!+\mathrm{}=0,$$

where $`m_n`$ are the higher Lie brackets.

Formal deformations of many algebraic and geometric structures give rise to formal pointed dg-manifolds. Formal dg-manifolds without marked points arise, for example, when one deforms categories (i.e. when objects of a category are deformed). In Example 2 the corresponding dg-manifold controls deformations of the category with one object $`X`$ such that $`Hom(X,X)=A`$.

One can develop a similar geometric language for $`A_{\mathrm{}}`$-algebras. Namely, an $`A_{\mathrm{}}`$-algebra gives rise to a non-commutative formal pointed dg-manifold. It is modelled by the cofree coalgebra $`T(A[1])`$ which carries a codifferential $`d_X`$. One also has the notion of non-commutative formal dg-manifold (no marked point is specified). In this case one can have a non-zero map $`m_0:kA`$. Geometrically this structure corresponds to a vector field $`d_X`$ of degree $`+1`$ such that $`[d_X,d_X]=0`$, without any condition at the marked point. The corresponding algebraic structure is defined in the following way.

###### Definition 3

We say that a codifferential on the coalgebra $`kT(A[1])`$ defines a structure of generalized $`A_{\mathrm{}}`$-algebra on $`A`$.

The notion of a small $`A_{\mathrm{}}`$-category is a natural generalization of the notion of $`A_{\mathrm{}}`$-algebra. Traditionally, such a category is defined by a set of objects $`Ob(𝒞)`$, $`𝐙`$-graded free $`k`$-modules of morphisms $`Hom(X,Y)`$, and structures of $`A_{\mathrm{}}`$-algebras on the spaces $`_{0i,jn}Hom(X_i,X_j)`$ for any collection of objects $`X_0,\mathrm{},X_n,n1`$. These structures are given by the higher compositions

$$m_n:_{0in}Hom(X_i,X_{i+1})Hom(X_0,X_n),$$

which are maps of $`𝐙`$-graded free $`k`$-modules of degrees $`2n`$ satisfying quadratic relations similar to those for $`A_{\mathrm{}}`$-algebras. The structures of $`A_{\mathrm{}}`$-algebras are compatible with inclusions of collections of objects.

One can think of an $`A_{\mathrm{}}`$-category $`𝒞`$ as of the large $`A_{\mathrm{}}`$-algebra $`End(_{XOb(𝒞)}X)`$ (compare with the relation between additive categories and associative algebras). The Hochschild complex and Hochschild cohomology of an $`A_{\mathrm{}}`$-category can be defined in terms of this $`A_{\mathrm{}}`$-algebra.

For any object $`X`$ the $`k`$-module $`End(X)=Hom(X,X)`$ is an $`A_{\mathrm{}}`$-algebra. Its truncated Hochschild complex gives rise to a formal pointed dg-manifold $`_X`$. Then the formal dg-manifold $`=_{XOb(𝒞)}_X`$ “controls” $`A_{\mathrm{}}`$-deformations of the category $`𝒞`$ with the fixed set of objects.

Replacing in the above discussion the truncated Hochschild complex by the full Hochschild complex one obtains a new structure called generalized $`A_{\mathrm{}}`$-category. Generalized $`A_{\mathrm{}}`$-categories do not have a fixed set of objects. This is due to the fact that now for the $`A_{\mathrm{}}`$-algebra $`End(X)`$ one can have $`m_00`$. As before, one can derive the formal dg-manifold $``$ (now using the generalized $`A_{\mathrm{}}`$-algebras $`End(X)`$). For a commutative nilpotent $`k`$-algebra $`R`$ one can consider $`k`$-points $`(R)`$. If $`_0`$ is the subset of zeros of the odd vector field $`d_{}`$ then one can speak about objects of some $`A_{\mathrm{}}`$-category. The objects are parametrized by $`_0`$. We omit here the description which can be given in terms of $`R`$-points of $`_0`$ and $``$.

We will keep the name of generalized $`A_{\mathrm{}}`$-category for a slightly more general structure. The point is that we allow the higher compositions $`m_n`$ be defined not for all collections of objects, but only for some of them (transversal collections). The Fukaya category discussed in the next section will be this kind of generalized $`A_{\mathrm{}}`$-category.

We summarize without details the data defining a generalized $`A_{\mathrm{}}`$-category in the following way:

a) We are a given a formal dg-manifold $`Ob(𝒞)=`$.

b) For any $`n1`$ we are given a formal dg-submanifold $`_{tr}^n^n`$ called the space of transversal $`n`$-families. It is assumed that $`_{tr}^1`$ (i.e. every object is transversal to itself).

c) For a point $`(X,Y)_{tr}^2`$ we are given a $`𝐙`$-graded free $`k`$-module $`Hom_𝒞(X,Y)`$ called a space of morphisms between $`X`$ and $`Y`$. These $`k`$-modules are organized in a formal dg-bundle $`Hom^𝒞_{tr}^2`$.

d) For $`n1`$ and any $`(X_0,\mathrm{},X_n)_{tr}^n`$ we are given a higher composition map $`m_n^{(X_0,\mathrm{},X_n)}`$ which gives rise to a morphism of the obvious pullbacks of $`Hom^𝒞`$ to $`_{tr}^n`$. The composition maps give rise to a structure of generalized $`A_{\mathrm{}}`$-algebra $`A(X_0,\mathrm{},X_n)`$, or, equivalently to a non-commutative formal dg-manifold $`(X_0,\mathrm{},X_n)`$.

e) Let $`(𝒞)`$ be the inductive limit of $`(X_0,\mathrm{},X_n)`$ taken over increasing collections of transversal objects. This is a non-commutative formal dg-ind-scheme (it can be properly defined as an inductive limit in the appropriate category).

f) Finally $`(𝒞)`$ is a non-commutative formal dg-manifold over the commutative scheme $`Spec(k)`$.

The structure defined in a)-e) is called generalized $`A_{\mathrm{}}`$-category. It gives rise to the usual $`k`$-linear $`A_{\mathrm{}}`$-category, which we will call associated to $`𝒞`$. The latter is defined by a non-commutative formal dg-ind-subscheme $`(𝒞)_0(𝒞)`$ of zeros of the odd vector field $`d_{(𝒞)}`$.

###### Remark 2

a) One can define the Hochschild complex of a generalized $`A_{\mathrm{}}`$-category. It gives rise to a formal dg-manifold. The corresponding deformation functor describes the formal deformation theory of the generalized $`A_{\mathrm{}}`$-category in the class of generalized $`A_{\mathrm{}}`$-categories. If the category has one object $`X`$ we have the full Hochschild complex of the corresponding generalized $`A_{\mathrm{}}`$-algebra $`End(X)`$.

b) We do not discuss the delicate problem of unital $`A_{\mathrm{}}`$-algebras (more generally, $`A_{\mathrm{}}`$-categories with identity morphisms). It is an interesting question because in practice identity morphisms may be defined up to a homotopy only. All this can be formulated in the language of non-commutative geometry.

There is a notion of $`A_{\mathrm{}}`$-functor between two generalized $`A_{\mathrm{}}`$-categories. $`A_{\mathrm{}}`$-functors form themselves a generalized $`A_{\mathrm{}}`$-category. Using $`A_{\mathrm{}}`$-functors one defines the notion of equivalence of $`A_{\mathrm{}}`$-categories (see \[KoSo2\]).

Let us illustrate the notion of equivalence in the case when both categories have only one object. Then we are dealing with generalized $`A_{\mathrm{}}`$-algebras, say $`A`$ and $`B`$. Assume in addition that $`m_0^A=0`$ and $`m_0^B=0`$ (i.e. we have ordinary $`A_{\mathrm{}}`$-algebras). If $`A`$ is equivalent to $`B`$ then the complexes $`(A,m_1^A)`$ and $`(B,m_1^B)`$ are quasi-isomorphic. Geometrically this means that the tangent spaces at marked points of equivalent non-commutative formal pointed dg-manifolds are quasi-isomorphic. The converse is also true (this is the $`A_{\mathrm{}}`$-version of the inverse function theorem). A generalized $`A_{\mathrm{}}`$-algebra with $`m_00`$ is equivalent to one which has $`m_00`$, and $`m_{n1}=0`$ (cf.: vector field which is non-trivial at a point is locally equivalent to a constant one). The latter observation explains why generalized $`A_{\mathrm{}}`$-categories should be studied in families rather than individually. Indeed generalized $`A_{\mathrm{}}`$-categories with $`m_00`$ are trivial in the sense that all higher compositions can be killed by an appropriate equivalence functor.

Similarly to the case of formal pointed dg-manifolds there is a theory of minimal models of non-commutative formal pointed dg-manifolds. For such a theory one needs to assume that the ground ring $`k`$ is a field of characteristic zero (there is more complicated theory for non-pointed dg-manifolds).

If two $`A_{\mathrm{}}`$-categories are equivalent then the formal pointed dg-manifolds of their deformations are quasi-isomorphic (i.e. tangent complexes at the marked points are quasi-isomorphic).

Finally, there is a theory of generalized $`A_{\mathrm{}}`$-categories over a formal dg-base. In the above discussion we discussed the case when the base was an ordinary scheme $`Spec(k)`$. We can also assume that the base is a formal scheme. As we will see in the next section the latter case is important in symplectic geometry.

## 3 Fukaya category

### 3.1 Fukaya category and non-commutative geometry

The Fukaya category $`F(X)`$ of a smooth symplectic manifold $`X`$ is a generalized $`A_{\mathrm{}}`$-category over a base. It can be constructed as an $`A_{\mathrm{}}`$-deformation of the following trivial $`𝐂`$-linear category $`F_0(X)`$. Objects of $`F_0(X)`$ are pairs $`(L,\rho )`$, where $`L`$ is a Lagrangian submanifold of $`X`$ and $`\rho `$ is a local system on $`L`$. Transversal collections of objects correspond to transversal collections of Lagrangian submanifolds (in fact we need more sofisticated transversality condition, see \[KoSo1\]). We set $`Hom_{F_0(X)}((L_0,\rho _0),(L_1,\rho _1))=_{xL_0L_1}Hom(\rho _{0x},\rho _{1x})`$ and $`Hom_{F_0(X)}((L,\rho ),(L,\rho ))=\mathrm{\Omega }^{}(L,End(\rho ))`$. All compositions $`m_n,n1`$ are trivial for collections of different objects. Otherwise $`m_{n3}=0`$, and $`m_2`$ is the natural product on the differential-graded algebra $`\mathrm{\Omega }^{}(L,End(\rho ))`$.

To a pair $`(L,\rho )`$ one can associate a generalized $`A_{\mathrm{}}`$-algebra $`A(L,\rho )`$. Let assume for simplicity that $`\rho `$ is a trivial rank one local system. Then the corresponding generalized $`A_{\mathrm{}}`$-algebra $`A(L)`$ is generated by geometric cycles in $`L`$. Higher multiplications $`m_n(C_1,\mathrm{},C_n)`$ between generic cycles $`C_i`$ are given by a kind of quantum cohomology construction. Namely, one counts with some weight pseudo-holomorphic maps $`f:(D^2,D^2)(X,L)`$ with marked points $`x_1,\mathrm{},x_nD^2`$. Here $`D^2𝐂`$ is the standard disc. It is required that the point $`x_i`$ is mapped to $`C_i`$. The weight is $`exp(\frac{1}{ϵ}_{D^2}f^{}(\omega ))`$, where $`\omega `$ is the symplectic form on $`X`$, and $`ϵ`$ is a parameter. The idea of this construction of $`A(L)`$ was suggested by Kontsevich. Difficult analytic details have been worked out in \[FOOO\]. In what follows we will assume the conditions on the data imposed in \[FOOO\]. The resulting generalized $`A_{\mathrm{}}`$-algebra $`A(L)`$ is defined over the valuation ring $`𝐂_ϵ^0=\{f=_{i1}a_ie^{\lambda _i/ϵ}\}`$, where $`a_i𝐂,a_10`$, and $`\lambda _i0`$ is a monotonically increasing sequence of real numbers such that $`lim_{i+\mathrm{}}\lambda _i=+\mathrm{}`$. The valuation map is given by $`v(f)=\lambda _1`$. The corresponding valuation field $`𝐂_ϵ`$ consists of series $`f`$ as above, with $`\lambda _i𝐑`$. It is useful to notice that $`𝐂_ϵ^0`$ contains the maximal ideal $`𝐂_ϵ^{>0}`$ consisting of series with all $`\lambda _i>0`$. One observes that the composition $`m_0=0`$ modulo $`𝐂_ϵ^{>0}`$, but in general $`m_00`$.

It can be proved that the disjoint union of non-commutative dg-manifolds associated with generalized $`A_{\mathrm{}}`$-algebras $`A(L,\rho )`$ gives rise to a generalized $`A_{\mathrm{}}`$-category $`F_ϵ(X)`$ over the formal spectrum of the ring $`𝐂_ϵ^0`$. Its reduction modulo the ideal $`𝐂_ϵ^{>0}`$ is equivalent to $`F_0(X)`$. If the symplectic form $`\omega `$ satisfies certain rationality conditions then one can introduce a new parameter $`q=exp(\frac{1}{ϵ})`$, thus replacing the ring $`𝐂_ϵ^0`$ by the ring of formal series $`𝐂[[q]]`$. Then the maximal ideal is just $`q𝐂[[q]]`$, and the field $`𝐂_ϵ`$ coincides with the field of Laurent series $`𝐂((q))`$.

The Fukaya category $`F(X)`$ is defined as an $`A_{\mathrm{}}`$-category obtained from $`F_ϵ(X)`$ by restriction to zeros of the odd vector field. In particular, for an object $`A`$ of $`F(X)`$ the composition $`m_0`$ vanishes: $`m_0^A=0`$. The condition $`m_0^A=0`$ defines a “subvariety” of the non-commutative moduli space $`_{Ob(F_ϵ(X))}^{NC}`$. Objects of the Fukaya category exist only along this “subvariety”. This geometric picture explains why it is too naive to work with the Fukaya category for fixed $`ϵ`$, even if one can prove convergence of the series defining $`m_n`$ (the latter is still an open problem).

Let us assume for simplicity the above-mentioned rationality conditions of $`\omega `$. Then the Fukaya category is defined over the formal spectrum $`Spf(𝐂[[q]])`$. In fact one can extend the definition so that the base will be $`Spf(𝐂[[q]](_{i2}𝐂[t_{i,\mu }]))`$, where $`q`$ has degree zero, and $`t_{i,\mu }`$ are parameters of degrees $`2i`$ corresponding to some basis in the graded vector space of cohomology $`H^i(X,𝐂)`$. We introduce new parameters $`z,t`$ of degree zero by setting $`z=qe^t`$. Then, inverting $`z`$ we obtain a family of $`A_{\mathrm{}}`$-categories over the field $`𝐂((z))`$ parametrized by $`Spf(𝐂[[t]](_{i2}𝐂[t_{i,\mu }]))`$. This family should be thought of as formal deformation of a certain $`A_{\mathrm{}}`$-category over $`𝐂((z))`$. The tangent space to the moduli space of the formal deformations of this category is isomorphic to the cohomology $`H^{}(X,𝐂((z)))`$. The latter cohomology group is isomorphic to $`_{i0}Ext^i(Id,Id)`$, where $`Id`$ is the identity functor, and the extensions are taken in the properly defined $`A_{\mathrm{}}`$-category of endofunctors. This description is useful for the purposes of quantum cohomology (Yoneda product on functors gives rise to the quantum product on the cohomology group).

### 3.2 Conventional approach to the Fukaya category

Below we briefly recall the “naive” definition of the Fukaya category, when the composition $`m_0`$ is ignored. It is useful in some questions, for example in mirror symmetry for abelian varieties (see \[KoSo1\]). As we will discuss below, this “naive” Fukaya category is related to deformation quantization.

Objects of $`F^{naive}(X)`$ are pairs $`(L,\rho )`$, where $`LX`$ is a Lagrangian submanifold and $`\rho `$ is a local system on $`L`$. Morphisms between $`(L_0,\rho _0)`$ and $`(L_1,\rho _1)`$ are defined only if $`L_0`$ and $`L_1`$ intersect transversally. In this case $`Hom((L_0,\rho _0),(L_1,\rho _1))=_{xL_0L_1}Hom(\rho _{0x},\rho _{1x})𝐂_ϵ`$. The space of morphisms is $`𝐙`$-graded by means of Maslov index. Thus we are dealing with graded Lagrangian manifolds (cf. \[Se1\]). There are higher compositions $`m_n,n1`$, which are linear maps of degrees $`2n`$:

$$m_n:_{0in}Hom((L_i,\rho _i),(L_{i+1},\rho _{i+1}))Hom((L_0,\rho _0),(L_n,\rho _n)).$$

They are defined by means of the Floer-type construction associated with the “transversal” collection of Lagrangian submanifolds $`L_i,0in`$. It is usually said that the maps $`m_n`$ give rise to an $`A_{\mathrm{}}`$-structure on $`F^{naive}(X)`$. We refer the reader to \[KoSo1\] about the details of this definition and to \[FOOO\] about definitions of the related moduli spaces.

There are several problems with the naive definition of the Fukaya category. One of the most essential is the presence of pseudo-holomorphic discs with the boundary mapped to a Lagrangian submanifold. This amounts to non-trivial maps $`m_0:𝐂_ϵHom(X,X)`$. As a result, the axioms of $`A_{\mathrm{}}`$-category are not satisfied, and one has to work with generalized $`A_{\mathrm{}}`$-categories. It was explained in the preceeding sections (for all details see \[KoSo2\]) how a consistent theory of such can be developed in the framework of non-commutative formal geometry. One can still work with $`F^{naive}(X)`$ with understanding that it is only a part of the “true” Fukaya category $`F(X)`$.

## 4 Reminder on deformation quantization

We recall that a symplectic 2n-dimensional manifold $`(X,\omega )`$ gives rise to an abelian category $`𝒞(X)`$ of modules over a non-commutative algebra $`A(X)`$ (deformation quantization of the algebra $`C^{\mathrm{}}(X)`$ of smooth functions on $`X`$). Such a deformation quantization is non-unique. We will use the one which has the characteristic class $`[\omega ]/t`$ (see \[De\]). The algebra $`A(X)`$ is a topological algebra over the ring of formal series $`𝐂[[t]]`$. As $`𝐂[[t]]`$-module it is isomorphic to the algebra of formal series $`C^{\mathrm{}}(X)[[t]]`$. The algebra $`A(X)`$ consists of global sections of a sheaf of non-commutative algebras $`A_X`$, such that locally $`A_X`$ is isomorphic to the sheaf of $`t`$-pseudo-differential operators on $`𝐑^n`$ (the latter are locally series $`P=_{|I|0}a_I(x)(t_x)^I`$). The Poisson structure induced on $`C_X^{\mathrm{}}A_X/tA_X`$ coincides with the one given by the symplectic form.

One has a category of $`A(X)`$-modules $`M`$ such that $`M`$ is t-adically complete, flat as $`𝐂[[t]]`$-module, and $`M/tM`$ is the space of sections of a sheaf of modules over the sheaf of smooth functions $`C_X^{\mathrm{}}`$. The category of $`A_X`$-modules will be denoted by $`𝒞(X)`$. Morphisms are defined as $`𝐂[[t]]`$-linear homomorphisms of topological modules. We will keep the same notation for the related category defined over the field $`𝐂((t))`$. It is obtained from $`𝒞(X)`$ respectively $`𝒞_X`$ by adding $`t^1`$, so that modules $`V`$ and $`tV`$ become equivalent.

Let $`hol(X)`$ be a full subcategory of $`𝒞(X)`$ which consists of modules $`M`$ such that the support $`Supp(M/tM)`$ is a Lagrangian submanifold.

We will call objects of $`hol(X)`$ holonomic. The Lagrangian support $`Supp(M)`$ of a holonomic module will be sometimes called its characteristic variety of $`M`$ and denoted by $`Ch(M)`$. The category $`hol(X)`$ contains objects $`V_{(L,\rho )}`$ which correspond to pairs $`(L,\rho )`$ where $`LX`$ is a Lagrangian submanifold and $`\rho `$ is a local system on $`L`$. In what follows only such objects will be considered.

###### Remark 3

In \[KaMa\] the authors constructed (for every Lagrangian submanifold $`L`$ satisfying some topological conditions) an $`A_X`$-module $`V_L`$ such that $`Ch(V_L)=L`$. One can easily generalize their construction including local systems on $`L`$. We will call the corresponding objects Karasev-Maslov modules.

We will need symplectic manifolds $`X^n,n2`$. The corresponding symplectic forms are given by $`(\omega ,\omega ,\omega ,\mathrm{},\omega )`$.

The identity functor $`Id_{A_Xmod}`$ is represented by the $`A_{X\times X}`$-module $`K_\mathrm{\Delta }`$ supported on the diagonal $`\mathrm{\Delta }X\times X`$. It can be identified with the sheaf $`A_X`$. Deformations of $`A_Xmod`$ as an $`A_{\mathrm{}}`$-category has the tangent complex quasi-isomorphic (after a shift) to the tangent complex to the deformations of the identity functor $`Id_{A_Xmod}`$. The latter deformations are described by the deformations of $`K_\mathrm{\Delta }`$ as an object of $`A_{X\times X}mod`$. The tangent space at $`K_\mathrm{\Delta }`$ to the moduli space of its deformations is isomorphic to $`_{i0}Ext_{A_{X\times X}mod}^i(K_\mathrm{\Delta },K_\mathrm{\Delta })`$. After changing scalars to $`𝐂((t))`$ the latter sum can be identified with $`H^{}(X,𝐂((t)))`$. We can restrict the deformation functor to the subcategory $`hol(X)`$. It is not difficult to see that the support of a holonomic module remains Lagrangian. These observations lead to the following result.

###### Proposition 1

a) The Hochschild cohomology of the category $`A_Xmod`$ is isomorphic to $`H^{}(X,𝐂((t)))`$. (Here we consider $`A_Xmod`$ as an $`A_{\mathrm{}}`$-category with $`m_{n2}=0`$ and $`m_2`$ given by the usual composition of morphisms).

b) The tangent space to the deformations of an object $`(L,\rho )hol(X)`$ is isomorphic to $`H^{}(L,End(\rho ))𝐂((t))`$.

We also mention the following proposition (see \[So1\]). It should be compared with the definition of $`Hom^{}s`$ in the Fukaya category.

###### Proposition 2

Let $`V_{(L,\rho )}`$ denotes the object of $`hol_X`$ correspodning to the pair $`(L,\rho )`$. If $`L_0`$ is transversal to $`L_1`$ then

a) $`Ext^i(V_{(L_0,\rho _0)},V_{(L_1,\rho _1)})`$ is trivial if $`in`$, where $`n=1/2dimX`$;

b) $`Ext^n(V_{(L_0,\rho _0)},V_{(L_1,\rho _1)})_{xL_0L_1}Hom(\rho _{0x},\rho _{1x})𝐂((t))`$;

c) the algebra $`Ext^{}(V_{(L,\rho )},V_{(L,\rho )})`$ is isomorphic to the cohomology $`H^{}(L,End(\rho ))𝐂((t))`$ (cf. part b) of the previous Proposition).

Let $`D_{\mathrm{}}^b(hol(X))`$ be the $`A_{\mathrm{}}`$-category associated with the category $`hol(X)`$. It is in fact a dg-category. In order to construct it one choses injective resolutions of $`A_X`$-modules $`I_M`$ and $`I_N`$ of two holonomic modules $`M`$ and $`N`$. Then one defines $`Hom_{D_{\mathrm{}}^b(hol(X))}(M,N)=Hom^{}(I_M,I_N)`$. In this way one obtains an $`A_{\mathrm{}}`$-model for the derived category of the category of holonomic modules.

The discussion above shows certain similarity between $`D_{\mathrm{}}^b(hol(X))`$ and $`F^{naive}(X)`$. At the same time their deformation theories induce different products on the cohomology of $`X`$. In case of the Fukaya category it is the quantum product, while in case of $`hol(X)`$ it is the usual cup product. The reader will also notice that Maslov index is not visible in the case of $`hol(X)`$.

On the other hand we will explain in the next section that:

a) The algebras $`End_{D_{\mathrm{}}^b(hol(X))}((L,\rho ))`$ and $`End_{F^{naive}(X)}((L,\rho ))`$ are $`A_{\mathrm{}}`$-equivalent.

b) If $`L`$ and $`L^{}`$ are Hamiltonian isotopic, then we can “twist” the space $`Hom_{D_{\mathrm{}}^b(hol(X))}((L,\rho ),(L^{},\rho ^{}))`$ in such a way that it becomes quasi-isomorphic to the corresponding complex of morphisms in $`F^{naive}(X)`$.

## 5 Comparison of the categories

Let $`(L,\rho )`$ be a pair as before, i.e. $`L`$ is a Lagrangian submanifold of $`X`$ and $`\rho `$ is a local system on $`L`$. Let us denote by $`E_{(L,\rho )}`$ the correspodning object of $`F(X)`$, and by $`V_{(L,\rho )}`$ the corresponding object of $`hol(X)`$. We assume that $`m_0=0`$ in the $`A_{\mathrm{}}`$-algebra $`A(L,\rho )`$. This means that in fact we are dealing with the category $`F^{naive}(X)`$. We also assume the conditions imposed on $`L`$ in \[FOOO\]. This allows us to make necessary choices without further explanations. In particular $`L`$ is relatively spin in the sense of \[FOOO\], so the moduli spaces of pseudo-holomorphic discs are orientable. Taking the Hochschild complex of $`A(L,\rho )`$ we can construct the formal pointed dg-manifold $`_{E_{(L,\rho )}}`$ of deformations of $`E_{(L,\rho )}`$. Similarly we can start with the Lie algebra $`Hom_{D_{\mathrm{}}^b(hol(X))}(V_{(L,\rho )},V_{(L,\rho )})`$ and construct the formal pointed dg-manifold $`_{V_{(L,\rho )}}`$ of deformations of $`V_{(L,\rho )}`$.

Let us imagine that both $`F^{naive}(X)`$ and $`D_{\mathrm{}}^b(hol(X))`$ are “sheaves of $`A_{\mathrm{}}`$-categories” on the “moduli space of objects”. Let us also imagine that there is a well-defined moduli space $`Lagr_X`$ of Lagrangian submanifolds of $`X`$. Then we should have a natural projection $`\pi :_XLagr_X`$. If $`L`$ is a Lagrangian submanifold, and $`[L]Lagr_X`$ the corresponding point of the moduli space then the fiber $`\pi ^1([L])`$ consists of local systems supported on $`L`$ (or on any Lagrangian submanifold representing the same equivalence class in the moduli space). We would like to compare $`F^{naive}(X)`$ and $`hol(X)`$ in a “small neighborhood of $`[(L,\rho )]`$”. From the categorical point of view we have two $`A_{\mathrm{}}`$-categories $`𝒜`$ and $``$ with the same “space” of objects, and such that for any object $`X`$ the $`A_{\mathrm{}}`$-algebras $`End_𝒜(X)`$ and $`End_{}(X)`$ are equivalent. We would like to find a functor $`\mathrm{\Phi }:𝒜𝒜`$ such that changing morphisms in $`𝒜`$ to $`Hom_𝒜^{new}(X,Y)=Hom_𝒜(X,\mathrm{\Phi }(Y))`$ one gets a new $`A_{\mathrm{}}`$-category equivalent to $``$.

### 5.1 Main conjecture

We will denote by $`hol(L)`$ the full subcategory of $`hol(X)`$ consisting of holonomic modules with the given support $`L`$. For simplicity we will assume the rationality condition imposed on the symplectic form $`\omega `$. Hence the Fukaya category is defined over $`𝐂((q))`$ (this is not a serious restriction because one can consider deformation quantization over any pro-nilpotent algebra, in particular $`𝐂_\epsilon `$).

Our main idea can be explained such as follows. Having in mind the intuitive picture of the previous subsection we consider both $`F^{naive}(X)`$ and $`D_{\mathrm{}}^b(hol(X))`$ in a small neighborhood of a given $`[L]Lagr_X`$. For a pair $`L_1,L_2`$ sufficiently close to $`L`$ we would like to find a functor $`\mathrm{\Phi }_{L_1,L_2}:D_{\mathrm{}}^b(hol(L_2))D_{\mathrm{}}^b(hol(L_1))`$, such that $`Hom(\rho _1,\mathrm{\Phi }_{L_1,L_2}(\rho _2))`$ (morphism as objects of $`D_{\mathrm{}}^b(hol(L_1)`$)) is quasi-isomorphic to $`Hom_{F^{naive}(X)}((L_1,\rho _1),(L_2,\rho _2))`$. Such a functor should be represented by a bimodule. Let us describe all this more precisely.

Let $`M_i=V_{(L_i,\rho _i)}hol(L_i),i=1,2`$. We expect there exists a Lagrangian submanifold $`\mathrm{\Lambda }_{12}=\mathrm{\Lambda }(L_1,L_2)X\times X`$ and $`K(L_1,L_2)=K_{\mathrm{\Lambda }_{12}}hol(X\times X)`$ such that:

1) If $`L_1`$ and $`L_2`$ have a non-empty intersection then $`\mathrm{\Lambda }_{12}L_1=L_2`$. Here $`\mathrm{\Lambda }L=\pi _2(\pi _1^1(L)\mathrm{\Lambda })`$, where $`\pi _i:X\times XX,i=1,2`$ are the natural projections. In particular we assume that the restrictions of $`\pi _i,i=1,2`$ to $`\mathrm{\Lambda }`$ are coverings.

2) $`Hom_{D_{\mathrm{}}^b(hol(X))}(M_1,K(L_1,L_2)M_2)Hom_{F^{naive}(X)}((L_1,\rho _1),(L_2,\rho _2))`$, where $``$ means quasi-isomorphism of complexes, and we consider both categories over the field $`𝐂((q))`$ (i.e. $`q=t`$ in the case of $`hol(X)`$). The composition $``$ for modules is given by the formula $`KM=\pi _2(K\pi _1^{}(M))`$.

We denote by $`Hom^{new}(M_1,M_2)`$ the left hand side of 2).

3) For a generic sequence of Lagrangian submanifolds $`L_1,L_2,\mathrm{},L_n,n2`$ and holonomic modules $`M_1,\mathrm{},M_n`$ such that $`M_i=V_{(L_i,\rho _i)}`$ for all $`i`$, we expect to have an isomorphism of $`A_{X^n}`$-modules

$$K(L_1,L_2)K(L_2,L_3)\mathrm{}K(L_{n1},L_n)K(L_1,L_n).$$

Such an isomorphism defines a linear map

$$m_n^{new}:_{1in1}Hom^{new}(M_i,M_{i+1})Hom^{new}(M_1,M_n).$$

We expect the above data satisfy the following

###### Conjecture 1

(i) Higher compositions $`m_n^{new},n1`$ give rise to a structure of an $`A_{\mathrm{}}`$-category on $`D_{\mathrm{}}^b(hol(X))`$.

(ii) This category is $`A_{\mathrm{}}`$-equivalent to $`F^{naive}(X,\omega )`$ (with $`q=t`$).

### 5.2 The conjecture in the case of cotangent bundle

Let us fix a Lagrangian submanifold $`LX`$, and consider only those $`L^{}`$ which are “very close” to $`L`$. More precisely we assume that they are not only close to $`L`$ but also Hamiltonian isotopic to $`L`$. We want to “restrict” $`F^{naive}(X)`$ to this “neighborhood of $`L`$”. This means that we consider an $`A_{\mathrm{}}`$-subcategory with the objects taken from the above-mentioned subset, and morphisms same as in $`F^{naive}(X)`$. We do the same thing with $`hol(X)`$ and $`D_{\mathrm{}}^b(hol(X))`$. We would like to compare these categories in the case when $`X=T^{}Y`$ is the cotangent bundle with the standard symplectic structure (notice that a neighborhood of a Lagrangian submanifold $`L`$ can be identified by a symplectomorphism with a neighborhood of the zero section in $`T^{}L`$). We are going to consider Lagrangian submanifolds of the type $`L_i=\{(x,df_i(x))|xY\}`$. Let $`\rho _i`$ be local systems on $`L_i`$.

For a pair of such Lagrangian submanifolds we have a symplectomorphism $`\varphi :XX`$ such that $`(x,\xi )(x,\xi +df_2(x)df_1(x))`$. Clearly it maps isomorphically $`L_1`$ into $`L_2`$ .

Let us define $`\mathrm{\Lambda }=\mathrm{\Lambda }_{12}`$ as $`graph(\varphi )X\times X`$. The corresponding bimodule $`K_\mathrm{\Lambda }`$ is the quotient of $`A_XA_X^{op}`$ by the left ideal generated by the relation $`a1=1e^{\frac{1}{t}ad(f_2f_1)}(a)`$, $`aA_X`$. Here $`ad(a)(b)=abba`$ (clearly $`A_X`$ contains $`C_Y^{\mathrm{}}`$ as a subalgebra, so the ideal is well-defined).

Notice that

$`exp\frac{1}{t}(ad(f_2f_1))exp\frac{1}{t}(ad(f_3f_2))\mathrm{}exp\frac{1}{t}(ad(f_nf_{n1}))=exp\frac{1}{t}(ad(f_nf_1))`$. Hence we have an isomorphism $`K(L_1,L_2)K(L_2,L_3)\mathrm{}K(L_{n1},L_n)K(L_1,L_n)`$.

In order to check the conjecture we may assume that $`f_1=0`$. Then we observe that

$$Hom_{D_{\mathrm{}}^b(hol(X))}(\rho _1,K(L_1,L_2)\rho _2)=\mathrm{\Omega }^{}(Y,\rho _1^{}\rho _2)$$

where the RHS is the complex of de Rham forms with values in the local system. Let $`_i,i=1,2`$ denotes the flat connection on $`\rho _i,i=1,2`$. Then the differential is given by $`_1^{}1+1_2+df_2()`$. The latter complex is equivalent to the standard de Rham complex (without $`df_2`$) if one twists the sections by $`exp(\frac{1}{t}f_2)`$, i.e. $`ssexp(\frac{1}{t}f_2)`$. Then according to \[KoSo1\], Section 4 the resulting complex is quasi-isomorphic to $`Hom_{F^{naive}(X)}((L_1,\rho _1),(L_2,\rho _2))`$. We remark that in the notation of \[KoSo1\] one has $`q=exp(\frac{1}{\epsilon })`$.

###### Remark 4

The case of general symplectic manifold does not follow automatically from the results of this section. Indeed, in order to define morphisms in the Fukaya category for Lagrangian submanifolds in a small neighborhood of a given $`L`$ one has to consider pseudo-holomorphic discs which do not belong entirely to the neighborhood (we are restricted by the boundary conditions only). Nevertheless, if our main conjecture is true, one can find a family of kernels $`K(L_1,L_2)`$ which takes care about such discs.

### 5.3 Complex structure on the moduli space of holonomic modules

We observe that the tangent space to the moduli space of deformations of a module $`Mhol(X),supp(M)=L`$ is isomorphic to $`Hom_{D_{\mathrm{}}^b(hol(X))}(M,M)`$ (derived deformations of $`Hom_{hol(X)}(Id_X,Id_X))`$. There is a natural embedding $`Hom_{D_{\mathrm{}}^b(hol(L))}(M,M)Hom_{D_{\mathrm{}}^b(hol(X))}(M,M)`$, corresponding to the deformations with the fixed support $`L`$. On the other hand there is a natural projection $`Hom_{D_{\mathrm{}}^b(hol(X))}(M,M)\mathrm{\Omega }^{}(L)`$, where $`\mathrm{\Omega }^{}(L)`$ denotes the de Rham complex of $`L`$. Indeed, a deformation of the module $`M`$ induce the deformation of the support of $`M`$. The latter are controlled by differential forms on the support. We can perform computations in the derived categories. Then we have an exact sequence of the tangent spaces to the formal moduli spaces of deformations:

$$Ext_{hol(L)}^{}(M,M)Ext_{hol(X)}^{}(M,M)H_{DR}^{}(L).$$

Suppose that $`M`$ is a simple module. Then

$$RHom_{hol(L)}(M,M)R\mathrm{\Gamma }(L,R\underset{¯}{Hom}(M,M))R\mathrm{\Gamma }(L,𝐂_L)\mathrm{\Omega }^{}(L).$$

Taking first cohomology (this corresponds to “classical” tangent space) we obtain in this case an exact sequence

$$H_{DR}^1(L)Ext_{hol(X)}^1(M,M)H_{DR}^1(L).$$

This means that the tangent space to the “classical” deformations of $`M`$ inside of $`hol(X)`$ is twice as big as the tangent space to the “classical” deformations of $`L`$ inside of $`Lagr_X`$. If $`X`$ is a Calabi-Yau manifold then one hopes to obtain a complex structure on the moduli space of formal deformations of a simple holonomic module $`M`$. We expect it is isomorphic to a subvariety in the dual Calabi-Yau manifold. In order to describe the complex structure explicitely we need to identify the tangent space $`T_L(Lagr_X)`$ with the tangent space $`T_M(hol(L))`$. It is sufficient to describe the lifting of paths from $`Lagr_X`$ to $`hol(L)`$. Given a path $`L(t)Lagr_X`$ such that $`L(0)=L`$, we define a path $`M(t)hol(L),M(0)=M`$ in the following way: $`M(t)=M\rho (t)`$. Here $`\rho (t)`$ is the restriction to $`L(t)`$ of the unitary bundle over $`X`$ with a connection $``$ such that $`curv()=\omega _X`$ (it is often called the pre-quantum line bundle). Restriction of $``$ to $`L(t)`$ is a flat connection.

## 6 Conclusion

We have suggested the way to compare the Fukaya category with the category of holonomic modules over the quantized algebra of smooth functions. The idea is for a pair of Lagrangian submanifolds $`L_1,L_2`$ to find a kernel $`K(L_1,L_2)`$ which transforms local systems (or more general $`A_X`$-modules) supported on $`L_2`$ to local systems supported on $`L_1`$. We have conjectured that it is possible to make these choices in such a way that the counting of instantons (i.e. higher compositions in the Fukaya category) can be replaced by pure algebraic operation of taking homomorphisms between local systems having the same Lagrangian support. We have checked the conjecture in the simplest case. It would be interesting to check it in other cases as well as “globalize” this picture, finding kernels $`K(L_1,L_2)`$ for Lagrangian submanifolds which are not close to each other.

References

\[AP\] D. Arinkin, A. Polishchuk, Fukaya category and Fourier transform, math.AG/9811023.

\[Be\] V. Berkovich, Spectral theory and analytic geometry over non-Archimedean fields. Mathematical Surveys and Monographs, 33. Amer. Math. Soc., 1990.

\[BD\] A. Beilinson, V. Drinfeld, Chiral algebras, preprint.

\[BK\] S. Barannikov, M. Kontsevich, Frobenius Manifolds and Formality of Lie Algebras of Polyvector Fields, alg-geom/9710032.

\[De\] P. Deligne, Defortmations de l’algebre des fonctions d’une variete symplectique. Selecta Math. 1 (1995), 667-698.

\[FOOO\] K. Fukaya, Y.G.Oh, H. Ohta, K.Ono, Lagrangian intersection Floer theory-anomaly and obstruction. Preprint Kyoto University (2000).

\[Fu1\] K. Fukaya, Floer homology of Lagrangian foliation and non-commutative mirror symmetry. Preprint 98-08, Kyoto University, 1998.

\[Gi\] V. Ginzburg, Characteristic varieties and vanishing cycles, Inv. Math. 84 (1986), 327-402.

\[GLO\] V. Golyshev, V. Lunts, D. Orlov, Mirror symmetry for abelian varieties, math.AG/9812003.

\[GM\] M. Grinberg, R. Macpherson, Euler characteristics and Lagrangian intersections, in Symplectic Geometry and Topology (ed. Ya. Eliashberg), 1999, 265-294.

\[K\] A. Kapustin, D-branes in a topologically nontrivial B-field, hep-th/9909089

\[KaMa\] M. Karasev, V. Maslov, Pseudo-differential operators and canonical operator on general symplectic manifolds. Izvestia AN SSSR, 47:5 (1983), 999-1029 (in Russian).

\[KaV\], M. Kapranov, E. Vasserot, Vertex algebras and the formal loop space, math.AG/0107143.

\[KO\] A. Kapustin, D. Orlov, Remarks on A-branes, mirror symmetry and the Fukaya category, hep-th/0109098

\[Ko1\] M. Kontsevich, Homological algebra of Mirror symmetry. Proceedings of the ICM in Zurich, vol. 1, p. 120-139.

\[Ko2\] M. Kontsevich, Deformation quantization of Poisson manifolds, I, q-alg/9709040.

\[Ko3\] M. Kontsevich, Operads and Motives in Deformation Quantization, math.QA/9904055.

\[KR\] M. Kontsevich, A. Rosenberg, Non-commutative smooth spaces, math.AG/9812158.

\[KoSo1\] M. Kontsevich, Y. Soibelman, Homological mirror symmetry and torus fibrations, math.SG/0011041

\[KoSo2\] M. Kontsevich, Y. Soibelman, Deformation Theory (book, to appear)

\[KoSo3\] M. Kontsevich, Y. Soibelman, Deformations of algebras over operads and Deligne’s conjecture”, math.QA/0001151.

\[MSV\] F. Malikov, V. Schechtman, A. Vaintrob, Chiral de Rham complex, math.AG/9803041.

\[Se1\] P. Seidel, Vanishing cycles and mutations, math.SG/0007115.

\[So1\] Y. Soibelman, Quantum tori, mirror symmetry and deformation theory, math.QA/0011162 and Lett. Math.Phys. v.56 (2001).

Addresses:

P.B.: Department of Mathematics, University of Arizona,

Tucson, AZ 85721

email: bressler@hedgehog.math.arizona.edu

Y.S.: Department of Mathematics, KSU, Manhattan, KS 66506, USA

email: soibel@math.ksu.edu


text:
# Separable-entangled frontier in a bipartite harmonic system

## Abstract

We consider a statistical mixture of two identical harmonic oscillators which is characterized by four parameters, namely, the concentrations ($`x`$ and $`y`$) of diagonal and nondiagonal bipartite states, and their associated thermal-like noises ($`T/\alpha `$ and $`T`$, respectively). The fully random mixture of two spins 1/2 as well as the Einstein-Podolsky-Rosen (EPR) state are recovered as particular instances. By using the conditional nonextensive entropy as introduced by Abe and Rajagopal, we calculate the separable-entangled frontier. Although this procedure is known to provide a necessary but in general not sufficient condition for separability, it does recover, in the particular case $`x=T=0`$ ($`\alpha `$), the 1/3 exact result known as Peres’ criterion. This is an indication of reliability of the calculation of the frontier in the entire parameter space. The $`x=0`$ frontier remarkably resembles to the critical line associated with standard diluted ferromagnetism where the entangled region corresponds to the ordered one and the separable region to the paramagnetic one. The entangled region generically shrinks for increasing $`T`$ or increasing $`\alpha `$.

Quantum entanglement is a quite amazing physical phenomenon, and has attracted intensive interest in recent years due to its possible applications in quantum computation, teleportation and cryptography, as well as to its connections to quantum chaos . A nonextensive statistical mechanics  was proposed in 1988 by one of us, and is currently applied  to a variety of thermodynamically anomalous systems which, in one way or another, exhibit (multi)fractals aspects. Among these anomalous systems, a prominent place is occupied by systems including long-range interactions and Lévy distributions. So being, it is after all not surprising that this thermostatistical formalism has interesting implications  in the area of quantum entanglement and its intrinsic nonlocality, thus showing the confluence of two concepts coming from distinct physical areas.

Quantum systems can be more or less entangled, which makes relevant the discussion of whether a given system is or not separable. Separability, which we shall define in detail later on, is a crucial feature in the discussion on whether a quantum physical system is susceptible of a local realistic description with hidden variables. These issues were first discussed in 1935 by Einstein, Podolsky and Rosen (EPR)  and by Schroedinger , and since then by many others . As we have mentioned above we shall see that entropic nonextensivity provides a path through which it is possible to discuss quantum entanglement .

In this work we analyze a composite system of two harmonic oscillators with identical energy spectra. Following along the lines of Abe and Rajagopal , we use the nonextensive entropy $`S_q\frac{1Tr\rho ^q}{q1}(q;`$$`S_1`$$`=`$ $`Tr`$ $`\rho \mathrm{ln}\rho )`$, $`\rho `$ being the density matrix, in order to study the frontier between separable (quantum nonentangled) and nonseparable (quantum entangled) regions. More precisely, we determine a frontier in some parameter-space which is either the exact one or an overestimation of the separable region. We remind that the entropic arguments that are used within this approach provide, like the Peres’ partial transpose method , necessary but not sufficient conditions for separability. The mixed state we shall consider for this bipartite system involves Boltzmann-like probabilities in such a way that the bipartite spin 1/2 system is recovered as a particular case. More precisely, the 1/3 Peres’ criterion for separability will emerge as the $`T=0`$ limit of this specific mixed state. An increase of the temperature $`T`$ incorporates higher energy levels in the mixed state under consideration, which has the effect of enlarging the separable region.

In order to make this work self-contained we will present, in the following, the connection between the nonextensive statistical mechanics and quantum entanglement. Let us begin with the definition of a generalized entropy for a quantum system

$$S_q=\frac{1Tr\rho ^q}{q1}(q\mathrm{};Tr\rho =1;S_1=Tr\rho \mathrm{ln}\rho ),$$
(1)
where $`\rho `$ is the density operator. Let us assume now that $`\rho `$ is the density operator associated with a composed system $`A+B`$. Then the marginal density operators are given by $`\rho _ATr_B\rho `$ and $`\rho _BTr_A\rho `$ ($`Tr_A\rho _A=Tr_B\rho _B=1`$). The systems $`A`$ and $`B`$ are said to be uncorrelated  (or independent) if and only if

$$\rho =\rho _A\rho _B.$$
(2)
Otherwise they are said to be correlated. Two correlated systems are said to be separable (or unentangled) if and only if it is possible to write $`\rho `$ as follows:

$$\rho =\underset{i=1}{\overset{W}{}}p_i\rho _A^{(i)}\rho _B^{(i)}(p_i0i;\underset{i=1}{\overset{W}{}}p_i=1).$$
(3)
The limiting case of independency is recovered for certainty, i.e., if all $`\{p_i\}`$ vanish excepting one which equals unity. Nonseparability (or entanglement) is at the basis of the amazing phenomena mentioned before and, as already pointed, at the center of the admissibility of a local realistic description of the system in terms of hidden variables. As it is known correlation is a concept which is meaningful both classically and quantically. Entanglement, in the present sense, is meaningful only within quantum mechanics. The characterization of quantum entanglement is not necessarily simple to implement, since it might be relatively easy in a specific case to exhibit the form of Eq. (3), but it can be nontrivial to prove that it cannot be presented in that form. Consequently, along the years appreciable effort has been dedicated to the establishment of general operational criteria, preferentially in the form of necessary and sufficient conditions whenever possible. Peres pointed out a few years ago a necessary condition for separability, namely the nonnegativity of the partial transpose of the density matrix. In some simple situations (like the simple mixed state of two spin 1/2) Peres’ criterion is now known to also be a sufficient condition. But, as soon as the case is slightly more complex (e.g., $`3\times 3`$ or $`2\times 4`$ matrices) it is known now to be insufficient. Within this scenario Abe and Rajagopal recently proposed a different condition, claimed to be a necessary one, based on the nonextensive entropic form $`S_q`$ given in Eq. (1). Let us summarize the idea. The quantum version of conditional probabilities is not easy to formulate in spite of being so simple within a classical framework. The difficulties come from the fact that generically $`\rho `$ does not commute with either $`\rho _A`$ or $`\rho _B`$. Consistently, the concept of quantum conditional entropy is a sloppy one. Abe and Rajagopal suggested a manner of shortcutting this difficulty, namely through the adoption, of the following definitions of the conditional entropies $`S_q(A|B)`$ and $`S_q(B|A)`$, respectively given by

$$S_q(A|B)\frac{S_q(A+B)S_q(B)}{1+(1q)S_q(B)},$$
(4)
and

$$S_q(B|A)\frac{S_q(A+B)S_q(A)}{1+(1q)S_q(A)},$$
(5)
where $`S_q(A+B)S_q(\rho )`$, $`S_q(A)S_q(\rho _A)`$ and $`S_q(B)S_q(\rho _B)`$. Obviously, for the case of independence, i.e., when $`\rho =\rho _A\rho _B`$, these expressions lead to $`S_q(A|B)=S_q(A)`$ and $`S_q(B|A)=S_q(B)`$, known to be true also in quantum mechanics.

Both classical and quantum entropies $`S_q(A+B)`$, $`S_q(A)`$ and $`S_q(B)`$ are always nonnegative. This is not the case of the conditional entropies $`S_q(A|B)`$ and $`S_q(B|A)`$, which are always nonnegative classically, but which can be negative quantically (see also ). It is therefore natural to expect that separability implies nonnegativity of the conditional entropies for all $`q`$. This is the criterion proposed in .

Peres’ criterion seems to be less restrictive than the entropic one in the sense that it might provide a larger quantum entangled region. Although we have no proof that it cannot be the other way around, we have not encountered such an example. In several cases, including some of increasingly many spins, both criteria have produced the same result, presumably the asymptotically exact answer. Details can be found in . As a general trend, it seems that the transposed matrix criterion overestimates the separable region (i.e., underestimates the quantum entangled region) less or equally than the conditional entropy criterion does. It could well be that whenever both criteria produce the same result, this result is the exact answer .

Equations (4) and (5) have in fact been proved for a classical system , not for a quantum one. It seems however plausible  that they preserve the same form in both classical and quantum cases. Such conditional entropies enable what is a necessary condition for separability. Imposing the conditions $`S_q(A|B),S_q(B|A)>0`$ for all values of $`q`$, the particular space of parameters becomes divided into two regions: one where any conditional entropy $`S_q`$ is positive and the other one \- a domain of entangled states - where at least one conditional entropy is negative for some $`q`$. Therefore, a line (or a frontier, generally speaking) like a critical one, emerges.

Within this standpoint, it becomes clear that the Boltzmann-Gibbs-Shannon entropy ($`q=1`$) is a concept too poor for properly discussing quantum entanglement, a conclusion recently reached also by Brukner and Zeilinger from a different path. An appreciable amount of arguments are now available in the literature which connect quantum entanglement and thermodynamics .

In what follows we consider, for the two oscillators $`A`$ and $`B`$, the basis $`|n_A|m_B|n,m`$, where $`n,m=0,1,2,\mathrm{}`$. Let us define the symmetric and antisymmetric states $`|n,m^\pm \frac{1}{\sqrt{2}}(|n,m\pm |m,n)`$ for $`n>m`$. These states will play a role similar to that of the EPR state. The mixed state we are interested in is the following one:

$`\rho _{A+B}={\displaystyle \frac{1xy}{4}}\left[|0,00,0|+|1,11,1|+|1,01,0|+|0,10,1|\right]`$ (6)
$`+xa{\displaystyle \underset{n=2}{\overset{\mathrm{}}{}}}e^{2n\alpha /T}|n,nn,n|+yb{\displaystyle \underset{n=1}{\overset{\mathrm{}}{}}}{\displaystyle \underset{m=0}{\overset{n1}{}}}e^{(n+m)/T}|n,m^{}n,m^{}|,`$ (7)
where $`T0`$, $`\alpha 0`$,

$$a1/\underset{n=2}{\overset{\mathrm{}}{}}e^{2n\alpha /T}=(1e^{2\alpha /T})e^{4\alpha /T}$$
(8)
and

$$b1/\underset{n=1}{\overset{\mathrm{}}{}}\underset{m=0}{\overset{n1}{}}e^{(n+m)/T}=2(1e^{1/T})\mathrm{sinh}(1/T).$$
(9)
We easily verify that $`Tr\rho _{A+B}=1`$. Since the eigenvalues of $`\rho _{A+B}`$ must be numbers within $`[0,1]`$, we have that $`0x,y1`$ and $`x+y1`$. Parameter $`\alpha `$ measures the importance of the noise associated with the nondiagonal terms with respect to the noise associated with the diagonal ones. Parameter $`T`$ is a temperature-like measure of noise. Notice that, for $`x=y=0`$, we have

$$\rho _{A+B}=\frac{1}{4}\left[|0,00,0|+|1,11,1|+|1,01,0|+|0,10,1|\right]=\frac{1}{4}\widehat{1}_4,$$
(10)
where $`\widehat{1}_4`$ is the 4-dimensional identity matrix. In other words, this state is isomorphic to a fully random state of two spins 1/2. Notice also that, for $`T=0`$ and $`(x,y)=(0,1)`$, we have $`\rho _{A+B}=|1,0^{}1,0^{}|=\frac{1}{\sqrt{2}}\left[|1,01,0||0,10,1|\right]`$, which is isomorphic to the celebrated EPR state. The separable-entangled frontier for the $`T=x=0`$ particular case is known , namely $`y=1/3(\alpha )`$. Before we address the discussion of the full frontier in the $`(x,y,T,\alpha )`$-space, it is convenient to rewrite Eq. (7) in the following equivalent form:

$`\rho _{A+B}`$ $`=`$ $`{\displaystyle \frac{1xy}{4}}\left[|0,00,0|+|1,11,1|+|1,0^+1,0^+|\right]`$ (11)
$`+`$ $`xa{\displaystyle \underset{n=2}{\overset{\mathrm{}}{}}}e^{2n\alpha /T}|n,nn,n|+\left({\displaystyle \frac{1xy}{4}}+ybe^{1/T}\right)|1,0^{}1,0^{}|`$ (12)
$`+`$ $`yb{\displaystyle \underset{n=2}{\overset{\mathrm{}}{}}}{\displaystyle \underset{m=0}{\overset{n1}{}}}e^{(n+m)/T}|n,m^{}n,m^{}|,`$ (13)
Due to the orthonormality of all the states appearing in this expression, the calculation of $`(\rho _{A+B})^q`$ is easily carried out. It follows

$`Tr(\rho _{A+B})^q`$ $`=`$ $`3\left({\displaystyle \frac{1xy}{4}}\right)^q+{\displaystyle \frac{x^q(1e^{2\alpha /T})^q}{1e^{2q\alpha /T}}}`$ (14)
$`+`$ $`\left({\displaystyle \frac{1xy}{4}}+2y(1e^{1/T})e^{1/T}\mathrm{sinh}(1/T)\right)^q`$ (15)
$`+`$ $`{\displaystyle \frac{\left(2y(1e^{1/T})e^{1/T}\mathrm{sinh}(1/T)\right)^q(1+e^{q/T}e^{2q/T})}{2\mathrm{sinh}(q/T)(1e^{q/T})}}`$ (16)
Let us now address the calculation of $`(\rho _A)^q`$. We have

$`\rho _BTr_A\rho _{A+B}`$ $`=`$ $`\left({\displaystyle \frac{1xy}{2}}+ye^{1/T}\mathrm{sinh}(1/T)\right)|00|`$ (17)
$`+`$ $`\left({\displaystyle \frac{1xy}{2}}+y\mathrm{sinh}(1/T)(e^{1/T}e^{2/T}+e^{3/T})\right)|11|`$ (18)
$`+`$ $`{\displaystyle \underset{n=2}{\overset{\mathrm{}}{}}}\left(x(1e^{2\alpha /T})e^{2(n2)\alpha /T}+y\mathrm{sinh}(1/T)(e^{n/T}e^{2n/T}+e^{(2n+1)/T})\right)|nn|,`$ (19)
where $`|n`$ denotes the $`n^{th}`$ state of oscillator $`B`$. It now follows

$`Tr(\rho _B)^q`$ $`=`$ $`\left({\displaystyle \frac{1xy}{2}}+ye^{1/T}\mathrm{sinh}(1/T)\right)^q`$ (20)
$`+`$ $`\left({\displaystyle \frac{1xy}{2}}+y\mathrm{sinh}(1/T)(e^{1/T}e^{2/T}+e^{3/T})\right)^q`$ (21)
$`+`$ $`{\displaystyle \underset{n=2}{\overset{\mathrm{}}{}}}\left(x(1e^{2\alpha /T})e^{2(n2)\alpha /T}+y\mathrm{sinh}(1/T)(e^{n/T}e^{2n/T}+e^{(2n+1)/T})\right)^q`$ (22)
Now that we have $`Tr(\rho _{A+B})^q`$ and $`Tr(\rho _B)^q`$ as explicit functions of $`(x,y,T,\alpha )`$, we can immediately obtain $`S_q(A|B)(=S_q(B|A))`$ through the use of Eqs. (1) and (4) yielding

$$S_q(A/B)=\frac{1}{q1}\left(1\frac{Tr(\rho _{A+B})^q}{Tr(\rho _B)^q}\right).$$
(23)
$`S_q(A|B)`$ as a function of $`(x,y)`$ for typical values of $`(q,T)`$ and $`\alpha =1`$ is represented in Figs. 1 and 2. By comparing these figures, notice the nonuniform convergence at $`T=1/q=0`$.

Fig. 3 exhibits, in $`(y,q)`$ space, the line verifying $`S_q(A|B)=0`$, for particular values of $`(x,T,\alpha )`$. The physical region is the one satisfying $`0x+y1`$. Given $`x`$, for $`y`$ above this critical line and within the physical region $`y1x`$, one has $`S_q(A|B)<0`$, which signals the existence of an entangled state. Notice that for sufficiently small $`x`$, the strongest restriction to the separable region is imposed at $`q\mathrm{}`$. As $`x`$ increases, the minimum of $`y`$ occurs at a finite $`q`$, $`q_{min}`$ (the possible nonmonotonicity of $`S_q(A|B)`$ with regard to $`q`$ has already been noticed by ). Since the strongest restriction to separability corresponds to the nonnegativity of both $`S_{q_{min}}(A|B)`$ and $`S_{q_{min}}(B|A)`$, then this condition is the one which better approaches the separable-entangled frontier we are looking for and which is represented in Fig. 4. The entangled region in the $`(x,y,T)`$-space is illustrated in Fig. 5 for various values of $`(x,\alpha )`$. For $`T=0`$ and arbitrary $`\alpha `$, the system is entangled if $`(1x)/3<y1x`$, which immediately recovers the Peres’ 1/3 criterion for $`x=0`$. As $`T`$ increases, the entangled domain shrinks against the border $`x+y=1`$ and dissapears at a critical $`T`$, $`T_c`$ ($`T_c=1/\mathrm{ln}2`$ for $`x=0`$).

If the conditional entropy $`S_q(A|B)`$ is a monotonically decreasing function of $`q`$, then the strongest case corresponds to the nonnegativity of $`S_{\mathrm{}}(A|B)`$. Whenever the frontier is fully defined by $`q\mathrm{}`$, the system is entangled in the region defined by

$$y^{}<y1x\text{and}x<x^{}$$
(24)
where

$$y^{}\frac{1x}{32e^{1/T}(2+e^{1/T}2e^{2/T})}$$
(25)
and

$$x^{}\frac{\left(2(1e^{2/T})(23e^{1/T}+e^{3/T}e^{4/T})1\right)y+1}{(54e^{2\alpha /T})}$$
(26)
However, for sufficiently small $`T`$, there is a curved line (corresponding to the nonlinear portion of the frontier in Fig. 4), given by $`y=f(x)`$, which is defined from the conditional entropy for finite $`q_{min}`$ and that can be numerically determined. In this case, the entangled region is given by

$$y^{}<y1x\text{and}y>f(x).$$
(27)
An increase of the noise parameter $`T`$ (see Figs. 4 and 5), i.e., an increase of the relative weight of the high energy states of the oscillators, causes shrinking of the entangled region. This is intuitively expected. Also, interestingly enough, decreasing the parameter $`\alpha `$ (see Figs. 4 and 5), which corresponds to an increase of the noise associated with the diagonal states, causes the entangled region to enlarge. This is not surprising after all, since this noise makes the diagonal terms to become less effective than the off-diagonal ones, which makes quantum coherence to be relatively stronger. The interplay of these effects may cause an interesting re-entrance, as illustrated in Fig. 5(a) for the value $`x=0.6`$. In this situation, corresponding in fact to small values of $`\alpha `$, it is possible to loose the quantum entanglement as $`T`$ increases, and then recover it back, until definitive loss for very large $`T`$. To conclude, we believe that the present results provide quantitative insight on the influence of the external world (here represented by two different types of noise) on a quantum device.

One of us (DP) acknowledges warm hospitality at the Centro Brasileiro de Pesquisas Físicas. Also, we acknowledge partial financial support from PRONEX, CNPq and FAPERJ.

FIGURE CAPTIONS

Fig. 1. $`S_q(A|B)`$ as a function of $`(x,y)`$ for $`\alpha =1`$, $`T=0.1`$ and typical values of $`q`$.

Fig. 2. $`S_q(A|B)`$ as a function of $`(x,y)`$ for $`\alpha =1`$, typical values of $`T`$ and $`q=5`$.

Fig. 3. The line $`S_q(A|B)=0`$, in $`(y,q)`$-space, for particular values of $`x`$, $`T=0.5`$ and $`\alpha =1`$.

Fig. 4. Frontier in the $`(x,y)`$-space for typical values of $`T`$ and $`\alpha `$= 0.1 (a), 1 (b), 5 (c). The dotted lines correspond to the locus of the vertices of the frontier for any $`T`$.

Fig. 5. Frontier in the $`(x,y,T)`$-space for fixed values of $`x`$ and $`\alpha `$= 0.1 (a), 1 (b). The entire plane $`x+y=1`$ ($`T`$) belongs to the separable-entangled frontier.


text:
# Changes of the topological charge of vortices

## I Introduction

Vortices can be found both in classical and quantum physics. One can encounter vortices, for instance, in water that spins around or in the air as a ring of smoke . Quantum mechanics can be formulated in a language of hydrodynamics (see e.g.  and references therein). Such a formulation (very useful also in quantum chemistry ) provides a basis for definition of topological defects like vortices , whose features are even more striking than those that we find in classical physics. Quantum vortices appear not only in systems described by the linear Schrödinger equation. Indeed, they were experimentally observed in superfluid HeII  and a Bose-Einstein condensate (BEC) of trapped alkali atoms  which are typically described, within the mean-field approximation, by a nonlinear equation .

In hydrodynamics of ordinary non-viscous fluids, the circulation of the velocity field is conserved in time evolution due to the celebrated Helmholtz-Kelvin theorem (HKT) . This theorem is often employed in quantum mechanics . However, uncritical usage of the HKT, may lead to the incorrect conclusion that stability of vortices and vortex rings in quantum fluids is fully guaranteed by the HKT . In the present paper we discuss the basic assumptions of the HKT and show analytical examples where these assumptions can be easily broken by quantum evolution of the wave function, leading to changes of the topological charge. Difficulties in fulfilling the assumptions of the HKT have been already pointed out in conclusions of Ref. .

## II Hydrodynamical formulation of quantum mechanics

To establish connections between quantum mechanics and fluid dynamics we write the wave function in the form $`\mathrm{\Psi }(\stackrel{}{r},t)=\sqrt{\rho (\stackrel{}{r},t)}\mathrm{exp}(i\chi (\stackrel{}{r},t))`$ , (where $`\rho (\stackrel{}{r},t)`$ stands for density of a probability fluid) and define the velocity field

$$\stackrel{}{v}=\frac{\mathrm{}}{m}\stackrel{}{}\chi (\stackrel{}{r},t).$$
(1)
Provided $`\stackrel{}{v}`$ and its partial derivatives are well defined, we may rewrite the Schrödinger equation \[with a potential $`V(\stackrel{}{r},t)`$\] in the form

$$m\frac{\stackrel{}{v}}{t}+\stackrel{}{}\left(\frac{1}{2}mv^2+V\frac{\mathrm{}^2}{2m}\frac{^2\sqrt{\rho }}{\sqrt{\rho }}\right)=0,$$
(2)
$$\frac{\rho }{t}+\stackrel{}{}\left(\stackrel{}{v}\rho \right)=0.$$
(3)
Equation (3) is an ordinary continuity equation, while Eq. (2), in the limit of $`\mathrm{}0`$, becomes similar to the dynamical equation of a curl-free non-viscous fluid .

In quantum mechanics the wave function has to be single valued. To satisfy this condition one arrives at the Feynman-Onsager quantization condition  for the circulation of the velocity field $`\mathrm{\Gamma }_C`$ around any closed contour $`C`$

$$\mathrm{\Gamma }_C=_C\stackrel{}{v}𝑑\stackrel{}{l}=n\frac{2\pi \mathrm{}}{m},$$
(4)
where $`n=0,\pm 1,\pm 2,\mathrm{}`$. Due to the definition (1), we may not expect $`n0`$ unless at a certain point on a surface encircled by the contour $`C`$, $`\stackrel{}{}\times \stackrel{}{v}`$ becomes singular (behaves as a Dirac delta function, see for example ). We refer to $`n`$ as a topological charge since any continuous deformation of the contour $`C`$, which does not incorporate any other points where curl of the velocity field does not vanish, can not change $`\mathrm{\Gamma }_C`$.

## III The Helmholtz-Kelvin theorem

One may ask whether the circulation $`\mathrm{\Gamma }_C`$ is conserved in time evolution of the velocity field. For a fixed contour, a possibly moving vortex may leave an area bounded by the contour and consequently $`\mathrm{\Gamma }_C`$ changes its value. Thus, the relevant situation we should consider corresponds to the case where we let an initially defined contour evolve in time according to the velocity field.

Parameterizing the initial contour by $`\xi `$ variable, i.e. $`C(t_0)=\{\stackrel{}{r}(\xi ,t_0)\}`$, and employing the equation of motion of a contour

$$\frac{d}{dt}\stackrel{}{r}(\xi ,t)=\stackrel{}{v}[\stackrel{}{r}(\xi ,t)],$$
(5)
yields

$$\frac{d}{dt}\mathrm{\Gamma }_{C(t)}=_{C(t)}\left(\frac{\stackrel{}{v}}{t}+(\stackrel{}{v}\stackrel{}{})\stackrel{}{v}+\frac{1}{2}\stackrel{}{}v^2\right)𝑑\stackrel{}{l}.$$
(6)
If the contour is drawn through points where the velocity field $`\stackrel{}{v}`$ and its partial derivatives are well defined, applying: Eq. (2), equality $`(\stackrel{}{v}\stackrel{}{})\stackrel{}{v}=\frac{1}{2}\stackrel{}{}v^2\stackrel{}{v}\times (\stackrel{}{}\times \stackrel{}{v})`$, and using the fact that, at least on the contour $`\stackrel{}{}\times \stackrel{}{v}=0`$, we get

$$\frac{d}{dt}\mathrm{\Gamma }_{C(t)}=0.$$
(7)
Equation (7) establishes the Helmholtz-Kelvin theorem originally proved for a non-viscous fluid . Applicability of this theorem to a non-linear quantum systems is presented in the Appendix.

The HKT is fulfilled provided the contour $`C(t)`$ that goes initially through points where the velocity field is well defined will evolve in time through such points only. In the following we will see how such an assumption may be broken by quantum evolution of the wave function leading to a change of the topological charge.

## IV Analytical examples

In this section we discuss a few possibilities of breaking of the HKT. To this end we analyze three different analytically solvable examples where breaking of the HKT becomes evident.

First example shows that changes of the topological charge of a vortex correspond to the appearance of a nodal line of the wave function. In Ref.  the authors show the instability of a vortex placed in an anisotropic two-dimensional harmonic trap. We would like to discuss the source of violation of the HKT in such a case. We will calculate the velocity field and analyze changes of its circulation from a point of view of the HKT.

Consider an anisotropic two-dimensional harmonic oscillator with the potential

$$V(x,y)=\frac{1}{2}x^2+\frac{\lambda ^2}{2}y^2,$$
(8)
(where we use the units of the harmonic oscillator corresponding to the $`x`$-direction). The initial wave function is prepared as a superposition of the two lowest excited states

$$\mathrm{\Psi }(x,y)(x+i\alpha y)e^{(x^2+\lambda y^2)/2},$$
(9)
with a real parameter $`\alpha >0`$. Choosing any contour $`C`$ that encircles the origin of the coordinate frame we find out that the circulation $`\mathrm{\Gamma }_C`$ corresponds to $`n=1`$ vortex. However, the circulation around that point is not conserved in time. Indeed, analysis of time evolution of the velocity field

$$\stackrel{}{v}(x,y,t)=\frac{\alpha \mathrm{cos}(Et)}{x^2+y^2\alpha ^2+2\alpha xy\mathrm{sin}(Et)}(y\stackrel{}{e}_x+x\stackrel{}{e}_y),$$
(10)
(where $`E=\lambda 1`$ is the energy difference of the two lowest excited states) reveals that for $`Et[0,\pi /2)`$, $`\mathrm{\Gamma }_C`$ corresponds to $`n=1`$ while for $`Et(\pi /2,\pi ]`$ to $`n=1`$.

To discuss applicability of the HKT we have to investigate time evolution of the initially defined contour. The problem requires careful treatment because for $`Et=\pi /2\epsilon `$ (where $`\epsilon 0^+`$) the velocity field goes to zero everywhere except on the nodal line, $`x+\alpha y=0`$, where it diverges $`\stackrel{}{v}\frac{1}{\epsilon x}(\stackrel{}{e}_x+\alpha \stackrel{}{e}_y)`$. To demonstrate violation of the HKT we have to show that a contour initially encircling the center is not extended to infinity but indeed faces singularity for $`Et=\pi /2`$.

Suppose we define a contour, encircling the origin, at a time $`t=0`$, and look for a time evolution of its arbitrary point $`i`$, whose position we denote by $`\{x_i(t),y_i(t)\}`$. Combining Eq. (5) and Eq. (10) we get

$`{\displaystyle \frac{dx_i}{dt}}=f(x_i,y_i)y_i,{\displaystyle \frac{dy_i}{dt}}=f(x_i,y_i)x_i,`$ (11)
with

$$f(x,y)=\frac{\alpha \mathrm{cos}(Et)}{x^2+y^2\alpha ^2+2\alpha xy\mathrm{sin}(Et)}.$$
(12)
From Eq. (11) it is apparent that the distance of the point $`\{x_i(t),y_i(t)\}`$ from the origin does not change in course of the evolution. Indeed the quantity $`x_i(t)^2+y_i(t)^2`$ is a constant of motion as long as the function (12) is well defined. Note that it diverges, at $`Et=\pi /2`$, for points lying on the nodal line. Therefore, for $`Et=\pi /2\epsilon `$ with arbitrary small $`\epsilon >0`$ nothing extraordinary happens with the contour until at $`Et=\pi /2`$ the contour faces singularity (see Fig. 1 for an example of the contour evolution). Thus, the HKT can not be applied because quantum evolution of the wave function pushes an evolving contour to singular points and the quantities involved in the theorem become undefined.

We would like to stress that the presented scenario where the vortex situated in the harmonic trap reveals periodic changes of the topological charge is possible only when the initial vortex is placed exactly at the center of the potential. Otherwise the vortex changes its position — it escapes to infinity and another vortex with the opposite charge arrives from infinity and the scenario repeats periodically.

In the second example we would like to present violation of the HKT due to annihilation of a vortex ring. This example comes from Ref.  where creation and annihilation of a vortex ring for freely moving particle is presented. Even though it is not clearly stated in Ref. , the HKT is not fulfilled in that case. The wave function of a vortex ring  (in dimensionless units) may be written in the form

$`\mathrm{\Psi }(x,y,z,t)`$ $``$ $`[(xkt)^2+y^2+z^21`$ (13)
$`+`$ $`i3(z+t)]e^{ikxik^2t/2},`$ (14)
where the wave vector $`\stackrel{}{k}=(k,0,0)`$ is related to a motion of the “center of mass” of the vortex ring with a constant velocity $`\stackrel{}{k}`$. The vortex ring corresponds to the nodal line of the wave function and is located at the intersection of the plane $`z+t=0`$ and sphere $`(xkt)^2+y^2+z^2=1`$. At time $`t=1`$, the vortex is born at a point $`(k,0,1)`$ and, at time $`t=1`$, it disappears at a point $`(k,0,1)`$. The radius of the vortex changes in time as $`\sqrt{1t^2}`$.

Suppose at any time $`t(1,1)`$, we define a contour $`C`$ so that it encircles the vortex (see Fig. 2) and the circulation of the velocity field corresponds to $`n=1`$. The evolving contour can not cross the vortex ring without facing a singularity in the velocity field. However, the ring at some moment starts shrinking and at $`t=1`$ it reduces to a point. Consequently, at $`t=1`$, the contour must go through a singularity of the velocity field and the integral (6) and also the HKT become meaningless.

The last example we would like to comment on shows collision of two vortices leading to the appearance of a single doubly charged vortex. Consider, for simplicity, a two-dimensional H atom initially in the first excited state with angular momentum $`L_z=1`$ \[i.e. $`\psi _1(r,\phi )r\mathrm{exp}(2r/3)\mathrm{exp}(i\phi )`$ in atomic units\] that is driven resonantly by a circularly polarized electromagnetic field. The field frequency is tuned to the transition between the first and second excited energy eigenstates. It results in the familiar Rabi oscillation  between the $`\psi _1(r,\phi )`$ state and the second excited state with angular momentum $`L_z=2`$ \[i.e. $`\psi _2(r,\phi )r^2\mathrm{exp}(2r/5)\mathrm{exp}(i2\phi )`$\]. Indeed, the time evolution of the wavefunction reads

$$|\psi (t)\mathrm{cos}(Dt)e^{iE_1t}|\psi _1+e^{iE_2t}\mathrm{sin}(Dt)|\psi _2,$$
(15)
where $`D`$ is a dipole matrix element ; $`E_1`$, $`E_2`$ are energies of $`|\psi _1`$, $`|\psi _2`$ states.

For $`t=0`$ there is a vortex with $`n=1`$ at the center of the coordinates as expected for the state with angular momentum $`L_z=1`$. When $`Dt`$ increases another vortex with $`n=1`$ moves in from infinity, collides with the first one (situated at the center of the coordinates during the whole time evolution) at $`Dt=\pi /2`$ and then moves out to infinity again and so on. During the collision, i.e. for $`Dt=\pi /2`$, a single vortex with $`n=2`$ is formed. In the considered example, the HKT does not apply: if we define a contour so that it encircles the vortex with $`n=1`$ situated at the center of the coordinate, such a contour encounters a singularity in the velocity field during the collision with the other vortex. Such a behavior, is expected to hold in an arbitrary collision between two vortices leading to formation of one, doubly charged, vortex.

These examples are not the only ones illustrating that vortices can disappear or change their charge in the course of time evolution. They were chosen to illustrate that such processes can happen in different physical systems and results in the violation of the HKT. The theorem is explored in the context of vortices’ stability (see e.g. ) to imply the constancy of a vortex topological charge. As we have clearly illustrated the HKT can not assure persistence of vortex currents in the quantum mechanical systems. Therefore we conclude that the vortex topological charge is not as robust quantity as it is commonly believed (see e. g.  and references therein, ). Finally, we would like to mention that appearance of vortices can happen in a way that does not violate the HKT. Namely, they can appear in the form of a closed vortex line that springs from a point or as a vortex-antivortex pair creation from a node of a wave function .

## V Summary and conclusions

We have analyzed the applicability of the Helmholtz-Kelvin theorem in the hydrodynamical formulation of the quantum mechanics. The velocity field of the probability fluid is defined as the gradient of the phase of the quantum wave function. This implies that nonzero circulation, along a given contour, may come out in the system if the field reveals a singularity at certain points on a surface encircled by the contour. Adopting the HKT to the quantum liquid may suggest that such a topological charge of the system can not change.

However, the HKT may be employed if a given contour evolves through points where the velocity field is well defined. It may happen even in classical hydrodynamics that such an assumption is not fulfilled if, e.g., liquid encounters obstacles in the flow . In quantum liquids the situation is more complicated because a singularity is necessary for a nonzero circulation. Indeed, phase of a wave function is undefined at a vortex core, which means that singularities appear in hydrodynamical formulation of quantum mechanics whenever vortices show up. This property makes distinction of hydrodynamical description of quantum system from the classical hydrodynamics. We have presented simple analytical examples where the quantum evolution of a wave function pushes a contour to a singular point. Such a process is accompanied by a change of the vortex topological charge.

We have illustrated the violation of the HKT choosing examples that correspond to linear quantum systems. Macroscopic quantum behavior is present in interacting many particle systems that are usually described in the mean field approximation by a nonlinear Schrödinger equation. In Appendix we complete the analysis of the HKT for a nonlinear Schrödinger evolution. Vortices maybe also investigated in propagation of classical light . Indeed, the evolution equation of the slowly varying envelope of a light beam can have identical form as the Schrödinger equation, which allows for exactly the same considerations as we have presented above. The dynamics of vortices and escapes of the off-center vortex in an analog of the asymmetric harmonic potential have been experimentally observed in such systems .

We are grateful to J. Zakrzewski and G. Molina-Terriza for fruitful discussion and to U. Fischer for the critical reading of the manuscript. Support of KBN under project 5 P03B 088 21 is acknowledged.

## VI Appendix: Helmholtz-Kelvin theorem for nonlinear systems

For completeness we analyze here the application of the HKT in hydrodynamical formulation of nonlinear quantum mechanics. The so-called Gross-Pitaevskii equation is successfully used to describe the properties of a Bose-Einstein condensate in trapped alkali atoms . In particular this equation exhibits various vortex solutions . To make the discussion more general we will also include the mean field term describing a possible dipolar interactions between condensed atoms . The nonlinear Schrödinger equation of interest reads

$`i\mathrm{}{\displaystyle \frac{}{t}}\mathrm{\Psi }(\stackrel{}{r},t)=`$ $``$ $`{\displaystyle \frac{\mathrm{}^2}{2m}}\stackrel{}{}^2\mathrm{\Psi }(\stackrel{}{r},t)+V(\stackrel{}{r},t)\mathrm{\Psi }(\stackrel{}{r},t)`$ (16)
$`+`$ $`\left({\displaystyle d^3r^{}V_{int}(\stackrel{}{r}\stackrel{}{r}^{})|\mathrm{\Psi }(\stackrel{}{r}^{},t)|^2}\right)\mathrm{\Psi }(\stackrel{}{r},t),`$ (17)
where $`V_{int}`$ corresponds to two body interactions (e.g. point or dipolar interactions ). Substitution $`\mathrm{\Psi }(\stackrel{}{r},t)=\sqrt{\rho (\stackrel{}{r},t)}\mathrm{exp}(i\chi (\stackrel{}{r},t))`$ leads to the continuity equation (3) and the following dynamical equation for the velocity field

$`m{\displaystyle \frac{\stackrel{}{v}}{t}}`$ $`+`$ $`\stackrel{}{}({\displaystyle \frac{1}{2}}mv^2+V+{\displaystyle }d^3r^{}V_{int}(\stackrel{}{r}\stackrel{}{r}^{})\rho (\stackrel{}{r}^{},t)`$ (18)
$``$ $`{\displaystyle \frac{\mathrm{}^2}{2m}}{\displaystyle \frac{^2\sqrt{\rho }}{\sqrt{\rho }}})=0.`$ (19)
The only difference in comparison with Eq. (2) is the appearance of two new terms in the bracket of Eq. (18). These terms, however, do not change the proof of the HKT — for example Eq. (6) does not possess any potential like terms. Therefore the Helmholtz-Kelvin theorem holds also for systems described by Eq. (16) under the same assumptions as in the case of linear quantum mechanics discussed in Sec. III.