text:
# The Physical Significance of Confidence Intervals

## I Introduction

The possibility to apply successfully Frequentist statistics to problematic cases in frontier research has received a fundamental contribution with the proposal of the Unified Approach by Feldman and Cousins . The Unified Approach uses Neyman’s method  to construct a confidence belt that guarantees the derivation of confidence intervals with a correct Frequentist coverage (see ). Furthermore, the Unified Approach allows to obtain an automatic transition from two-sided confidence intervals to upper limits in the case of negative results, preserving the property of a correct Frequentist coverage. Several works  have followed the Unified Approach paper discussing alternative methods to construct the confidence belt. Hence, at present several Frequentist methods with interesting properties are available and under discussion. Their performances are also often compared with those of the Bayesian Theory (see Ref. ).

In this paper we define some appropriate statistical quantities that could help to decide which is the most appropriate method to use in order to obtain confidence intervals with the desired level of physical significance (see also Ref. ). We also define the sensitivity of an experiment to a signal.

The physical significance of a confidence interval is its degree of reliability, which is very important, because scientists use confidence intervals provided by experiments or specialized compilations (for example, the Review of Particle Physics ) as inputs for their calculations. Unreliable confidence intervals may lead to wrong or hazardous conclusions.

In Section II we consider the expectation value of the upper limit in absence of a signal (that we propose to call *“exclusion potential”*) and its standard deviation. In Sections III and IV we discuss the possibility to calculate the expectation value (*“upper and lower detection functions”*) and the standard deviation of the upper and lower limits of the confidence interval produced by an experiment in the presence of a signal and we propose an appropriate definition of the *“sensitivity”* of an experiment to the signal searched for.

In this article we consider explicitly the case of a Poisson process with known background in the framework of Frequentist and Bayesian statistical theories, but similar considerations apply also to the case of a Gaussian distribution with boundary.

## II Exclusion potential

An important quantity introduced by Feldman and Cousins  is the average upper limit for the signal $`\mu `$ that would be obtained by an ensemble of experiments if $`\mu =0`$ (in the case of a Poisson process with known background, “the average upper limit that would be obtained by an ensemble of experiments with the expected background and no true signal”). They called this quantity *“sensitivity”*, but we think that this name is quite misleading, because it gives the impression that this quantity represents the expected capability of the experiment to reveal a true signal *** For example, one can check on the on-line Webster Dictionary at http://www.m-w.com/dictionary that “sensitivity” is “the quality or state of being sensitive” and the adjective “sensitive” means “capable of being stimulated or excited by external agents (as light, gravity, or contact)”. In our case, since the background is known, it can be considered an “internal agent”, whereas the true signal is the “external agent” under investigation. From the physical point of view the sensitivity is related in many fields of application to the concept of minimum detectable signal, therefore it is a quantity defined for $`\mu >0`$, not when $`\mu =0`$. . Instead, the true signal is assumed to be absent. Hence, it is clear that the quantity under consideration does not represent the sensitivity of the experiment to the signal that is searched for, but it represents the expected upper limit for $`\mu `$ that will be obtained if there is no signal. Therefore, we propose to call this quantity *“exclusion potential”* The on-line Webster Dictionary at http://www.m-w.com/dictionary says that “potential” is “something that can develop or become actual”. , a name that we will use in the following. As a further justification of this name, we note that in the case of neutrino oscillation experiments the exclusion potential is associated with the so-called “exclusion curves” in the space of the neutrino mixing parameters.

In the case of a Poisson process with known background the exclusion potential is given by

$$\mu _{\mathrm{ep}}(b,\alpha )=\underset{n=0}{\overset{\mathrm{}}{}}\mu _{\mathrm{up}}(n,b,\alpha )P(n|\mu =0,b),$$
(1)
where $`n`$ is the number of counts, $`b`$ is the expected mean background, $`\mu `$ is the mean true signal, $`P(n|\mu ,b)`$ is the Poisson p.d.f. for the process and $`\mu _{\mathrm{up}}(n,b,\alpha )`$ is the upper limit of the $`100(1\alpha )\%`$ confidence interval for $`\mu `$ corresponding to $`n`$ counts. The exclusion potential $`\mu _{\mathrm{ep}}`$ depends on the values of the upper limits $`\mu _{\mathrm{up}}(n,b,\alpha )`$, which are different in the different methods for calculating the confidence belt.

It is interesting to note that the above definition of exclusion potential can be extended to the results obtained with the Bayesian Theory (see, for example, ) if $`\mu _{\mathrm{up}}`$ is interpreted as the upper limit of the Bayesian credibility interval for $`\mu `$. In the following we will consider the Bayesian Theory assuming a flat prior and shortest credibility intervals for $`\mu `$. In this case the posterior p.d.f. for $`\mu `$ is

$$P(\mu |n,b)=(b+\mu )^ne^\mu \left(n!\underset{k=0}{\overset{n}{}}\frac{b^k}{k!}\right)^1,$$
(2)
and the probability (degree of belief) that the true value of $`\mu `$ lies in the range $`[\mu _1,\mu _2]`$ is given by

$$P(\mu [\mu _1,\mu _2]|n,b)=\left(e^{\mu _1}\underset{k=0}{\overset{n}{}}\frac{(b+\mu _1)^k}{k!}e^{\mu _2}\underset{k=0}{\overset{n}{}}\frac{(b+\mu _2)^k}{k!}\right)\left(\underset{k=0}{\overset{n}{}}\frac{b^k}{k!}\right)^1.$$
(3)
The shortest $`100(1\alpha )\%`$ credibility intervals $`[\mu _{\mathrm{low}},\mu _{\mathrm{up}}]`$ are obtained by choosing $`\mu _{\mathrm{low}}`$ and $`\mu _{\mathrm{up}}`$ such that $`P(\mu [\mu _{\mathrm{low}},\mu _{\mathrm{up}}]|n,b)=1\alpha `$ and $`P(\mu _{\mathrm{low}}|n,b)=P(\mu _{\mathrm{up}}|n,b)`$ if possible (with $`\mu _{\mathrm{low}}0`$), or $`\mu _{\mathrm{low}}=0`$.

The solid lines in Figs. 1 and 3 represent the exclusion potential as a function of the background in the interval $`0b20`$ for a confidence level $`90\%`$ ($`\alpha =0.10`$) obtained with the Unified Approach  (Figs. 1A and 3A), with the Bayesian Ordering method The Bayesian Ordering method, as the Unified Approach, is a Frequentist method with correct coverage and automatic transition from two-sided confidence intervals to upper limits in the case of negative results.   (Figs. 1B and 3B) and with the Bayesian Theory assuming a flat prior and shortest credibility intervals (Figs. 1C and 3C).

Feldman and Cousins  suggested that in the cases in which the measurement is less than the estimated mean background and a stringent upper bound on $`\mu `$ is inferred, the experimental collaboration should report also the exclusion potential (that they call “sensitivity”) of the experiment. This is also recommended by the Particle Data Group .

In practice, the comparison of the upper bound and the exclusion potential is used as an assessment of the reliability of the upper bound. However, one can notice that a simple comparison of the upper bound with the exclusion potential does not give any information unless a meaningful scale of comparison is given. We think that a meaningful scale is the possible fluctuation of the upper bound $`\mu _{\mathrm{up}}(n,b,\alpha )`$ in an ensemble of experiments with the expected background and no true signal. A quantification of this fluctuation is provided by the standard deviation $`\sigma _{\mathrm{ep}}(b,\alpha )`$ of the upper limit $`\mu _{\mathrm{up}}`$ calculated assuming $`\mu =0`$,

$$\sigma _{\mathrm{ep}}^2(b,\alpha )=\underset{n=0}{\overset{\mathrm{}}{}}\left[\mu _{\mathrm{up}}(n,b,\alpha )\mu _{\mathrm{ep}}(b,\alpha )\right]^2P(n|\mu =0,b).$$
(4)
The shadowed regions delimited by the dashed lines in Fig. 1 represent the range $`[\mu _{\mathrm{ep}}\sigma _{\mathrm{ep}},\mu _{\mathrm{ep}}+\sigma _{\mathrm{ep}}]`$ for $`\alpha =0.10`$ obtained with the Unified Approach (A), with the Bayesian Ordering method (B) and with the Bayesian Theory assuming a flat prior and shortest credibility intervals. The corresponding probability to obtain an upper bound in the interval $`[\mu _{\mathrm{ep}}\sigma _{\mathrm{ep}},\mu _{\mathrm{ep}}+\sigma _{\mathrm{ep}}]`$ if $`\mu =0`$ is shown in the three upper figures 1a, 1b and 1c. One can see that, except for small values of the background $`b`$, the probability to obtain an upper bound in the interval $`[\mu _{\mathrm{ep}}\sigma _{\mathrm{ep}},\mu _{\mathrm{ep}}+\sigma _{\mathrm{ep}}]`$ if $`\mu =0`$ is not far from 68% in the three considered methods. The probability curves in Fig. 1 have wild jumps because $`n`$ is an integer and $`\mu _{\mathrm{up}}`$ has discrete jumps as $`n`$ is varied.

As an illustration, in Fig. 2 we have plotted the probability of the possible values of the 90% CL upper bound $`\mu _{\mathrm{up}}`$ if $`\mu =0`$ and $`b=13`$. One can see that the upper bound $`\mu _{\mathrm{up}}`$ can assume only discrete values. The probability to obtain an upper bound in the interval $`[\mu _{\mathrm{ep}}\sigma _{\mathrm{ep}},\mu _{\mathrm{ep}}+\sigma _{\mathrm{ep}}]`$, delimited by the dotted vertical lines in Fig. 2, has discrete jumps as $`b`$ is changed, depending on which possible values of the upper bound are included in the interval.

It is also possible to calculate the possible range of fluctuation of the upper bound $`\mu _{\mathrm{up}}`$ with a desired probability $`\gamma `$. The shadowed regions delimited by the dashed lines in Fig. 3 show the 90% width of the 90% CL upper limit, i.e. the possible range of fluctuation of the $`\mu _{\mathrm{up}}(n,b,\alpha =0.90)`$ with probability $`\gamma =0.90`$ as a function of $`b`$ if $`\mu =0`$. This range of fluctuation has been calculated in order to obtain a band as symmetric as possible around $`\mu _{\mathrm{ep}}`$. In practice this is done by a computer program that simultaneously decreases the lower limit $`\mu _{\mathrm{up}}^{()}(b,\alpha ,\gamma )`$ of the band (lower dashed lines in Fig. 3) and increases the upper limit $`\mu _{\mathrm{up}}^{(+)}(b,\alpha ,\gamma )`$ of the band (upper dashed lines in Fig. 3) until the condition

$$\underset{n=n^{()}(b,\alpha ,\gamma )}{\overset{n^{(+)}(b,\alpha ,\gamma )}{}}P(n|\mu =0,b)\gamma $$
(5)
is reached. Here $`n^{()}(b,\alpha ,\gamma )`$ and $`n^{(+)}(b,\alpha ,\gamma )`$ are the values of $`n`$ such that

$$\mu _{\mathrm{up}}(n^{()}(b,\alpha ,\gamma ),b,\alpha )=\mu _{\mathrm{up}}^{()}(b,\alpha ,\gamma ),\mu _{\mathrm{up}}(n^{(+)}(b,\alpha ,\gamma ),b,\alpha )=\mu _{\mathrm{up}}^{(+)}(b,\alpha ,\gamma ).$$
(6)
The inequality sign in Eq. (5) is needed because $`n`$ is an integer and in general it is not possible to obtain exactly the desired probability $`\gamma `$. This is also the reason for the fact that the dashed lines in Fig. 3 are not smooth. The dotted lines in Fig. 3 represent the lower limit for $`\mu _{\mathrm{up}}(n,b,\alpha =0.90)`$ as a function of $`b`$, that is obtained for $`n=0`$.

It is clear that having analyzed the data using, for example, the Unified Approach and having obtained an upper bound $`\mu _{\mathrm{up}}`$ with an expected background $`b`$, looking at Fig. 3A one can judge if the upper bound $`\mu _{\mathrm{up}}`$ is reasonable or the result of an unlikely statistical fluctuation. For example, the Heidelberg-Moscow double-beta decay experiment  measured recently $`n=7`$ events with an expected background $`b=13`$. Using the Unified Approach they obtained $`\mu _{\mathrm{up}}=2.07`$ at 90% CL, with exclusion potential $`\mu _{\mathrm{ep}}=7.51`$. Looking at Fig. 3A one can see that the 90% lower limit for $`\mu _{\mathrm{up}}`$ assuming $`\mu =0`$ is $`\mu _{\mathrm{up}}^{()}=2.07`$, so the discrepancy between $`\mu _{\mathrm{up}}`$ and $`\mu _{\mathrm{ep}}`$ is just acceptable at the border of 10% probability. Using the Bayesian Ordering method they would have obtained $`\mu _{\mathrm{up}}(n=7,b=13,\alpha =0.10)=3.33`$, with exclusion potential $`\mu _{\mathrm{ep}}(b=13,\alpha =0.10)=8.11`$, and with the Bayesian Theory (with a flat prior and shortest credibility intervals) $`\mu _{\mathrm{up}}(n=7,b=13,\alpha =0.10)=4.01`$ and $`\mu _{\mathrm{ep}}(b=13,\alpha =0.10)=7.81`$.

Of course, one can obtain the same results using the measured value of $`n`$ and the Poisson p.d.f. $`P(n|\mu =0,b)`$ with the expected background $`b`$. The advantage of Figs. 1 and 3 is that one can easily transform $`\mu _{\mathrm{up}}`$ and $`\mu _{\mathrm{ep}}`$ in the physical quantity of interest, presenting the result only in terms of physical quantities. For example, in double-beta decay experiments $`\mu `$ is connected to the effective Majorana mass $`|m|`$ of the electron neutrino by the relation

$$|m|=\xi \frac{\sqrt{\mu }}{||},$$
(7)
where $`\xi `$ is a constant with dimension of mass that depends on the decaying nucleus and $``$ is the nuclear matrix element. In the case of the Heidelberg-Moscow double-beta decay experiment , the decaying nucleus is 76Ge, with $`\xi =0.57\mathrm{eV}`$. Using the nuclear matrix element $`||=2.80`$ calculated in , the 90% CL upper bound for the effective Majorana mass obtained with the Unified Approach is $`|m|_{\mathrm{up}}=0.29\mathrm{eV}`$, the corresponding exclusion potential is $`|m|_{\mathrm{ep}}=0.56\mathrm{eV}`$ and the 90% lower limit for the fluctuations of $`|m|_{\mathrm{up}}`$ (if $`|m|=0`$) is $`|m|_{\mathrm{up}}^{()}=0.29\mathrm{eV}`$. Using instead the Bayesian Ordering method, we obtain $`|m|_{\mathrm{up}}=0.37\mathrm{eV}`$, $`|m|_{\mathrm{ep}}=0.58\mathrm{eV}`$ and $`|m|_{\mathrm{up}}^{()}=0.37\mathrm{eV}`$, and with the Bayesian Theory (with a flat prior and shortest credibility intervals) we have $`|m|_{\mathrm{up}}=0.41\mathrm{eV}`$, $`|m|_{\mathrm{ep}}=0.57\mathrm{eV}`$ and $`|m|_{\mathrm{up}}^{()}=0.41\mathrm{eV}`$.

Let us define the Pull of a null result as

$$\mathrm{Pull}(n,b,\alpha )=\frac{\mu _{\mathrm{up}}(n,b,\alpha )\mu _{\mathrm{ep}}(b,\alpha )}{\sigma _{\mathrm{ep}}(b,\alpha )}.$$
(8)
If $`\mathrm{Pull}(n,b,\alpha )1`$, the experimental upper limit $`\mu _{\mathrm{up}}`$ is significantly weaker than the exclusion potential $`\mu _{\mathrm{ep}}`$ and may be considered as a weak indication that a signal may be present ($`\mu >0`$). On the other hand, if $`\mathrm{Pull}(n,b,\alpha )1`$ and there is no doubt on the value of the mean background $`b`$, it means that the experiment has experienced an unlikely low fluctuation of the background and the resulting upper bound, that is significantly more stringent than the exclusion potential, is not reliable from a physical point of view and it is likely to increase if the experiment is continued (a quite undesirable behavior), as shown by the example of the KARMEN experiment (see Fig. 4 of Ref. ). Moreover, a method that gives values of the Pull closer to zero produces upper bounds that are more reliable from a physical point of view.

For example, in the case of the Heidelberg-Moscow double-beta decay experiment , we have $`n=7`$, $`b=13`$ and $`\alpha =0.10`$, giving $`\sigma _{\mathrm{ep}}=3.91`$ and $`\mathrm{Pull}=1.39`$ in the Unified Approach, $`\sigma _{\mathrm{ep}}=3.52`$ and $`\mathrm{Pull}=1.36`$ with the Bayesian Ordering method, and $`\sigma _{\mathrm{ep}}=2.98`$ and $`\mathrm{Pull}=1.27`$ in the Bayesian Theory with a flat prior and shortest credibility intervals. Hence, among the two Frequentist methods that we have considered, the upper bound obtained with Bayesian Ordering is slightly more reliable, from a physical point of view, than the one obtained with the Unified Approach. If one is willing to accept the Bayesian Theory, the corresponding upper bound is clearly the most reliable one from a physical point of view.

Let us point out that the knowledge of the possible fluctuation of the upper bound $`\mu _{\mathrm{up}}`$ with respect to the exclusion potential $`\mu _{\mathrm{ep}}`$ can also help to *decide, before looking at the data, which is the most appropriate Frequentist method for the statistical analysis*.

This can be understood comparing, for example, Figs. 3A and 3B, obtained with two Frequentist methods with correct coverage. One can see that the upper dashed lines in the two figures almost coincide and the exclusion potential is slightly lower in the Unified Approach, with a difference going from 0.30 for $`b=1`$ to 0.73 for $`b=19`$, with a relative difference of 8–9%. On the other hand, the lower dashed lines and the dotted lines obtained with the Bayesian Ordering are significantly higher than those obtained with the Unified Approach. The difference of the lower dashed lines, goes from 0.43 for $`b=1`$ to 1.40 for $`b=19`$, with a relative difference going from 27% to 64%. The difference between the lowest possible values of $`\mu _{\mathrm{up}}`$ (dotted lines) is quite large: for $`b4`$ the lowest possible values of $`\mu _{\mathrm{up}}`$ is about 0.8 in the Unified Approach and about 1.8 in the Bayesian Ordering method, more than twice!

Hence, if one does not want to risk to have to present a very stringent limit, which would be statistically correct but physically misleading, in the case of observation of less events than the expected background, one can look at figures like Figs. 3A and 3B and the corresponding ones for other Frequentist methods and decide which is the method more suitable for his tastes. Let us emphasize that this choice *must be done before looking at the data*. If one chooses the statistical method on the basis of the data, the property of coverage is lost.

Furthermore, the exclusion potential of an experiment can be calculated and published before starting the experiment or before the data are known, in order to have an indication of the excluded region that will be obtained in the case of absence of a signal (or a signal much smaller than the expected background). We think that it would be useful to publish together with the exclusion potential also the standard deviation of the upper limit in the absence of a signal, in order to illustrate the possible fluctuations of the excluded region and to give, at the same time, a quantitative statement on the precision of the experiment.

## III Detection functions

The exclusion potential and the standard deviation of the upper limit in the absence of a signal are interesting quantities, but they give only information on the possible experimental result in the worst-case scenario, that in which the signal is absent or so small to be undetectable. Usually researchers are more interested in finding positive signals. For example, they would like to know in advance which would be the most likely outcome of the experiment if there is a true signal. In this case, we propose to calculate the *upper and lower detection functions*, $`\mu _+(\mu ,b,\alpha )`$ and $`\mu _{}(\mu ,b,\alpha )`$, obtained averaging the upper and lower limits $`\mu _{\mathrm{up}}(n,b,\alpha )`$ and $`\mu _{\mathrm{low}}(n,b,\alpha )`$ over $`n`$ with the Poisson p.d.f. $`P(n|\mu ,b)`$:

$$\mu _\pm (\mu ,b,\alpha )=\underset{n=0}{\overset{\mathrm{}}{}}\mu _{\genfrac{}{}{0pt}{}{\mathrm{up}}{\mathrm{low}}}(n,b,\alpha )P(n|\mu ,b).$$
(9)
The standard deviation of $`\mu _{\genfrac{}{}{0pt}{}{\mathrm{up}}{\mathrm{low}}}(n,b,\alpha )`$ is given by

$$\sigma _\pm ^2(\mu ,b,\alpha )=\underset{n=0}{\overset{\mathrm{}}{}}\left[\mu _{\genfrac{}{}{0pt}{}{\mathrm{up}}{\mathrm{low}}}(n,b,\alpha )\mu _\pm (\mu ,b,\alpha )\right]^2P(n|\mu ,b).$$
(10)
In Fig. 4 we have plotted $`\mu _+`$, $`\mu _{}`$ (upper and lower solid lines, respectively) as functions of $`\mu `$ for $`\alpha =0.10`$ and $`b=13`$ in the Unified Approach  (Fig. 4A), in the Bayesian Ordering method  (Fig. 4B) and in the Bayesian Theory assuming a flat prior and shortest credibility intervals (Fig. 4C). The shadowed regions delimited by the dashed lines in Fig. 4 represent the bands $`\mu _+\pm \sigma _+`$ (upper band) and $`\mu _{}\pm \sigma _{}`$ (lower band). From Fig. 4 one can see that the average upper bound $`\mu _+`$ is almost identical in the three methods, but the range $`\mu _+\pm \sigma _+`$ for small values of $`\mu `$ is shortest in the Bayesian Theory and largest in the Unified Approach. The average lower bound $`\mu _{}`$ and the range $`\mu _{}\pm \sigma _{}`$ are similar in the three methods, with the small difference that $`\mu _{}\sigma _{}>0`$ for $`\mu 9`$ in the two Frequentist methods (Unified Approach and Bayesian Ordering) and $`\mu 10`$ in the Bayesian Theory. The three upper plots in Fig. 4 show the probability to find $`\mu _{\mathrm{low}}`$ in the interval $`\mu _{}\pm \sigma _{}`$ (solid lines) and the probability to find $`\mu _{\mathrm{up}}`$ in the interval $`\mu _+\pm \sigma _+`$ (dashed lines). The latter probability is high (larger than 80%) for small values of $`\mu `$, where the interval $`\mu _{}\pm \sigma _{}`$ includes zero, and stabilizes around 68% for higher values of $`\mu `$ (the fluctuations and discontinuities of the probability as a function of $`\mu `$ are due to the discreteness of $`n`$).

The detection functions and the standard deviations of the lower and upper bounds show the expected result and its possible fluctuations if the signal under measurement is not negligibly small. In the next section we present a definition of sensitivity of an experiment to a signal.

## IV Sensitivity to a signal

Often researchers would like to plan an experiment capable of revealing a signal whose value is indicated by previous measurements or predicted by theory. Hence, it is useful to define the *sensitivity of an experiment to a signal*.

Two probabilities must be involved in the definition of the sensitivity to a signal: the confidence level $`1\alpha `$ of the confidence interval that represents the result of the experiment and the probability $`\lambda `$ to find a confidence interval with a positive lower bound.

We think that an appropriate definition of the $`100\lambda \%`$ sensitivity corresponding to a $`100(1\alpha )\%`$ confidence level, $`\mu _\mathrm{s}(b,\alpha ,\lambda )`$, of an experiment measuring a Poisson process with known background $`b`$ is *the value of $`\mu `$ for which there is a probability $`\lambda `$ to find a positive lower limit for $`\mu `$ with confidence level $`100(1\alpha )\%`$*. Hence, we define $`\mu _\mathrm{s}(b,\alpha ,\lambda )`$ through the equation§§§ After the completion of this work, we have been informed that Hernandez, Nava and Rebecchi  defined similar criteria in order to calculate “discovery limits” in prospective studies. Our Eq. (11) coincides with their Eq. (6) with $`\delta =1\lambda `$, and our Eq. (12) corresponds to their Eq. (5) with $`ϵ=\alpha `$ in the case of Frequentist methods with correct coverage and automatic transition from two-sided confidence intervals to upper limits for a small number of counts.

$$\underset{nn_\mathrm{s}(b,\alpha )}{}P(n|\mu _s(b,\alpha ,\lambda ),b)=\lambda ,$$
(11)
where $`n_\mathrm{s}(b,\alpha )`$ is the smallest integer such that

$$\mu _{\mathrm{low}}(n_\mathrm{s}(b,\alpha ),b,\alpha )>0.$$
(12)
Here $`\mu _{\mathrm{low}}(n,b,\alpha )`$ is the lower limit of the $`100(1\alpha )\%`$ confidence interval (credibility interval in the Bayesian Theory) corresponding to the observation of $`n`$ events. In all Frequentist methods with correct coverage that guarantee an automatic transition from two-sided confidence intervals to upper limits for $`nb`$ (as the Unified Approach  and the Bayesian Ordering method ), the acceptance interval for $`\mu =0`$ starts at $`n_1(\mu =0,b,\alpha )=0`$ and ends at $`n_2(\mu =0,b,\alpha )`$, where $`n_2(\mu =0,b,\alpha )`$ is the smallest integer such that

$$\underset{n=0}{\overset{n_2(\mu =0,b,\alpha )}{}}P(n|\mu =0,b)1\alpha .$$
(13)
Then it is clear that

$$n_\mathrm{s}(b,\alpha )=n_2(\mu =0,b,\alpha )+1$$
(14)
is the smallest integer that satisfies Eq. (12).

Figure 5 shows the value of $`\mu _\mathrm{s}(b,\alpha ,\lambda )`$ as a function of $`b`$ in Frequentist methods (solid lines) and in the Bayesian Theory with a flat prior and shortest credibility intervals (dashed lines) for $`1\alpha =0.90,\mathrm{\hspace{0.17em}0.95},\mathrm{\hspace{0.17em}0.99}`$ and $`\lambda =0.50,\mathrm{\hspace{0.17em}0.90},\mathrm{\hspace{0.17em}0.99}`$. The lines are not smooth because of the discreteness of $`n_\mathrm{s}(b,\alpha )`$ that causes jumps of the solution of Eq. (11) as $`b`$ varies from one value to another with different $`n_\mathrm{s}(b,\alpha )`$.

The sensitivity $`\mu _\mathrm{s}(b,\alpha ,\lambda )`$ provides useful information for the planning of an experiment with the purpose of exploring a range $`[\mu _{\mathrm{min}},\mu _{\mathrm{max}}]`$ of possible values of $`\mu `$ that could be inferred from the results of other experiments or from theory. In order to do this, the background in the experiment must be small enough that the sensitivity $`\mu _\mathrm{s}(b,\alpha ,\lambda )`$ is smaller than $`\mu _{\mathrm{min}}`$. In this case the experiment will have probability $`1\alpha `$ to obtain a correct result (i.e. a confidence interval that contains the true value of $`\mu `$) and a probability bigger than $`\lambda `$ to obtain a positive lower limit, i.e. to reveal a true signal, if $`\mu >\mu _\mathrm{s}(b,\alpha ,\lambda )`$. The probability that the experiment will reveal a true signal within a correct confidence interval is larger the product $`(1\alpha )\lambda `$ (if $`\mu >\mu _\mathrm{s}(b,\alpha ,\lambda )`$). Therefore, it is desirable to have both $`1\alpha `$ and $`\lambda `$ large. In Fig. 5 we have chosen $`1\alpha =0.90,\mathrm{\hspace{0.17em}0.95},\mathrm{\hspace{0.17em}0.99}`$, that are commonly used values for the confidence level, and $`\lambda =0.50,\mathrm{\hspace{0.17em}0.90},\mathrm{\hspace{0.17em}0.99}`$. We think that $`1\alpha `$ should be always chosen large, preferably $`1\alpha =0.99`$, because getting a correct result is the most important thing. As for $`\lambda `$, a large value is important in order to have good chances to reveal the signal (if it exist!). For example, having $`1\alpha =0.99`$ and $`\lambda =0.99`$ gives a probability bigger than 98% to find a true signal within a correct confidence interval (if $`\mu >\mu _\mathrm{s}(b,\alpha ,\lambda )`$). On the other hand, for $`1\alpha =0.90`$ and $`\lambda =0.50`$ ($`\lambda =0.90`$) the probability to find a true signal within a correct confidence interval can be as low as 45% (81%).

From Fig. 5 one can see that the sensitivity increases sub-linearly as the background increases. Since the background increases linearly with the time of data-taking, the sensitivity of the experiment increases sub-linearly as a function of data-taking time. Let us consider an experiment searching for a signal produced by a new process for which there is an indication from previous experiments or from theory. Since the signal, as the background, increases linearly with the time of data-taking, there is a time such that the signal becomes larger than the sensitivity and this time provides an estimate of the data-taking time necessary to reveal the new process.

It is interesting to notice that the sensitivity for $`b=13`$, $`1\alpha =0.90`$ is $`\mu _\mathrm{s}7`$ for $`\lambda =0.50`$ and $`\mu _\mathrm{s}13`$ for $`\lambda =0.90`$, in rough agreement with the lower bands in Fig. 4, which show that there is a good chance to find a lower limit for $`\mu `$ bigger than zero if $`\mu 10`$.

From this example it is clear that the sensitivity of an experiment is different from its exclusion potential. The proposals of new experiments on the search for a signal should present the sensitivity as the most interesting characteristic of the experiment. The exclusion potential should be also presented as an illustration of the potentiality of the experiment in the most unfortunate case of absence of a signal.

Let us remind the reader that the word “probability” has different definitions in the Frequentist and Bayesian theories. In the Bayesian theory “probability” is defined as “degree of belief”, whereas in the Frequentist theory it is defined as ratio of the number of positive cases and total number of trials in a large ensemble. The Frequentist definition avoids the need of subjective judgment, but it is not clear what is its meaning in the planning and realization of *one* experiment (or a few experiments). Whatever the meaning of Frequentist probability in this case, we think that it is comforting to see from Fig. 5 that the Frequentist and Bayesian values for $`\mu _\mathrm{s}(b,\alpha ,\lambda )`$ are quite close.

## V Conclusions

In conclusion, we have defined some quantities that may help to asses the reliability (physical significance) of the confidence intervals obtained with different methods. We have also defined appropriately the sensitivity of an experiment to a signal.

In Section II we have considered the quantity called “sensitivity” by Feldman and Cousins  and we have argued that this name is not appropriate because this quantity does not represent the capability of an experiment to reveal a signal. We proposed to call this quantity *“exclusion potential”*.

Considering the case of a Poisson process with known background, we have shown how the exclusion potential and the standard deviation of the upper limit in the absence of a signal may help to choose the method that is more appropriate for obtaining reliable upper limits (Section II). We have also defined the Pull of a null result, that quantifies the reliability of an experimental upper limit. In Section III we have defined the upper and lower detection functions, that give the most likely outcome of an experiment if there is a signal. In Section IV we proposed an appropriate definition of sensitivity of an experiment to a signal. These definitions apply to both Frequentist and Bayesian statistical theories and can be easily generalized to any process in which a quantity $`\mu `$ with known probability distribution is measured: the upper (lower) detection function is the average of the upper (lower) limit and the $`100\lambda \%`$ sensitivity is the lower value of $`\mu `$ for which there is a probability $`\lambda `$ to find a positive lower limit.

We considered explicitly the case of a Poisson process with known background in the framework of Frequentist and Bayesian statistical theories, but similar considerations and conclusions apply also to the case of a Gaussian distribution with boundary.


format: text

text:
# Use of DPOSS data to study globular cluster halos: an application to M~92

## 1 Introduction

The tidal radii of globular clusters (GCs) are important tools for understanding the complex interactions of GCs with the Galaxy. In fact, they have traditionally been used to study the mass distribution of the galactic halo (Innanen et al. Inn83 (1983)), or to deduce GCs orbital parameters (Freeman & Norris Fre81 (1981); Djorgovski et al. DJ96 (1996)). Tidal radii have usually been estimated (only in few cases directly measured), by fitting King models to cluster density profiles rarely measured from the inner regions out to the tidal radius, because of the nature of the photographic material, that prevented any measure in the cluster center, and the small format of the first digital cameras. Only in the last few years, the advent of deep digitized sky surveys and wide field digital detectors has allowed us to deal with the overwhelming problem of contamination from field stars and to probe the outer region of GCs directly (Grillmair et al. Gri95 (1995), hereafter G95; Zaggia et al. Zag95 (1995); Zaggia et al. Zag97 (1997); Lehman & Scholz LS97 (1997)). The study of tidal tails in galactic satellites is gaining interest for many applications related to the derivation of the galactic structure and potential, the formation and evolution of the galactic halo, as well as the dynamical evolution of the clusters themselves. Recent determinations of proper motion for some globular clusters with HIPPARCOS have made it possible to estimate the orbital parameters of a good number of them (Dinescu et al. Din99 (1999)). This helps to clarify the nature and structure of tidal extensions in GCs.

In principle, available tools to enhance cluster star counts against field stars rely on the color-magnitude diagram (CMD), proper motions, radial velocities, or a combination of the three techniques. The application of these techniques to GCs have led to the discovery that tidal or extra-tidal material is a common feature: Grillmair (Gri98 (1998)), for instance, reported the discovery of tidal tails in 16 out of 21 globular clusters. Interestingly, signature of the presence of tidal tails in GCs has also been found in four GC’s in M31 (Grillmair Gri96 (1996)). For galactic clusters, the discovery was made by using a selection in the CMD of cluster stars on catalogs extracted from digitized photographic datasets. The CMD selection technique is an economical and powerful method to detect GC tails, since it significantly decreases the number of background and/or foreground objects.

In order to test the feasibility of a survey of most GCs present in the Northern hemisphere, we applied the CMD technique to the galactic globular cluster M~92 (NGC 6341), with the aim of measuring the tidal radius and searching for the possible presence of extra-tidal material. We used plates from the Digitized Second Palomar Sky Survey (hereafter DPOSS), in the framework of the CRoNaRio (Caltech-Roma-Napoli-Rio de Janeiro) collaboration (Djorgovski et al. DJ97 (1997), Andreon et al. And97 (1997), Djorgovski et al. DJ99 (1999)). A previous account on this work was given in Zaggia et al. (Zag98 (1998)). This is the first of a series of papers dedicated to the subject –an ideal application for this kind of all-sky surveys.

## 2 The Color$``$Magnitude Diagram

The material used in this work are the $`J`$ and $`F`$ DPOSS plates of the field 278. For each band, we extracted from the whole digitized plate a sub-image (size: $`8032\times 8032`$ pixels), corresponding to an area of $`136\mathrm{}\times 136\mathrm{}`$ , with a pixel size of $`1\mathrm{}`$, centered on M~92 at coordinates (Harris Har96 (1996)):

$$\alpha _{J2000}=17^h\mathrm{\hspace{0.33em}17}^m\mathrm{\hspace{0.33em}07.3}^s$$

$$\delta _{J2000}=+43^o\mathrm{\hspace{0.33em}08}\mathrm{}\mathrm{\hspace{0.33em}11.5}\mathrm{}$$

The two images were linearized by using a density-to-intensity (DtoI) calibration curve, provided by the sensitometric spots available on the DPOSS plates. The $`F`$ plate is contaminated by two very similar satellite tracks (as an alternative, the two tracks come from a high altitude civil airplane) lying $`9\mathrm{}`$ and $`13\mathrm{}`$ from the cluster center and crossing the field in a South-East/North-West direction. The effect of these tracks can be seen as empty strips on the lower panel of Fig. 1. Other thin, fainter tracks and some galaxies are present on the same plate, but at larger distances from the cluster core region. We applied the CMD technique to datasets obtained with different astronomical packages, in order to test the reliability of object detection and photometry in crowded stellar fields. On the DPOSS plates containing M~92, we used both the SKICAT and DAOPHOT packages. SKICAT, written at Caltech (see Weir et al. 1995a , and refs. therein), is the standard software used by the CRoNaRio collaboration for the DPOSS plate processing and catalog construction. DAOPHOT is a well-tested program for stellar photometry, developed by Stetson (Ste87 (1987)), and widely used by stellar astronomers. In this work we have used DAOPHOT only to obtain aperture photometry, with APPHOT, of objects detected with the DAOFIND algorithm on the DPOSS plates.

### 2.1 The data set

The SKICAT output catalog only contains objects classified as stars in both filters. For each object, we used $`M_{\mathrm{Core}}`$ (the magnitude computed from the central nine pixels), because the other aperture magnitude is measured on an area far too large for crowded regions. The final SKICAT catalog consists of 108779 objects. Since SKICAT is optimized for the detection of faint galaxies, in the present case we needed to test its performances in crowded stellar fields to ensure that it properly detected the stellar population around the cluster.

Thus, SKICAT has been compared to DAOPHOT, which is specifically designed for crowded fields stellar photometry and has been repeatedly tested in a variety of environments, including globular clusters. The DAOPHOT dataset was built using aperture photometry on the objects detected with the DAOFIND. The threshold was set at $`3.5\sigma `$, similar to the one used by SKICAT. Aperture photometry was preferred to PSF fitting photometry, due to the large variability of the DPOSS point-spread function which makes the PSF photometry less accurate than the aperture photometry. We used an aperture of $`1.69`$ pixels of radius, corresponding to an area of approximately 9 pixels, i.e. equivalent to the area used by FOCAS/SKICAT to compute $`M_{\mathrm{Core}}`$. Indeed, the advantages of using PSF fitting are more evident in the central and more crowded regions of the cluster, while we are mainly interested in the outskirts, where crowding is less dramatic. Thus, we adopted the results from the aperture photometry, and we refer to this dataset as the DAOFIND+PHOT dataset.

The total number of objects detected in the $`J`$ and $`F`$ plates is, respectively, 240138 and 253977. The larger number of objects detected by DAOFIND, compared to those from SKICAT, is mainly due to the better capacity of DAOFIND in detecting objects in the crowded regions of the core. In the case of DAOFIND, since the convolution kernel, which is set essentially by the pixel size and seeing value, is much smaller than in SKICAT, we also have objects measured near the satellite tracks.

The FOCAS/SKICAT and DAOFIND+PHOT aperture photometry were then compared, and the results are shown in Fig. 2 where the SKICAT aperture magnitude is plotted versus the difference between itself and $`M_{\mathrm{Core}}`$. The average difference is zero, with an error distribution typical of this kind of tests, i.e. a fan-like shape with growing dispersion at fainter magnitudes. The distribution of F magnitudes in Fig. 2 clearly shows the effects of saturation at the bright end, and in both plots there are several outliers, owing to the field crowding. In fact, these outliers are much more concentrated in the inner $`12\mathrm{}`$, where their density is 0.439 arcmin-2, than at larger distances from the center, where the density drops to 0.022 arcmin-2. These objects are mostly classified as non stellar by SKICAT and DAOPHOT, since they are either foreground galaxies or, more often, unresolved multiple objects, and were rejected in the final catalogs. Their area is taken into account later on, when we compute the effective area of the annuli in the construction of the radial profile. The outliers show an asymmetric distribution with SKICAT magnitudes being brighter at bright magnitudes and viceversa at fainter magnitudes. This is due to two reasons: in the case of large objects, SKICAT splits them into multiple entries, but keeps the $`M_{\mathrm{Core}}`$ value of the originally detected (big) object; at fainter magnitudes, where objects are small, $`M_{\mathrm{Core}}`$ is computed on a number of pixels less than 9, while the aperture photometry of the objects in the DAOFIND catalog are always computed on a circle of 1.69 pixel radius.However, the contribution of these outliers to the counts is far below 1 percent of the total, as can be seen from the histogram plotted on the right hand side of fig. 2.

The above analysis shows that SKICAT catalogs are, after a suitable cleaning, usable “as they are” also for studies of moderately crowded stellar fields. We shall use the DPOSS-DAOPHOT dataset because it can better detect objects in highly crowded fields, which allows us to probe into the inner ($`200\mathrm{}<r<400\mathrm{}`$) regions of the cluster and merge our star counts profile with the published one of Trager et al. (Tra95 (1995), hereafter T95).

### 2.2 The color-magnitude diagram

In order to build the CMD of the cluster, individual catalogs were matched by adopting a searching radius of 5 $`\mathrm{}`$, and keeping the matched object with the smallest distance. The derived CMD is shown in Fig. 3 for two annular regions: the inner one, between 5$`\mathrm{}`$ and 12$`\mathrm{}`$ from the center (left-hand panel) and the outer one, referring to the background, between 60$`\mathrm{}`$ and 67$`\mathrm{}`$ (right-hand panel). The M~92 turn-off region, as well as part of the horizontal branch, are clearly visible. At bright magnitudes, the giant branch turns to the blue, due to plate saturation. At large angular distances from the center of M~92, most objects are galactic stars with only a small contribution from the cluster.

For reducing the background/foreground field contribution, we used an approach similar to that of Grillmair et al. Gri95 (1995). First of all, We selected an annular region ($`200\mathrm{}<r<300\mathrm{}`$) around the cluster center to find the best fiducial CMD sequence of the cluster stars. Then, the CMD of this region was compared with the CMD of the field at a distance greater than $`1^{}`$ from the cluster center. The two CMD’s were normalized by their area and then we binned the CMD and computed the $`S/N`$ ratio of each element, just like in G95 (their Eq. 2). Finally, we then obtained the final contour of the best CMD region by cutting at a $`S/N1`$. Contours are shown in Fig. 3, on which a solid line marks the CMD region used to select the “bona fide” cluster stars as described above. We must say that this CMD selection is not aimed at finding all the stars in the cluster but only at the best possible enhancement of the cluster stars as compared to the field stars. This is why the region of the sub-giant/giant branch is not included, since here the cluster stars are fewer than in the field. By extracting objects at any distance from the center of the cluster, in the selected CMD region, the field contamination is reduced by a factor of $`7`$. In absence of strong color gradients, the fraction of lost stars does not depend on the distance from the center.

## 3 Extra-tidal excess in M~92

### 3.1 Radial density profile

As first step, we built a 2-D star density map by binning the catalog in step of $`1\mathrm{}\times 1\mathrm{}`$. Then, we fitted a polynomial surface to the background, selecting only the outermost regions of the studied area. The background correction is expressed in the same units of 2-D surface density counts, and can be direclty applied to the raw counts. A tilted plane was sufficient to interpolate the background star counts (SCs). Higher-order polynomials did not provide any substantial improvement over the adopted solution. We compared the fitted background with IRAS maps at 100$`\mu `$, but we did not find any direct signs of a correlation between the two. Rather, the direction of the tilt is consistent with the direction of the galactic center. Hence, the tilt of the background, which is however very small ($`0.01\mathrm{mag}/\mathrm{arcmin}`$), can be considered as due to the galactic gradient.

The cluster radial density profile was obtained from the background subtracted SC’s by counting stars in annuli of equal logarithmic steps. The uncorrected surface density profile (hereafter SDP) is expressed as:

$$SDP_i^{\mathrm{uncorr}}=2.5log(N_{i,i+1}/A_{i,i+1})+const,$$

where $`N_{i,i+1}`$ indicates the number of objects in the annulus between $`r_i`$ and $`r_{i+1}`$, and $`A_{i,i+1}`$ the area of the annulus. The constant was determined by matching the profile with the published profile of T95 in the overlap range. The effective radius at each point of the profile is given by:

$$r_i^{\mathrm{eff}}=\sqrt{\frac{1}{2}\times (r_i^2+r_{i+1}^2)}.$$

The SDP must now be corrected for crowding. When dealing with photographic material, it is not possible to apply the widely known artificial star technique used with CCD data. Therefore, we used a procedure similar to the ones described in Lehman & Scholz (LS97 (1997)) and Garilli et al. (Gar99 (1999)): we estimated the area occupied by the objects in each radial annulus by selecting all the pixels brighter than the background noise level plus three sigmas, and considered as virtually uncrowded the external annuli in which the percentage (very small, $`0.5\%`$) of filled area did not vary with the distance. The external region starts at $`1000\mathrm{}`$ from the cluster center with a filling factor smaller than $`2\%`$. After correcting the area covered by non-stellar objects, the ratio unfilled/filled area gives the crowding correction. This correction was computed at the effective radii of the surface brightness profile (hereafter SBP) and smoothed using a spline function. The corrected surface brightness profile was then computed as:

$$SBP_i=SBP_i^{\mathrm{uncorr}}+2.5\times \mathrm{log}(1.0frac)$$

where $`frac`$ is the crowding correction factor determined at the i$`th`$ point on the profile.

The crowding corrected SBP of M~92 derived from DPOSS data is shown in Fig. 4 as filled dots and the uncorrected counts as crosses. Table 1 lists the measured surface brightness profile. With a simple number counts normalization we joined our profile to the one (open circles) derived by T95, in order to extend the profile to the inner regions. We then fitted a single-mass King model to our profile. The fitting profile is drawn on Fig. 4 as a continuous line. Our value for the tidal radius, $`r_\mathrm{t}=740\mathrm{}`$, turned out to be similar to the value given in Brosche et al. (Bro99 (1999)), $`r_\mathrm{t}=802\mathrm{}`$ and slightly smaller than the one given in T95, $`r_\mathrm{t}=912\mathrm{}`$. As it can be seen from the figure, DPOSS data extend at larger radial distances than the T95 compilation and reveal the existence of a noticeable deviation from the isotropic King model derived from the direct fitting of the SBP. This deviation is a clear sign of the presence of extra tidal material. We also tried fitting anisotropic King models to the SBP, but the fit was not as good as in the isotropic case.

At what level is this deviation significant? The determination of the tidal radius of a cluster is still a moot case. While fitting a King model to a cluster density profile, the determination of the tidal radius comes from a procedure where the overall profile is considered, and internal points weigh more than external ones. On the one hand, this is an advantage since the population near the limiting radius is a mix of bound stars and stars on the verge of being stripped from the cluster by the Galaxy tidal potential. On the other hand, the tidal radius obtained in this way can be a poor approximation of the real one. In the classical picture, and in presence of negligible diffusion, the cluster is truncated at its tidal radius at perigalacticon (see Aguilar et al. Agu88 (1988)). Nevertheless, Lee & Ostriker (Lee87 (1987)) pointed out that mass loss is not instantaneous at the tidal radius, and, for a given tidal field, they expect a globular cluster to be more populated than in the corresponding King model. Moreover, a globular cluster along its orbit also suffers from dynamical shocks, due to the crossing of the Galaxy disk and, in case of eccentric orbits, to close passages near the bulge, giving rise to enhanced mass-loss and, later on, to the destruction of the globular cluster itself. Gnedin & Ostriker (Gne97 (1997)) found that, after a gravitational shock, the cluster expands as a whole, as a consequence of internal heating. In this case, some stars move beyond the tidal radius but are not necessarily lost, and are still gravitationally bound to the cluster. This could explain observed tidal radii larger than expected for orbits with a small value of the perigalacticon. Brosche et al. (Bro99 (1999)) point out that the observed limiting radii are too large to be compatible with perigalacticon $`r_\mathrm{t}`$, and suggest that the appropriate quantity to be considered is a proper average of instantaneous tidal radii along the orbit. It can be seen from Fig. 4 that the cluster profile deviates from the superimposed King model before the estimated tidal radius, and has a break in the slope at about $`r850\mathrm{}`$, after which the slope is constant. We shall come back later to this point.

In Fig. 5 we show the surface density profile, expressed in number of stars to allow a direct comparison with J99, and binned in order to smooth out oscillations in the profile, due to the small S/N ratio arising with small-sized annuli. J99 predict that stars stripped from a cluster, and forming a tidal stream, show a density profile described by a power law with exponent $`\gamma =1`$. We fitted a power law of the type $`\mathrm{\Sigma }(r)r^\gamma `$ to the extra-tidal profile (dashed line). The best fit gives a value $`\gamma =0.85\pm 0.08`$ and is shown as a dashed line in Fig. 5. The errors on the profile points include also the background uncertainty, in quadrature, so that the significance of the extra-tidal profile has been estimated in terms of the difference $`f_i3\sigma _i`$, where $`f_i`$ is the number surface density profile at point $`i`$, and $`\sigma _i`$ its error, which includes the background and the signal Poissonian uncertainties. This quantity is positive for all the points except for the outermost one. The fitted slope is consistent with the value proposed by J99 and in good accordance with literature values for other clusters (see G95 and Zaggia et al. Zag97 (1997)).

We then fit an ellipse to the extra-tidal profile in order to derive a position angle of the tidal extension, and checked whether the profile in that direction differs from the one obtained along the minor axis of the fitting ellipse, to confirm that the extra-tidal material is a tail rather than a halo. The best fitting ellipse, made on the “isophote” at the $`2\sigma `$ level from the background (approximately $`1.5r_\mathrm{t}`$ from the center of the cluster), turned out to have a very low ellipticity, ($`e0.05\pm 0.01`$ at P.A.$`54^{}\pm 15^{}`$). We have also measured the radial profiles along the major and minor axes, using an aperture angle of $`\pm 45^{}`$, in order to enhance the S/N ratio in the counts. The two profiles turned out to be indistinguishable within our uncertainties. This result shows that the halo material has a significantly different shape than the internal part of the cluster which shows an ellipticity of $`0.10\pm 0.01`$ at P.A. $`141^{}\pm 1`$ as found by White & Shawl (Whi87 (1987)).

### 3.2 Surface density map

In the attempt to shed more light upon presence and characteristics of the extra-tidal extension, we used the 2-D star counts map, as described at the beginning of the previous section. We applied a Gaussian smoothing algorithm to the map, in order to enhance the low spatial frequencies and cut out the high frequency spatial variations, which contribute strongly to the noise. We smoothed the map using a Gaussian kernel of $`6\mathrm{}`$. The resulting smoothed surface density map is shown in Fig. 6. Since the background absolute level is zero, the darkest gray levels indicate negative star counts. In this image, the probable tidal tail of M~92 (light-gray pixels around the cluster) is less prominent than in the radial density profile: this is because data are not averaged in azimuth. On the map we have drawn three “isophotal” contours at 1, 2 and 3$`\sigma `$ over the background. The fitted tidal radius is marked as a thick circle and the two arrows point toward the galactic center (long one) and in the direction of the measured proper motion (see Dinescu et al. Din99 (1999)). The tidal halo does not seem to have a preferred direction. A marginal sign of elongation is possibly visible along a direction almost orthogonal to that of the galactic center.

As pointed out in the previous section, if we build the profile along this direction and orthogonally to it, we do not derive clear signs of any difference in the star count profiles in one direction or the other, mainly because of the small number counts.

On the basis of these results, we can interpret the extra-tidal profile of M~92 as follows: at radii just beyond the fitted King profile tidal radius, the profile resembles a halo of stars –most likely still tied up to the cluster or in the act of being stripped away. As the latter process is not instantaneous, these stars will still be orbiting near the cluster for some time. We cannot say whether this is due to heating caused by tidal shocks, or to ordinary evaporation: a deep CCD photometry to study the mass function of extra-tidal stars would give some indications on this phenomenon. At larger radii, the 1 $`\sigma `$ “isophote” shows a barely apparent elongation of the profile in the direction SW to NE, with some possible features extending approximately towards S and E. Although the significance is only at 1 $`\sigma `$ level, these structures are visible and might be made up by stars escaping the cluster and forming a stream along the orbit. As pointed out in Meylan & Heggie (Mey97 (1997)), stars escape from the cluster from the Lagrangian points situated on the vector connecting the cluster with the center of the Galaxy, thus forming a two-sided lobe, which is then twisted by the Coriolis force. A clarifying picture of this effect is given in Fig. 3 of Johnston (Jon98 (1998)).

## 4 Summary and conclusions

We investigated the presence and significance of a tidal extension of the brightness profile of M~92. The main results of our study are:

1. The presence of an extra-tidal profile extending out to $`0.5^{}`$ from the cluster center, at a significance level of $`3\sigma `$ out to $`r2000\mathrm{}`$. We found no strong evidence for preferential direction of elongation of the profile. This may imply that we are detecting the extra-tidal halo of evaporating stars, which will later form a tidal stream. Moreover, the tidal tail might be compressed along the line of sight –see, for instance, Fig. 18 of G95. In fact, G95 point out that tidal tails extend over enormous distances ahead and behind the cluster orbit, and the volume density is subject to the open-orbit analogous of Kepler’s third law: near apogalacticon, stars in the tidal tail undergo differential slowing-down, so that the tail converges upon the cluster. Actually, most models (e.g., Murali & Dubinsky Mur99 (1999)) predict that the extra-tidal material should continue to follow the cluster orbit and thus take the shape of an elongated tail, or a stream. The stream has been already revealed in dwarf spheroidal galaxies of the local group (Mateo et al. Mat98 (1998)), but whether the stream can also be visible in significantly smaller objects like globular clusters is currently a moot point.
2. By constructing the surface density map and performing a Gaussian smoothing, the low-frequency features are enhanced over the background. We find some marginal evidence for a possible elongation in the extra-tidal extention based on a visual inspection of this map. This elongation may be aligned in a direction perdendicular to the Galactic center, although we already know that the significance of this result is low; additional observations will be required to settle the issue. A similar displacement is described in Fig. 3 of Johnston (Jon98 (1998)).

Finally, we want to stress the power of the DPOSS material in conducting this kind of programs, either by using the standard output catalogs, as they come out from the processing pipeline, or the specific re-analysis of the digitized plate scans. In the future we will extend this study to most of the globular clusters present on the DPOSS plates.

###### Acknowledgements.

SGD acknowledges support from the Norris Foundation. We thank the whole POSS-II, DPOSS, and CRoNaRio teams for their efforts.


text:
# Shape and Motion of Vortex Cores in Bi2Sr2CaCu2O8+δ

## I introduction

The study of the vortex phases in high temperature superconductors (HTS’s) has lead to both theoretical predictions of several novel effects and experiments accompanied by challenging interpretations. The reasons are multiple. First, the unconventional symmetry of the order parameter — most likely $`d_{x^2y^2}`$ — leads to the presence of low-lying quasiparticle excitations near the gap nodes, which in turn has inspired the predictions of a nonlinear Meissner effect , of a $`\sqrt{H}`$ dependence of the density of states at the Fermi level near the vortex cores ($`N(0,𝐫)`$) , and of a four-fold symmetry of the vortices . However, the experimental evidence for the first two effects is still controversial , and scanning tunneling spectroscopy (STS) measurements on vortex cores have not shown any clear signature of a $`\sqrt{H}`$ dependence of $`N(0,𝐫)`$ . Concerning the four-fold symmetry, a tendency of square vortices was found in previous measurements , but inhomogeneities make it difficult to be decisive about it. Interestingly, a four-fold symmetry has been observed around single atom zinc impurities in Bi2Sr2CaCu2O8+δ (BSCCO), which are of a smaller size than the vortex cores .

A second reason is related to the interaction of vortices with pinning centers, responsible for the rich vortex phase diagram of the HTS’s . The pinning of vortices is mainly due to local fluctuations of the oxygen concentration , and facilitated by their highly 2D ”pancake” character. In BSCCO the areal density of oxygen vacancies per Cu-O double layer is in fact surprisingly large: $`10^{17}`$ m-2 , corresponding to an average distance between the oxygen vacancies of the order of 10 Å. This vortex pinning results for BSCCO in the absence of any regular flux line lattice at high fields, as demonstrated by both neutron diffraction  and STS  experiments. Moreover, since the distance between the oxygen vacancies is of the order of the vortex core size, one may expect that not only the vortex distribution, but also the vortex core shape will be dominated by pinning effects, and not by intrinsic symmetries like that of the order parameter. A detailed understanding of the interaction of vortices with pinning centers will thus be of importance to explain the lack of correspondence between theoretical predictions about the vortex shape and STS measurements.

Third, and again as a consequence of the anisotropy of the order parameter, low-energy quasiparticles are not truly localized in the vortex core (contrarily to the situation in $`s`$-wave superconductors ). For pure $`d`$-wave superconductors these quasiparticles should be able to escape along the nodes of the superconducting gap. Thus in the vortex core spectra one expects a broad zero-bias peak of spatially extended quasiparticle states . However, tunneling spectra of the vortex cores in YBa2Cu3O7-δ (YBCO) showed two clearly separated quasiparticle energy levels, which were interpreted as a signature of localized states . In BSCCO two weak peaks have been observed in the some vortex core spectra , suggesting a certain similarity to the behaviour in YBCO. Another important characteristic of HTS’s that follows from the STS studies mentioned above, is the extremely small size of the vortex cores in these materials. The large energy separation between the localized quasiparticle states directly implies that the vortex cores in YBCO are of such a size that quantum effects dominate. This is even more true in BSCCO: not only the in-plane dimensions of the vortex cores are smaller than in YBCO and become of the order of the interatomic distances , but due to the extreme anisotropy of the material also their out-of-plane size is strongly reduced. This highly quantized character of vortices in HTS’s is equally demonstrated by the non-vanishing magnetic relaxation rate in the limit of zero temperature, attributed to quantum tunneling of vortices through the energy barriers between subsequent pinning centers .

In this paper we present a detailed STS study of the shape of the vortices in BSCCO. We will show that this shape is influenced by inhomogeneities. The samples presented here, which we characterize as moderately homogeneous, are used to study the behaviour of vortex cores under these conditions. Apart from the vortex core shape, this also includes the evolution in time of the vortices. We will show that both effects can be related to tunneling of vortices between different pinning centers. This is another indication of the possible extreme quantum behaviour of vortex cores in HTS’s. A corollary of this paper is that only extremely homogeneous samples will show intrinsic shapes of vortex cores.

## II Experimental Details

The tunneling spectroscopy was carried out using a scanning tunneling microscope (STM) with an Ir tip mounted perpendicularly to the (001) surface of a BSCCO single crystal, grown by the floating zone method. The crystal was oxygen overdoped, with $`T_c=77`$ K, and had a superconducting transition width of 1 K (determined by an AC susceptibility measurement). We cleaved in situ, at a pressure $`<10^8`$ mbar, at room temperature, just before cooling down the STM with the sample. The sharpness of the STM tip was verified by making topographic images with atomic resolution. Tunneling current and sample bias voltage were typically 0.5 nA and 0.5 V, respectively. We performed the measurements at 4.2 K with a low temperature STM described in Ref. , and those at 2.5 K with a recently constructed 3He STM . A magnetic field of 6 T parallel to the $`c`$-axis of the crystal was applied after having cooled down the sample. The measurements presented here were initiated 3 days after having switched on the field.

The $`dI/dV`$ spectra measured with the STM correspond to the quasiparticle local density of states (LDOS). In the superconducting state one observes two pronounced coherence peaks, centered around the Fermi level, at energies $`\pm \mathrm{\Delta }_p`$. The gap size $`\mathrm{\Delta }_p`$ varied from 30-50 meV. In the vortex cores the spectra are remarkably similar to those of the pseudogap in BSCCO measured above $`T_c`$ , with a total disappearance of the coherence peak at negative bias, a slight increase of the zero bias conductivity, and a decrease and shift to higher energy of the coherence peak at positive bias. To map the vortex cores we define a gray scale using the quotient of the conductivity $`\sigma (V_p)=dI/dV(V_p)`$ at a negative sample voltage $`V_p=\mathrm{\Delta }_p/e`$ and the zero bias conductivity $`\sigma (0)=dI/dV(0)`$. Thus we obtain spectroscopic images, where vortex cores appear as dark spots. Since we measure variations of the LDOS, which occur at a much smaller scale (the coherence length $`\xi `$) than the penetration depth $`\lambda `$, we can get vortex images at high fields. A tunneling spectrum is taken on the time scale of seconds, spectroscopic images typically take several hours (about 12 hours for the images of 100x100 nm2 presented below). The images therefore necessarily reflect a time-averaged vortex density.

In all large-scale images we have suppressed short length-scale noise by averaging each point over a disk of radius $`20`$ Å. When zooming in to study the shape of individual vortices, we strictly used raw data. Further experimental details can be found in previous publications .

## III Results

### A Vortex Distribution

In Fig. 1 we show spectroscopic images of the surface of a BSCCO crystal, at different magnetic field strengths. The large dark structure, clearly visible at the right of Figs. 1(b) and (c), corresponds to a degraded region resulting from a large topographic structure, already observed in the topographic image Figs. 1(a). The presence of this structure allows an exact position determination throughout the whole experimental run. As can be seen in Fig. 1, the number of vortices at 6 and at 2 T, in exactly the same region, scales very well with the total number of flux quanta ($`\mathrm{\Phi }_0`$) that one should expect at these field strengths. This clearly proves that the observed dark spots are directly related to vortex cores, and not to inhomogeneities, defects or any form of surface degradation. The large spot in the upper left corner of Fig. 1(c) forms an exception: it appeared after a sudden noise on the tunnel current while we were scanning on that position, showed semiconducting spectra (typical for degraded tunneling conditions) afterwards, and remained even after having set the external field to 0 T. One should however not exclude that a vortex is pinned in this degraded zone. Finally, the size and density of the vortices are fully consistent with previous measurements .

Instead of a well ordered vortex lattice, one observes patches of various sizes and shapes scattered over the surface. This clearly indicates the disordered nature of the vortex phase in BSCCO at high fields, again in consistency with previous STM studies  and neutron scattering data , and stressing the importance of pinning for the vortex distribution.

### B Vortex Shapes

As a next step we increase the spatial resolution in order to investigate individual vortex cores. Some vortices appear with square shapes, but most vortices in this study have irregular shapes. Closer inspection of the tunneling spectra reveals small zones inside the vortex core that show superconducting behaviour. That is, when scanning through a vortex core one often observes (slightly suppressed) coherence peaks (Fig. 2(a)), typical for the superconducting state, at some spots inside the vortex core. The latter is generally characterized by the absence of these peaks. In some cases, the vortex cores are even truly split into several smaller elements (Fig. 2(b)), totally separated by small zones showing the rise of coherence peaks. This has been verified by measuring the full spectra along lines through the vortex core, as in Fig. 2(a).

The smaller elements of a split vortex core cannot be related to separate vortices: first, the vortex-vortex repulsion makes it highly improbable that several vortex cores are so close to each other; second, counting all these elements as a flux quantum in Fig. 1(b), one finds a total flux through the surface that is far too large compared to the applied field. One should note here that the magnetic size of a flux line is of the order of the penetration depth $`\lambda `$, two orders of magnitude larger than the vortex core splitting observed here.

### C Vortex Motion

With subsequent spectroscopic images like Fig. 1(b), one can also study the vortex distribution as a function of time. We expect the vortex motion to be practically negligible, since we allowed the vortices to stabilize for more than 3 days . However, in Fig. 3 one can see that many vortices still have not reached totally stable positions. Many of them roughly stay on the same positions over the time span of our measurement, but others move to neighboring positions. Five different cases of moving vortices are indicated by the ellipses and the rectangle in Fig. 3.

In the panels on the left side the precise intensity of each point is difficult to read out directly. In order to investigate more quantitatively the time evolution of the vortex distribution, from one frame to the next, we show in the right part of Fig. 3 3D representations of the area that is marked by the rectangle in the 2D spectroscopic images. They give an idea of the gray scale used in the 2D images, and provide a detailed picture of the movement of the vortex core in front, from the right in Fig. 3(a) to the left in Fig. 3(c). The vortex core at the back does not move, and serves as a reference for the intensity. We remind that the intensity, or height in the 3D images, is a measure of the LDOS, which in a vortex core is different from the superconducting DOS. It is most interesting to see what happens in Fig. 3(b): the (moving) vortex core is divided between two positions. Thus, the vortex core moves from one position to the other, passing through an intermediate state where the vortex splits up between the two positions. Note that these two positions do not correspond to two vortices. In fact, the split vortex is characterized by the lower intensity compared to the nearby (reference) vortex. This means that the coherence peak at negative voltage does not completely disappear, as it should if we had a complete and stable vortex at each of these positions. Note also that the density of vortices around the rectangular area on the left side in Fig. 3 will clearly be too high if we count the mentioned positions and all positions in the ellipses as individual flux quanta. The split vortex discussed here is not a unique example. Similar behaviour can be found for several other vortex cores, as indicated by the ellipses in Fig. 3. This gradual change of position is in striking contrast to the STS observations of moving vortices in NbSe2  and YBCO .

### D Temperature Dependence

We performed measurements both at 4.2 and at 2.5 K, on samples cut from the same batch of crystals. The data taken at 2.5 K (see also Fig. 2) are fully consistent with the presented work at 4.2 K. In Fig. 4 we provide a general view of the vortex cores at 2.5 T, including an analogue of the moving vortex core of Fig. 3. Though it is hard to obtain any quantitative data, one can conclude that the vortex cores roughly have the same size, similar irregular shapes, and examples of split vortex cores can be easily found.

## IV Discussion

### A Experimental Considerations

The observation of such a highly irregular pattern of vortex cores, as presented above, requires a careful analysis of the experimental setup. However, the fact that keeping exactly the same experimental conditions the number of vortex cores scales with the magnetic field, is a direct proof of the absence of artificial or noise-related structures in the spectroscopic images. Furthermore, since topographic images showed atomic resolution, there is no doubt that the spatial resolution of the STM is largely sufficient for the analysis of vortex core shapes.

The stability of the magnetic field can be verified by counting the number of vortices in the subsequent images at 6 T (Fig. 3). Since, excluding the split vortices marked by the ellipses, this number is constant ($`26\pm 3`$), we can exclude any substantial long time-scale variation of the magnetic field. Some variation in the total black area from one image to the other can be related to the tunneling conditions: a little more noise on the tunnel current will give a relatively large increase of the small zero-bias conductance. Since we divide by the zero-bias conductance to obtain the spectroscopic images, this may lead to some small variations in the integrated black area of the images.

### B Delocalization

Keeping in mind the randomness of the vortex distribution at 6 T due to pinning of vortices, we now relate both the split vortex cores (Fig. 2) and the intermediate state between two positions (Fig. 3 and Fig. 4) to the same phenomenon: the vortex cores appear to be delocalized between different positions which correspond to pinning potential wells, and during the measurement hop back and forth with a frequency that is too high to be resolved in this experiment. According to this analysis not only the distribution, but also the observed shape of the vortex cores is strongly influenced by pinning.

The pinning sites most probably result from inhomogeneities in the oxygen doping, which are thought to be responsible for the variations of the gap size (see experimental details). The distance over which the vortices are split corresponds to the average spacing between oxygen vacancies ($`10100`$ Å ). We did not observe any sign of resonant states related to impurities, as in recent STM experiments on BSCCO . The driving forces causing vortex movements in Fig. 3 and Fig. 4 are most probably due to a slow variation of the pinning potential, resulting from the overall rearrangement of vortices.

The vortex delocalization and movement presented here can directly be connected to the vortex creep as measured in macroscopic experiments, like magnetic relaxation . The main difference, of course, is that we do not observe whole bundles of vortices moving over relatively large distances, but only single vortex cores that are displaced over distances much smaller than the penetration depth $`\lambda `$. That is, it will not be necessary to displace whole groups of vortices, many of which might be pinned much stronger than the delocalized vortices we observe. A second difference is the absence of a uniform direction of the movements in the STM images, most probably because the Lorentz driving forces have been reduced to an extremely small value (which also follows from the very gradual changes in Fig. 3 and Fig. 4).

### C Thermal Fluctuations versus Quantum Tunneling

Regarding now the mechanism responsible for the vortex delocalization, the main question is whether we are dealing with thermal fluctuations, or quantum tunneling between pinning potential wells. In fact magnetic relaxation measurements on BSCCO show a crossover temperature from thermal to quantum creep of $`25`$ K , which means that with these STM measurements we are on the limit between the two.

In the case of thermally induced motion, there is a finite probability for the vortex to jump over the energy barrier between the two potential wells. The vortex is continuously moving from one site to the other, with a frequency that is too high to be resolved by our measurements. In the case of quantum tunneling, the vortex is truly delocalized. That is, the vortex can tunnel through the barrier, and one observes a combination of two base states (i.e. positions), like in the quantum text book example of the ammonia molecule . Thermal fluctuations will lead to a continuous dissipative motion  between the two sites; quantum tunneling gives a dissipationless state in which the vortex is divided between two positions.

An instantaneous observation of several base states of a quantum object would be impossible, since each measurement implies a collapse of the quantum wave function into one state. However, the STM gives only time averaged images, and with the tunneling current in this experiment we typically detect one electron per nanosecond. If the vortex core relaxes back to its delocalized state on a time scale smaller than nanoseconds, the vortex can appear delocalized in the STM images. Moreover, it should be clear that the long time ( 12 hours) between the subsequent images in Fig. 3 and 4 has nothing to do with the vortex tunneling time; it is tunneling of the vortex that allows the intermediate state. The creep of vortices (either by quantum tunneling or by thermal fluctuations) is a slow phenomenon here. At a given region the pinning potential due to inhomogenieties and interactions with other vortices evolves on a time scale of hours, shifting the energetically most favorable position from one site to the other. However, the tunneling occurs much faster ($`<`$ns) than this slow potential evolution, creating the possibility of a superposition of both states (positions), each with a probability depending on the local value of the pining potential. Following the analogy of the ammonia molecule: a moving vortex core corresponds to an ammonia molecule submitted to an external external field (”overall pinning potential”) that is slowly changing. Initially, due to this external field the state with the hydrogen atom at the left is more favorable, then the field changes such that left and right are equally stable (”split vortex”), and finally the right state is most probable: the hydrogen atom has moved from left to right on a long time scale, whereas the tunneling itself occurs on a much faster time scale. By the way, since the change of the local pinning potential results from a rearrangement of all surrounding vortices (over a distance $`\lambda `$), the time scale of this change will still be strongly dependent on the (short) tunneling time itself.

In order to discuss the possibility of quantum tunneling, one should consider the tunneling time and the for our measurements negligible temperature dependence . The importance of the tunneling time is two-fold: the tunneling rate is strongly dependent on the time needed to pass through the pinning barriers (an effect which will be extremely difficult to measure directly); and in our measurements the tunneling time must be faster than the probe response time (as explained in the previous paragraph). From collective creep theory  one can get an order-of-magnitude estimate for the tunneling time $`t_c\mathrm{}/U(S_E/\mathrm{})10^{11}`$ s, with the Euclidian action for tunneling $`S_E/\mathrm{}10^2`$ and the effective pinning energy $`U10^2`$ K derived from magnetic relaxation measurements . This is clearly below the upper limit of 1 ns set by the probe response time.

For a discussion about the implications of the temperature independence of quantum tunneling, one should first consider the temperature dependence expected for vortex movements that result from thermal fluctuations. Thermally induced hopping between different pinning sites should be proportional to $`\mathrm{exp}(U/k_BT)`$, where $`U`$ is again the effective pinning energy . From magnetic relaxation measurements on BSCCO one can derive a value of about $`1010^3`$ K for this quantity . Assuming for the moment that this $`U`$ determines the hopping of individual vortices, it should then be compared to the Euclidian action for quantum tunneling, which with magnetic relaxation measurements is estimated to be $`S_E/\mathrm{}10^2`$ , and plays a role like $`U/k_BT`$ in the Boltzman distribution. For measurements presented here, it is important to note again that they were taken more than 3 days after having increased the field from 0 to 6 T. Since for $`B=6`$ T the induced current density $`j`$ relaxes back to less than $`0.01`$ of its initial value in about 10 seconds , we are clearly in the limit where $`j`$ and thus the Lorentz driving forces (which reduce the energy barrier for vortex creep) approach zero. This means that the effective pinning potential $`U`$ rises, if not to infinity like in isotropic materials, to a value which in principle is much higher than the one which determines vortex creep in magnetic relaxation measurements at comparable field strengths . With nearly zero Lorentz forces the tilt of the overall pinning potential will thus be small compared to the pinning barriers, making thermal hopping over the barriers highly improbable (at low temperatures). Quantum creep, in the limit of vanishing dissipation, is independent of the collective aspect of $`U`$, while the probability for thermal creep decreases as $`\mathrm{exp}(U/k_BT)`$ . So one can expect quantum creep to become more important than thermal creep when more time has passed after having changed the field. In other words, in spite of the fact that the tilt of the overall pinning potential is small compared to the pinning barriers, the vortices can still move a little, i.e. disappearing and reappearing elsewhere.

However, the collective $`U`$ may be higher than the pinning barriers for the individual vortex movements observed in our experiments. Thus, in order to find a lower bound for the latter, we also estimate $`U`$ for the moving vortices from our microscopic measurement. First we calculate the magnetic energy of a vortex due to the interaction with its nearest neighbors, using

$$E_{int}=d\frac{\mathrm{\Phi }_0^2}{8\pi ^2\lambda ^2}\underset{i}{}\{ln(\frac{\lambda }{r_i})+0.12\},$$
(1)
where $`d`$ is the length of the vortex segment, $`\mathrm{\Phi }_0`$ is the flux quantum, $`\lambda `$ the in-plane penetration depth and $`r_i`$ the distance to its $`i`$th neighbor . Parameters are conservatively chosen such to give a true minimum estimate for $`E_{int}`$ (and thus for $`U`$): we restrict the out-of-plane extent of the vortices to zero and thus only take $`d=15`$ Å, the size of one double Cu-O layer  (”pancake vortices”), and for $`\lambda `$ take the upper bound of different measurements, 2500 Å . Taking the vortex in Fig. 3(b), and determining the positions between which it is divided as well as the positions of the neighboring vortices, one can find the difference between the magnetic interaction energies of the delocalized vortex at its two positions. We obtain $`E_{int}120`$ K. Now the absence of any vortex lattice indicates that the pinning potential wells are generally larger than the magnetic energy difference between the subsequent vortex positions, and Fig. 3(b) reflects a vortex state that is quite common in our measurements (Fig. 1). Following these arguments one can safely assume that the effective potential well pinning the vortex in Fig. 3 is larger than this difference: $`U>E_{int}=120`$ K. in agreement with the estimates given above. So we obtain $`U/k_BT>1010^2`$ for temperatures around 4 K. In the limit of zero dissipation $`S_E/\mathrm{}(k_F\xi )^2`$. On the basis of STS experiments  this can be estimated to be $`10`$. This value is smaller than the one quoted above, and suggests that quantum tunneling is dominant in our measurements.

The most direct evidence for quantum creep can be obtained from measurements at different temperatures. The hopping rate for thermally induced movements is given by $`\omega _0\mathrm{exp}(U/k_BT)`$, where $`U`$ is the pinning potential, and $`\omega _0`$ the characteristic frequency of thermal vortex vibration . Assuming $`U=100`$ K, and a conservatively large estimate of $`\omega _010^{11}`$ s-1, the hopping rate should drop from 1 s-1 to 10-7 s-1 on cooling from 4.2 to 2.5 K. This gives a huge difference between the respective measurements at these temperatures. However, spectroscopic images at 4.2 and 2.5 K show the same pattern of moving and delocalized vortices. Following the same kind of estimations as above, the delocalized vortex at 2.5 K (Fig. 4) gave $`U>210`$ K, which makes thermal creep even more unlikely here. Even if the frequency of the individual thermal vortex movements were too high to be resolved by our measurements both at 4.2 and at 2.5 K (this would mean a rather unrealistic characteristic frequency $`\omega _0>10^{15}`$), one would still expect to see a difference. As a matter of fact, the driving force for the vortex movements results from an overall rearrangement of vortices. This means that the displacements of vortices will always depend on the hopping frequency, and that even for very high hopping rates one should observe a reduction of the number of vortices that are displaced in our images, when the hopping rate is reduced by a factor 107.

## V Conclusion

We observed vortex cores that were delocalized over several pinning potential wells. Regardless of the exact mechanism (thermal hopping or quantum tunneling) responsible for this delocalization, our measurements point out that pinning effects not only dominate the distribution of the vortex cores, but also their shape. As a consequence intrinsic (four-fold?) symmetries of the vortex cores will be obscured in microscopic measurements. The delocalization of the vortex cores implies that the vortex cores in this study appear larger than their actual — unperturbed — size, indicating a coherence length that is even smaller than was expected on the base of previous studies .

The analysis given above strongly favors an interpretation in terms of quantum tunneling of vortex cores. This would not only mean the first microscopic signature of the vortex quantum tunneling as derived from magnetic relaxation measurements, it is also a further indication  that objects of larger size and complexity than one or several atoms can appear as a superposition of different quantum states.

###### Acknowledgements.

This work was supported by the Swiss National Science Foundation.