text:
# 1 The Apparent Contradiction

## 1 The Apparent Contradiction

According to the Holographic Principle  objects deep in the interior of a spatial region should have a description in terms of a holographic theory that in some sense lives on the region’s boundary. A concrete realization of this idea has been given by the AdS/CFT duality . It has become a subject of active investigation to find out exactly how particular objects in $`AdS`$ space are represented in the corresponding conformal field theory.

There are two apparently contradictory claims in the literature concerning this question. According to  the field theoretic representation of an object or event far from the $`AdS`$ boundary is through nonlocal operators such as Wilson loops whose degree of nonlocality increases as the object recedes from the boundary. This follows from the usual UV/IR correspondence .

In apparent contradiction with this view, Horowitz and Hubeny in an extremely interesting paper  have presented evidence that local operators of very high dimensionality contain information about the size and shape of small111By small we do not mean microscopic. The objects we have in mind may be macroscopic and classical as long as they are much smaller than the $`AdS`$ radius. objects. The resolution of this conflict will be seen to lie in an ambiguity in what we mean when we say that a bulk quantity is represented by a certain field theoretic quantity.

We will begin by very briefly describing the results of Horowitz and Hubeny. These authors consider objects in $`AdS_5\times S_5`$ which are much smaller than the AdS radius of curvature and which are localized at a point on the 5-sphere. In addition they are also localized near the center of $`AdS`$ in appropriately chosen coordinates. Using the gravitational dual theory they find that the expectation values of certain scalar operators of dimension $`n=l+4`$ are of order

$$\mathrm{\Phi }_ne^{c\rho l}$$
(1.1)
where $`\rho `$ is the size of the object in units of the $`AdS`$ radius and $`c`$ is a numerical constant which we will ignore. It follows from eq. (1.1) that for $`l>\rho ^1`$ the signal is appreciable and that by examining the $`l`$ dependence the size of the object can be determined. This apparently contradicts .

Our conventions for describing $`AdS`$ space are as follows: Global coordinates for $`AdS`$ can be defined so that the metric is given by

$$ds^2=\frac{R^2}{(1r^2)^2}\left\{(1+r^2)^2dt^24dr^24r^2d\mathrm{\Omega }^2\right\}$$
(1.2)
where the radial coordinate runs from $`0`$ to $`1`$. Here, $`R`$ is the radius of curvature of the space and $`d\mathrm{\Omega }^2`$ represents the metric of a unit sphere. In the case of $`AdS_5`$ the sphere is a $`3`$-sphere.

These coordinates are especially useful when trying to recover infinite flat space in the limit $`R\mathrm{}`$. Indeed the $`AdS`$ space as defined above behaves in many respects like a finite cavity of size $`R`$ with a reflecting boundary at $`r=1`$. We will refer to the metric in eq. (1.2) as cavity coordinates.

Another coordinate system which is particularly useful when studying the properties of the CFT in flat space is given by

$$ds^2=\frac{R^2}{y^2}(d\stackrel{~}{t}^2dx^2dy^2)$$
(1.3)
where $`\stackrel{~}{t},x`$ labels 4-dimensional Minkowski space and $`y`$ is the 5th direction perpendicular to $`x`$. At time $`\stackrel{~}{t}=0`$, the center of AdS can be taken to be the point $`x=0`$, $`y=1`$ in these co-ordinates.

The quantum field theory lives on the 4-dimensional boundary whose metric we take to be either

$$ds_b^2=dt^2d\mathrm{\Omega }^2$$
(1.4)
in the case of cavity coordinates, or

$$ds_b^2=d\stackrel{~}{t}^2dx^2$$
(1.5)
for the flat space representation.

As in ref.  we will be thinking of the boundary quantum field theory in Wilsonian terms. Thus we imagine the boundary field theory to be defined in terms of a bare set of degrees of freedom at some very small coordinate length scale $`\delta _0`$. The details of the cutoff are not important but an example to keep in mind is the Hamiltonian lattice cutoff sometimes used to study QCD. In that case $`\delta _0`$ would be the spatial lattice constant. We assume $`\delta _0`$ is much smaller than any other length scale we will encounter.

According to the UV/IR correspondence, a connection exists between the UV regulator length scale $`\delta _0`$ of the QFT and an IR cutoff of the bulk theory. The bulk cutoff is implemented by replacing the $`AdS`$ boundary at $`r=1`$ or $`y=0`$ with the surface $`r=1\delta _0`$ in global $`AdS`$ or $`y=\delta _0`$ in the Poincare patch.

There are two different large $`N`$ limits of the QFT that have two different purposes. The first is the ’t Hooft limit

$`N`$ $``$ $`\mathrm{}`$ (1.6)
$`g_sN`$ $`=`$ $`constant.`$ (1.7)
From the bulk point of view this is the limit of classical gravity or classical string theory in an $`AdS`$ space of fixed radius in string units;

$$R=(g_sN)^{1/4}l_s.$$
(1.8)
In this limit the ratio of the size of a physical object to the $`AdS`$ radius is fixed.

The limit of interest for analyzing the holographic principle as defined in  is a different one . For this purpose we take

$`N`$ $``$ $`\mathrm{}`$ (1.9)
$`g_s`$ $`=`$ $`constant.`$ (1.10)
In this limit the $`AdS`$ radius becomes much larger than the size of any physical object. This is the limit discussed in  where it was claimed that objects at $`r=0`$ or $`y=1`$ should be represented by Wilson loops of roughly unit size.

## 2 What it Means to Represent

One reason for confusion is that different people may mean different things when they say a certain field theory quantity represents a corresponding bulk quantity. In order to resolve the paradox raised by the Horowitz-Hubeny result we need to have a clear understanding of what it means for a particular set of observables in the CFT to describe a particular set of circumstances in the $`AdS`$ space.

The ability to find observables (hermitian operators) in the field theory to represent physical quantities in the bulk theory follows from the assumption that the Hilbert space of bulk states is the same as that of the boundary field theory. We argue that $`faithfully`$ representing a given bulk quantity $`\alpha `$ by a field theory quantity $`A`$ should mean more than just requiring a correspondence between their expectation values. Ideally we would like the probability distribution for $`\alpha `$ and $`A`$ to be the same. For example, a faithful representation of a highly classical bulk quantity such as the size of a macroscopic object should involve a field theory quantity with very small fluctuation.

Let us suppose that the quantum state of the system determines a probability distribution $`P(\alpha )`$ centered at $`\alpha _0`$ with a width $`\mathrm{\Delta }(\alpha )`$. Now consider a second state characterized by a second distribution $`P^{}(\alpha )`$ centered at $`\alpha _0^{}`$. These two states are clearly distinguishable if the two distributions do not overlap. In particular two different macroscopic classical configurations of $`\alpha `$ should have negligible overlap in their probabilities. A minimum condition for $`A`$ to faithfully represent $`\alpha `$ is that two probability distributions for $`A`$ will not overlap if the two corresponding distributions do not overlap for $`\alpha `$. In other words two configurations which disagree on the value of $`\alpha `$ must be represented by probability distributions in $`A`$ which are almost orthogonal. We will regard this to be a minimal requirement for a faithful representation of a bulk variable by a corresponding holographic variable.

## 3 The Resolution

Let us now ask whether the high dimension operators $`\mathrm{\Phi }_n`$ are a faithful representation of the size of an object at the center of $`AdS`$ space. From what we have said in the previous section the question comes down to whether or not the probability distributions for the $`\mathrm{\Phi }`$’s are orthogonal or almost orthogonal for two classically distinguishable values of the size $`\rho `$. We emphasize again that we are working in a Wilsonian framework where it is assumed that the field theory is defined by a concrete regularized system.

The $`\mathrm{\Phi }_n`$’s are defined to have vanishing vacuum expectation values. We must also specify a convention for normalizing them. We follow the same convention as in , namely the two point function $`\mathrm{\Phi }_n(x)\mathrm{\Phi }_n(x^{})`$ is of order one at unit coordinate separation. Now consider the width of the probability distribution for $`\mathrm{\Phi }_n`$, in other words the fluctuation $`\mathrm{\Delta }`$ in $`\mathrm{\Phi }_n`$:

$$\mathrm{\Delta }^2=\mathrm{\Phi }_n^2.$$
(3.1)
Obviously if the difference in expectation values of $`\mathrm{\Phi }_n`$ for two distinct configurations is much less than $`\mathrm{\Delta }`$ then these variables do not faithfully represent the variables they were intended to describe.

The scalar fields $`\mathrm{\Phi }_n`$ are given by

$$\mathrm{\Phi }_n=TrF^2X^l$$
(3.2)
where $`X`$ stands for the six fundamental scalars of maximally supersymmetric $`SU(N)`$ Super Yang-Mills Theory. The trace is over the adjoint representation of $`SU(N)`$ and $`X^l`$ represents a polynomial of order $`l`$ in the $`X^{}s`$. The dimension of $`\mathrm{\Phi }_n`$ is $`n=l+4`$. The operators are normal ordered meaning that their vacuum expectation value has been subtracted out. They are normalized so that their two point function at unit coordinate separation is of order one.

Now consider the fluctuation in $`\mathrm{\Phi }_n`$. This is given by the square root of the connected two point function at vanishing separation. In the continuum theory this fluctuation will be divergent. In the Wilsonian cutoff theory the fluctuation will be of order $`\delta _0^n`$ which we assume is extremely large 222This is true for the expectation value in any generic low energy state since the expectation value will be dominated by high energy intermediate states.. Thus unless these operators are somehow further regulated the fluctuation is divergent.

This is true for any local operator $`\mathrm{\Phi }`$. It means that $`\mathrm{\Phi }`$ can not faithfully describe anything. A measurement of $`\mathrm{\Phi }`$ gives completely random results in any state. This point was made forcefully in a famous paper by Bohr and Rosenfeld in the earliest days of quantum field theory. According to Bohr and Rosenfeld the correct observables for a quantum field theory are what we would today call “regulated” fields. This entails introducing a regulator scale $`\delta `$ chosen to be much larger than the cutoff scale $`\delta _0`$. The observables are defined by some form of smearing or point-splitting of the composite operators $`\mathrm{\Phi }`$. This will be discussed further in the next section.

The regulated fluctuation in $`\mathrm{\Phi }_n`$ is of order

$$\mathrm{\Delta }_n\delta ^{(l+4)}.$$
(3.3)
This follows from dimensional analysis and the fact that $`\mathrm{\Phi }_n`$ has mass dimension $`n=l+4`$.

Evidently from eq. (1.1), the condition that the expectation value of $`\mathrm{\Phi }_n`$ is larger than the fluctuation is

$$e^{c\rho l}\delta ^{(l+4)}.$$
(3.4)
Since $`\rho `$ is defined to be the size of an object measured in units of $`R`$, it will vanish in the limit $`R\mathrm{}`$. Thus we find that the inequality is satisfied only if

$$\delta 1.$$
(3.5)
In other words the operators $`\mathrm{\Phi }_n`$ must not only be regulated but the regulator scale has to be comparable to the coordinate distance of the object from the $`AdS`$ boundary. The meaning of this is clear. For an operator to faithfully represent a property of a small object near the center of $`AdS`$, it must be non-local as described in .

## 4 Regulating $`\mathrm{\Phi }`$

Granted that we must regulate the operators $`\mathrm{\Phi }`$, the question arises as to exactly how to do so. We will begin with an implicit construction. The problem with the unregulated operators is that they have large matrix elements connecting two very high energy states. In calculating their fluctuation most of the contribution comes from these high energy intermediate states. We can easily regulate the operators by simply throwing away the matrix elements between states whose energies differ by more than $`\delta ^1`$. Equivalently we can integrate the operators over time with a smooth test function with support over a time interval $`\delta `$. By solving the equations of motion we can express the regulated operator in terms of operators at a fixed time. The result will be spatially nonlocal over a scale $`\delta `$. As we have seen in the previous section, $`\delta `$ must be $`1`$ so that the regulated operator is nonlocal over the entire boundary sphere.

On the other hand smearing an operator over time will not change its expectation value in a time independent configuration. Thus, for such configurations, the local operator and its nonlocal counterpart have the same expectation value. This accounts for the results in . However, for time dependent configurations such as those described in  only the nonlocal operator faithfully represents the relevant instantaneous property of a small object near $`r=0`$.

The field theory of interest in this paper is a gauge theory in which all fundamental fields are in the adjoint representation. If the bare theory is a Hamiltonian lattice gauge theory, then any operator at a fixed time can be expressed in terms of generalized Wilson loops in which the Wilson loop is “decorated” with insertions of adjoint fields. The regulated operators will be expressible as linear superpositions of such Wilson loops of size $`\delta `$. Horowitz and Hubeny provide important information on how to decorate the Wilson loop in order to describe particular features of small objects.

Another important example concerns the “precursors” described in . Suppose that an event takes place near the center of $`AdS`$ which results in the emission of a wave propagating towards the boundary. Bulk causality ensures that all local supergravity fields evaluated within a neighborhood of the boundary will retain their original expectation values until the wave itself arrives at the boundary. Therefore, for a period of time of order one, all local QFT operators corresponding to the bulk supergravity fields will retain their original expectation values carrying no information about the wave. At some time, when the wave arrives at the boundary ($`t=0`$), some local operators in the QFT will begin to oscillate. According to the AdS/CFT correspondence, their expectation value at $`t=0`$ will be given by the boundary data of the wave. Furthermore, their expectation value will be insensitive to the $`R\mathrm{}`$ limit and proportional to the amplitute of the wave. Thus, in regulating the local operators at $`t=0`$ so as to keep the signal bigger than their fluctuations, we only need to introduce a cutoff of order the width of the wave pulse. Now to find the non-local “precursors” describing the wave at an earlier time, when say the wave is at co-ordinate distance $`\delta `$ from the boudary, we use the equations of motion to express the regulated local operators at $`t=0`$ in terms of operators at $`t=\delta `$. Then, the “precursors” will be spatially non-local over a scale $`\delta `$. The results of  suggest that the resulting non-local operator will involve superpositions of Wilson loops of size $`\delta `$.

A point worth mentioning involves the possibility of constructing operators with small fluctuation by spatially averaging $`\mathrm{\Phi }`$ over $`\mathrm{\Omega }`$. It is not hard to see that this diminishes its fluctuation by a factor $`\delta ^{3/2}`$. This would have no important effect on our conclusion.

Finally we want to emphasize that expectation values are not the observables of a system. The observables representing the results of measurements have uncertainties. A correct representation of a variable should not only represent its expectation value but also its entire probability distribution. The wild fluctuation of local fields makes them bad representations of weakly fluctuating positions and sizes of macroscopic objects.

## 5 Acknowledgements

We would like to acknowledge G. Horowitz and S. Thomas for useful discussions. The work of V. Hubeny was supported in part by NSF grants PHY-9507065 and PHY-0070895. The work of L. Susskind and N. Toumbas was supported in part by NSF grant PHY-9870115.


format: text

text: # 1 2


text:
# A Novelty-based Evaluation Method for Information Retrieval

## 1. Introduction

In information retrieval (IR) research, the notion of precision and recall have commonly been used to evaluate the empirical performance of systems \[Keen, 1992, Salton, 1992\]. Precision is the ratio of the number of relevant documents retrieved by a system under evaluation, compared to the total number of documents retrieved by the system. On the other hand, recall is the ratio of the number of relevant documents retrieved by the system, compared to the total relevant documents in a given benchmark test collection.

In other words, the precision/recall-based evaluation method regards all the relevant documents as equally important or informative for the user, and thus highly values systems that retrieve as many relevant documents as possible, with little noise.

However, in the real world, where a number of IR systems are available, for example, on the World Wide Web, it is often the case that the user has already read some of relevant documents using other systems. Thus, systems that always retrieve relevant documents similar to those retrieved by ubiquitous systems have little practical utility. In addition, meta search systems, which integrate document sets retrieved by more than one system, are less effective, in the case where individual systems retrieve similar documents.

In view of these problems, our proposed IR evaluation method favors systems that retrieve more novel documents, that is, relevant documents which cannot be retrieved by other existing systems.

From a different perspective, our evaluation method is also effective in producing test collections. The pooling method \[Voorhees, 1998\], which has commonly been used to produce test collections, requires a variety of participating systems. However, in the case where most participating systems adopt similar techniques, it is not feasible to collect a sufficient “pool” (i.e., a set of candidates for relevant documents). Our evaluation method is expected to promote a development of IR systems with various concepts, and therefore resolve the above problem.

Section 2. formalizes the evaluation measure based on the novelty of documents, and Section 3. applies this measure to evaluate IR systems that participated in the IREX workshop \[Sekine and Isahara, 1999\].

## 2. Formalizing the Measure

Instead of the notion of precision and recall, we propose as a new evaluation measure the utility of system $`x`$ with respect to relevant document $`d`$, $`U_d(x)`$. This measure denotes the extent to which $`x`$ contributes to providing the user with $`d`$, for a given query. Note that in this paper, $`d`$ generally refers to a relevant document.

From an information theoretical point of view, we calculate $`U_d(x)`$ as the ratio of the probability that the user reads document $`d`$ by using system $`x`$, $`P(D=d|S=x)`$, compared to the probability that the user reads $`d`$ by using another system (i.e., even without using $`x`$), $`P(D=d)`$, as shown in Equation (1).

$$U_d(x)=\mathrm{log}\frac{P(D=d|S=x)}{P(D=d)}$$
(1)
In the case where system $`x`$ adopts a ubiquitous retrieval technique, the value of $`P(D=d|S=x)`$ becomes similar to that of $`P(D=d)`$, and thus the utility of $`x`$ becomes small. On the other hand, the utility of $`x`$ becomes greater as the number of novel  relevant documents provided by $`x`$ increases.

We then calculate the total utility of $`x`$, $`U(x)`$, by summing up $`U_d(x)`$’s of all the relevant documents for the query, as shown in Equation (2).

$$U(x)=\underset{d}{}U_d(x)$$
(2)
To sum up, our evaluation method favors systems with greater $`U(x)`$.

In Equation (1), $`P(D=d)`$ is the summation of $`P(D=d|S=y)`$’s for existing systems, averaged by the probability that the user utilizes system $`y`$, $`P(S=y)`$. Thus, given a set of existing system excluding $`x`$, $`E`$, we calculate $`P(D=d)`$ as in Equation (5).

$`\begin{array}{ccc}P(D=d)\hfill & =\hfill & {\displaystyle \underset{yE}{}}P(D=d|S=y)P(S=y)\hfill \\ \multicolumn{3}{c}{}\\ & \hfill & {\displaystyle \underset{yE}{}}P(D=d|S=y){\displaystyle \frac{1}{|E|}}\hfill \end{array}`$ (5)
Here, note that we assume uniformity with respect to $`P(S=y)`$.

Finally, the crucial content is the way to estimate $`P(D=d|S=x)`$, i.e., the probability that the user reads document $`d`$ by using system $`x`$. It can safely be assumed that the user always reads the top document, $`d_1`$, and thus $`P(D=d_1|S=x)`$ always takes 1. However, the probability that the user reads remaining documents becomes smaller according to their ranking.

Given $`N`$ documents sorted according to their relevance degree, in descending order, the user can choose a threshold for the ranking (i.e., the boundary until which he/she continues to read) out of $`N`$ choices. Consequently, documents ranked lower than the threshold will be discarded.

In other words, we can calculate $`P(D=d|S=x)`$ as the probability that the user chooses a threshold equal to or greater than the ranking of $`d`$, as in Equation (6).

$$\begin{array}{ccc}P(D=d|S=x)\hfill & =\hfill & \underset{i=r_{x,d}}{\overset{N}{}}\frac{1}{N}\hfill \\ \multicolumn{3}{c}{}\\ & =\hfill & \frac{Nr_{x,d}+1}{N}\hfill \end{array}$$
(6)
Here, $`r_{x,d}`$ is the ranking of document $`d`$ determined by system $`x`$.

## 3. A Case Study using the IREX Collection

Our concern in this section is to investigate the characteristic of our evaluation method. For this purpose, we targeted IR systems participated in the IREX workshop \[Sekine and Isahara, 1999\], and compared the result obtained based on our newly proposed evaluation method, with that based on the precision/recall. We also investigated reasons behind the difference between those two results, if any.

### 3.1. Overview of the IREX Collection

The IREX collection was produced through the IREX workshop \[Sekine and Isahara, 1999\], which consists of TREC-style IR and MUC-style named entity (NE) tasks for Japanese.111http://cs.nyu.edu/cs/projects/proteus/irex/
index-e.html Hereafter, the IREX collection/workshop refers solely to that related to the IR task.

The IREX collection consists of 30 queries, 211,853 articles collected from two years worth of “Mainichi Shimbun” newspaper articles \[Mainichi Shimbun, 1994 1995\],222Practically speaking, the IREX collection provides only article IDs, which corresponds to articles in Mainichi Shimbun newspaper CD-ROM’94-’95. Participants must get a copy of the CD-ROMs themselves. relevance assessment for each query, retrieval results of 22 participating systems, and technical details of each system.

Each query consists of the ID, description and narrative. While descriptions are usually phrases to briefly express the topic, narratives consist of several sentences and synonyms associated with the topic. Figure 1 shows an example query in the SGML form (translated into English by one of the organizers of the IREX workshop).

Relevance assessment was performed based on the pooling method \[Voorhees, 1998\]. That is, candidates for relevant documents were first pooled using the 22 participating systems. Thereafter, for each candidate document, human experts assigned one of three ranks of relevance, i.e., “relevant”, “partially relevant” and “irrelevant”. The average number of documents pooled for each query is 2,105, among which the number of relevant and partially relevant documents are 68 and 116, respectively.

Each retrieval result consists of the top 300 articles submitted in the same form as used in the TREC.333http://trec.nist.gov/pubs.html For each of the 22 results, the TREC evaluation software was used to investigate the performance (e.g., non-interpolated average precision). Figure 2 shows a fragment of the retrieval result obtained with one of the participating systems, which consists of the query ID, dummy field, article ID, ranking of the article, relevance degree computed by the system, and system ID.

It should be noted that using relevance assessment and retrieval results for each system, we can easily calculate $`P(D=d|S=x)`$ in Equation (6), which is the central issue in estimating our evaluation measure.

Technical details of participating systems were collected from questionnaires answered by each participant, where questions ranged from retrieval algorithms used to execution time. Although several questions are relatively vague, a number of questions are effective to characterize each system.

Table 1 shows representative questions in terms of retrieval accuracy. In this table, the number of answers are indicated in parentheses. However, answers classified as “no”, “unknown” and “etc.” are not shown. Roughly speaking, most systems adopted the word-based indexing and vector space model combined with TF$``$IDF term weighting.

On the other hand, note that in the IREX workshop, the correspondence between system IDs and participants is not available to the public. Additionally, several participants did not have oral presentations and papers in the proceedings. Consequently, for some systems it is difficult to obtain sufficient technical details.

For example, although most participants answered “TF$``$IDF” for the question about term weighting method, it is not possible to identify the exact formula used, out of a number of variants \[Salton and Buckley, 1988, Zobel and Moffat, 1998\], for several systems.

### 3.2. Experimentation

As explained in Section 3.1., the 22 IREX participating systems have already been ranked based on the conventional precision/recall, using the TREC evaluation software.

Thus, we re-evaluated the 22 systems based on our evaluation method, and compared results derived from different evaluation methods. To put it more precisely, we conducted 22 trials in each of which a different system was under evaluation and the rest were regarded as existing systems. That is, the former and latter correspond to $`x`$ and $`E`$ in Section 2., respectively.

Note that in this evaluation, we did not regard “partially relevant” documents as relevant ones, because interpretation of “partially relevant” is not fully clear to the authors.

Table 2 compares rankings obtained based on non-interpolated average precision and the utility factor we proposed in this paper. Table 3 compares rankings obtained with two evaluation methods on a query-by-query basis, where we show solely the difference of rankings for enhanced readability. Since in the IREX collection, every query ID consists of four digits stating with “10”, we simply show the remaining two digits in Table 3.

### 3.3. Discussion

Looking at Table 2, one may notice that rankings of systems “1106”, “1145b”, “1133a” and “1133b” were significantly improved within our evaluation method. Thus, we investigated properties that characterize each of those four systems, in a comparison with other systems.

First, we found that “1106” adopted a relatively simple implementation, while most systems used more elaborate ones. To put it more precisely, morphological analysis was performed, and nouns/verbs were extracted for a word-based indexing. For term weighting, a TF$``$IDF formula as in Equation (7) was used, while most systems used different methods, such as the logarithmic TF formulation as in Equation (8) and one proposed by Robertson and Walker \[Robertson and Walker, 1994\].

$$f_{t,d}\mathrm{log}\frac{N}{n_t}$$
(7)
$$(1+\mathrm{log}f_{t,d})\mathrm{log}\frac{N}{n_t}$$
(8)
Here, $`f_{t,d}`$ denotes the frequency that term $`t`$ appears in document $`d`$, and $`n_t`$ denotes the number of documents containing term $`t`$. $`N`$ is the total number of documents in the collection.

Second, “1145b” conducted a query expansion \[Qiu and Frei, 1993\], while a few systems used query expansion (e.g., one based on a thesaurus). In addition, a term weighing method based on mutual information between two terms was introduced. Possible rationales behind this method include that two terms frequently co-occur are effective to characterize the domain of documents, and are thus assigned with greater term weights.

Third, “1133a” and “1133b” also used domain knowledge for term weighting. However, unlike the case of “1145b”, they regarded pages of news articles as domain. In practice, a greater weight is assigned to terms whose distribution varies more strongly depending on the page, because they are expected to characterize the domain. On the other hand, terms commonly appear in more pages are assigned with a lesser weight.

To sum up, our novelty-based evaluation revealed the effectiveness of those properties above, specifically term weighting methods introduced in “1145b”, “1133a” and “1133b”, which were overshadowed or underestimated within the precision/recall-based evaluation.

We devote a little space to consider Table 3 for further investigation. We arbitrarily regarded improvements above seven as significant, and focused solely on systems with relatively many significant improvements, that is, “1103a” and “1132”. Although “1145b” is associated with the same number of significant improvements as “1132”, we previously discussed system “1145b” above.

We found that “1103a” is one of five systems that conducts a proper noun identification, and that five of six queries where “1103a” achieved significant improvements are directly or indirectly associated with proper nouns.

Samples of query descriptions directly and indirectly related to proper nouns include “1016: Nick Price (a golfer)” and “1011: arrest of suspects of robbery in the Kanto region”, respectively. Note that in the latter (indirect) case, Japanese prefectures within the “Kanto” region, which are not explicitly described in the query (e.g., “Tokyo” and “Kanagawa”), must be identified in news articles.

Finally, “1132” is the only system that used Latent Semantic Indexing (LSI), which is an extension of the vector space model, so as to retrieve relevant documents including no common terms in a given query. While as shown in Table 2, “1132” had the lowest ranking in terms of the average precision, our evaluation method indicated that in many cases (queries) an LSI-based method is expected to retrieve relevant documents that other types of methods fail to retrieve.

## 4. Conclusion

Evaluation methods based on precision and recall have long been used in information retrieval (IR) research, where systems that retrieve as many relevant documents as possible are usually highly valued.

However, given the fact that a number of retrieval systems resembling one another are available to the public (not only in laboratories), it is valuable to retrieve relevant documents that can never be retrieved by those existing systems. This notion is also true in various contexts that require a variety of IR systems, such as meta search systems and the pooling method in producing IR test collections.

In consideration of these factors, we proposed a new evaluation method for IR, which favors systems that retrieve more novel documents, i.e., relevant documents that many systems fail to retrieve. To realize this notion, we estimated the utility of a system in question by comparing the probability that the user reads relevant documents by using the system, and the probability that the user can read those documents even without using the system.

We also applied our evaluation method to the 22 systems that participated in the IREX workshop, and identified several effective techniques that have been underestimated in the conventional precision/recall-based evaluation method.

## Acknowledgments

The authors would like to thank organizers and participants of the IREX workshop for their support with the IREX collection.


text:
# Robust Insulator-Superconductor transition on solid inert gas and other substrates

## Abstract

We present observations of the insulator-superconductor transition in ultrathin films of Bi on amorphous quartz, quartz coated with Ge, and for the first time, solid xenon condensed on quartz. The relative permeability $`ϵ_r`$ ranges from 1.5 for Xe to 15 for Ge. Though we find screening effects as expected, the I-S transition is robust, and unmodified by the substrate. The resistance separatrix is found to be close to $`h/4e^2`$ and the crossover thickness close to $`25\mathrm{\AA }`$ for all substrates. $`IV`$ studies and Aslamazov-Larkin analyses indicate superconductivity is inhomogeneous. The transition can be understood in terms of a percolation model.

PACS Numbers : 73.50.-h, 74.40.+k, 74.80.Bj

The insulator-superconductor (I-S) transition has been extensively investigated over the last decade, in a variety of systems such as thin films  single Josephson junctions , arrays , one dimensional wires . Values of limiting resistance close to the quantum resistance for pairs in one dimensional wires , and two dimensional, nominally homogeneous ultra-thin films  are reported. Differing values of the limiting resistance at the transition have been observed  in different systems, and attributed to structure, i.e. homogeneous and granular films are expected to behave differently.

A phase-only picture, first proposed by Ramakrishnan , and further elaborated by Fisher  has been considered appropriate for such systems. A scaling theory of the I-S transition has been developed.  This theory predicts that the critical resistance will be universal, i.e. independent of all microscopic details, if the system is invariant under the interchange of the roles of charge and flux. Thus the I-S transition may be self-dual . The precise value of the critical resistance appears to depend on the nature of the interaction between charges, being equal to $`h/4e^2`$ only when the interaction is logarithmic in the separation, one of the conditions which must exist for self-duality. The interaction between charges can be logarithmic in their separation if the 2D films have a sufficiently high dielectric constant.  This condition may be met for semiconductor and semimetal films.

This has to be reconciled with experimental observations, which show that superconductivity in ultrathin films can be enhanced by the proximity of a metal or dielectric.  This is attributed to partial screening of the Coulomb interaction between conduction electrons in the films. Screening also affects the properties of insulating films, by changing the localization length.  Glover studied the effect of the substrate dielectric constant on transition temperature of thin films, over thirty years ago, inconclusively. Ge or Sb has been extensively used as underlayers in ultrathin film studies . A natural question arises at this point about the effect of the substrate/underlayer on the I-S transition.

For a film on a underlayer, the general expression for the interaction potential between charges in the film is

$$U(r)=\frac{e^2}{4\pi ϵ_0ϵ_r}\left(\frac{1}{r}\frac{1}{\sqrt{r^2+4d^2}}\right)$$
(1)
where $`d`$ is the screening distance (the distance between the midplane of the film and the midplane of the underlayer) and $`r`$ the separation between the charges in the film. If the separation of two charges is small compared with the distance between a charge and its image ($`2d`$), the underlayer makes little difference, and the interaction remains monopolar. At large distances, however the charge and its image behave as a dipole and interaction falls off more rapidly with distance. For the films studied here, $`2d`$ would be close to the film thickness. Although screening by charge carriers in the film is considered in the usual BCS treatment, the relative permeability for a highly disordered film is unknown.  It is expected to lie between the metallic and insulating limits of $`ϵ_r\mathrm{}`$ and $`ϵ_r10`$. This issue becomes important in the I-S transition region. $`U(q)`$, the term in the BCS interaction Hamiltonian becomes

$$U(q)=\frac{4\pi e^2}{q^2+k_s^2}$$
(2)
where $`k_s`$ is the inverse screening length. In principle a film with a transition temperature of 0 K defines $`R_c`$. As we show, screening influences the transition temperature of superconducting films, and the normal state resistance of all films. Therefore it should affect the I-S transition.

It has also been proposed that the I-S transition in the limit $`T=0`$ is a combined effect of pairing and localization in 2D systems.  Experiments have refuted this conjecture.c̃itekagawa How disorder, cluster size, and the Coulomb interaction influence the behavior of Cooper pairs is still not very clear. It has been conjectured that the Coulomb interaction may not influence the critical normal state sheet resistance.  However, this is refuted by observations of the M-I transition in silicon MOSFETS and other systems in zero magnetic field.  It has been shown that interactions may increase or decrease the conductance of a disordered 2D electron system - weak interactions increase the dc conductance in the localized regime while they decrease the conductance in the diffusive regime.  Strong interactions were always found to decrease the conductance. These considerations motivated our experiments.

In superconductors, Coulomb interactions suppress the fluctuations in the number of electrons, and increase the fluctuations of the phase of the superconducting order parameter. This affects the Josephson coupling energy $`E_J`$ in inhomogeneous systems, where the competition between the charging energy $`E_C`$ and $`E_J`$ drives the transition.c̃itekagawa Typical systems exhibiting such behavior are granular films, and junction arrays. $`E_J`$ may be estimated using the Ambegaokar-Baratoff equation, $`E_J=\pi \mathrm{}\mathrm{\Delta }/4e^2R_N`$, where $`\mathrm{\Delta }`$ is the BCS gap for Bi, $`R_N`$ is the normal state resistance of the film. $`E_C`$ can be estimated as $`E_C=q^2/4\pi ϵ_0\kappa d`$ where $`\kappa =ϵ_r(d/2s+1)`$, where $`ϵ_r`$ is the relative permeability of the substrate, $`d`$ is the size of the grains/clusters, and $`s`$ is the spacing. Since $`R_N`$, and $`\kappa `$ depend on the substrate/underlayer, one expects the I-S transition to occur at different $`R_N`$ on different substrates. Since the effects of disorder, cluster size, and interactions on Cooper pairs are unclear, exlpicit predictions of $`R_N`$ for the I-S transition on various substrates cannot be made.

In this work, we report studies of the I-S transition in quench condensed Bismuth films, as a function of disorder (or film thickness) on a variety of substrates - amorphous quartz, quartz coated with Ge, and solid xenon condensed on quartz. The relative permeability $`ϵ_r`$ ranges from 1.5 for Xe to 15 for Ge. Despite screening effects the I-S transition is robust. Studies on other substrates such as single crystal sapphire, and sapphire coated with Ge, Xe yield similar results. The experiments were done in a custom UHV cryostat equipped with reflection electron diffraction (RHEED), capable of a hydrocarbon free vacuum of $``$ 5$`\times `$10-10 Torr, described elsewhere . The Ge underlayers were deposited on the substrates in a separate UHV system at room temperature. The Xe underlayers were grown in-situ. Our experimental setup resembles that of Ref.\[1-2\], although we cannot attain such low temperatures. We quench condense our ultra-thin films in the temperature range 1.8 K to 15 K. The substrate temperature influences the disorder in the film, and thereby its properties. Here we report results on films that were quench condensed at 15 K, which facilitates comparison with published results .

RHEED studies show that the Bi is almost amorphous. It is difficult to unambiguously distinguish between amorphous and nanocrystalline at such low temperatures, on poorly conducting films due to charging effects. Based on the Scherer formula for the peak broadening, we estimate that films thicker than 10 $`\mathrm{\AA }`$ are composed of clusters that vary in size from 25 $`\mathrm{\AA }`$ to 100 $`\mathrm{\AA }`$ . Since the information obtained is in reciprocal space, it is difficult to comment on the real space surface morphology. Our RHEED observations are consistent with previous results.  Superconductivity in granular systems of rhombohedral Bi clusters has also been reported.

Figure 1 shows the evolution of the temperature dependence of the sheet resistance R(T) with thickness for (a) Bi films on Ge (10 $`\AA `$ thick), which has been deposited on amorphous quartz, (b) Bi films on quartz, and (c) Bi films on solid xenon condensed on amorphous quartz. A transition from insulating type behavior, to superconducting behavior as the thickness of the films is increased is clear. This type of zero field transition is considered a zero temperature quantum phase transition, controlled either by disorder, carrier concentration, or thickness. The normal state resistance at an arbitrarily high temperature $`\mathrm{R}_\mathrm{N}`$ has traditionally been used to parameterize the transition, although it may be weakly temperature dependent above the superconducting transition temperature, and becomes ill defined as the I-S transition is approached. The value of the normal state resistance of a film on the boundary between superconducting and insulating behavior has been referred to as the resistance separatrix, and has been denoted by $`\mathrm{R}_0`$ . We obtain $`\mathrm{R}_0`$ as an algebraic average of the sheet resistances of the last insulating and the first superconducting films, measured at a relatively high temperature (10 K). $`\mathrm{R}_0`$ is close to $`h/4e^2`$ for all three sets of data. This observation indicates that the value of $`\mathrm{R}_0`$ is substrate independent, and possibly experiment independent.

The transition temperature for a film of a given thickness is higher on substrates of higher relative permeability, and the normal state resistance is also higher. A 65 $`\mathrm{\AA }`$ film on Xe, quartz and Ge has $`T_c`$’s of 3.8, 4.2 and 4.42 K respectively. These results are consistent with published data.  This shows that the conductance of a disordered film depends on interplay between interaction and disorder.  We caution that since our experiments are limited to 1.8 K, we cannot rule out the possibility that a film which appears to be the last film on the insulating side, may, at lower temperature turn out to be superconducting. . A plot of conductance vs thickness at different temperatures gives the crossover thickness, which is shown in Fig. 2 for films on Xe. We find the crossover thickness to be between $`25\mathrm{to}28\mathrm{\AA }`$ for all substrates.

The insulating films follow a behavior that is consistent with the results reported.  We find that the conductivities of all our insulating Bi films, on all substrates to be:

$$\sigma (T)=\sigma _0exp[(A/T)^x]$$
(3)
where x changes with film thickness. For the thinnest films, x is close to 0.75, due to collective hopping . As the thickness is further increased, x reduces to 0.5, the Efros-Shklovskii form , which describes hopping modified by Coulomb interactions, and finally for the thickest insulating films x is close to 0.33, the Mott value .

The form of the R(T) for these films may lead us to the conclusion that these films are nominally homogeneous. Such a conclusion is incorrect as we show below. Although STM studies of surface morphology  report that even at 75% coverage the films are not conducting, hence no evidence for percolative behavior, we find that superconductivity in our films is indeed percolative in nature.  Aslamazov and Larkin  considered the possibility of fluctuations causing superconductivity. The total conductivity is given by $`\sigma `$ = $`\sigma _N`$ \+ $`\sigma _{2D}^{^{}}`$, where $`\sigma _N`$ is the normal state dc conductivity, and $`\sigma ^{^{}}`$ the paraconductivity. Its temperature dependance is similar to that of the magnetic susceptibility at T$`>`$Tc. They derived the result,

$$\frac{\sigma _{2D}^{^{}}}{\sigma _n}=\frac{e^2}{16\mathrm{}}\frac{R_{\mathrm{}}^N}{\tau }=\frac{\tau _o}{\tau }$$
(4)
where R$`{}_{\mathrm{}}{}^{}{}_{}{}^{N}`$ is the normal state sheet resistance, $`\tau _o`$ = 1.52 $`\times `$ 10-5 R$`{}_{\mathrm{}}{}^{}{}_{}{}^{N}`$ and $`\tau =\frac{TT_c}{T_c}`$ is the width factor. Tc is the mean field transition temperature. $`\tau `$/R$`{}_{\mathrm{}}{}^{}{}_{}{}^{N}`$ is a constant ($`g_{AL}`$=e2/16$`\mathrm{}`$) for all materials. We have evaluated $`\tau `$/R$`{}_{\mathrm{}}{}^{}{}_{}{}^{N}`$ ($`g_{exp}`$) for various films. A systematic dependence of $`g_{exp}`$ on the thickness $`d`$ is shown in Fig. 3. This parameter deviates from $`g_{AL}`$ for thinner films. It approaches the AL value ($`g_{AL}`$) as the thickness is increased. It is assumed that theory predicts $`g_{exp}`$ = $`g_{AL}`$ for all films, independent of microstructure. Both the normal state conductance and paraconductance depend on sample shape. Glover  has shown that as the microstructure deviates from a “uniform rectangular slab”, $`g_{exp}`$ exceeds the AL value. The thinner the film, higher the disorder, larger are the deviations from a slab geometry, and larger the deviation of $`g_{exp}`$ from the AL value. Hence, films close to the transition are inhomogeneous.

Further evidence is in the form of hysteretic I-V curves, shown in Fig. 4. These curves can be understood in terms of a resistively and capacitively shunted random Josephson junction array model. These I-V’s have been discussed in detail in a separate publication . The physical picture is of superconducting islands connected by thin regions of normal metal, which act as the weak links between the islands, thus forming a random JJ array. The distribution of grain/cluster, each with a finite number of electrons, requires a finite N BCS treatment. This results in a spread in $`T_c`$ , so that at some temperature all regions of the film are not superconducting. Normal regions exist, and act as weak links. Areal maps of quasiparticle conductance or superconducting order parameter amplitude can resolve such regions.

Feigel’man and Larkin  and Spivak et al.  have discussed the quantum superconductor-metal transition in a 2D proximity coupled array. Thickness larger than the coherence length was considered by the former, with the opposite extreme analyzed by the latter. They predict intervention of a normal phase. Limitations on the accessible temperatures prevent our study confirming this. From the $`H_{c2}(T)`$ curve, we have determined the coherence lengths ($`\xi _c`$) to be close to the film thickness. This regime merits further theoretical study. We have evaluated the localization lengths ($`\xi _l`$), and we find that at the I-S transition, $`\xi _l>>\xi _c`$, consistent with the results of Ref. .

The effect of strong disorder on the superconducting order parameter amplitude has been investigated within the Bogoliubov-de Gennes framework . The local pairing amplitude develops a broad distribution with significant weight near zero, as disorder increases. The density of states showed a finite spectral gap. Persistence of the gap was found to arise due to the breakup of the system into superconducting islands. Superfluid density and off-diagonal correlations showed a substantial reduction at high disorder. Incorporation of phase fluctuations lead to a non-superconducting state. Our data are consistent with such a picture.

Our results suggest that a percolation type model, proposed by Meir  to explain the M-I transition in 2D systems is most appropriate and merits investigation. Meir suggested the following physical picture for the M-I transition in 2D - potential fluctuations due to disorder define density puddles of size $`L_\varphi `$ or larger, within which the electron wave function totally dephases. Locally between these puddles, transport is via quantum tunneling. Support for this idea comes from fits of the conductance at a high temperature (10 K) for the insulating films which follows a power law, with an exponent of 1.33 characteristic of 2D percolation, as shown in Fig. 5. As an interesting observation, we also show that the superconducting films also fit this same power law, albeit with a different critical thickness. The critical thickness for the insulating films is the thickness for onset of electrical conduction. We have no explanation as to why the critical thickness for normal state conduction depends on whether or not the film superconducts at lower temperature, but present this as an interesting observation. Keeping in mind the model discussed above, it is logical to associate the puddles with superconducting regions and the quantum point contacts with the normal regions. The competition between the $`E_C`$ and $`E_J`$ probably drives the transition. However, it is intriguing that the transition is robust at $`h/4e^2`$ on all substrates, despite $`E_C`$ and $`E_J`$ being dependent on $`ϵ_r`$. Percolation issues have been studied earlier.

The conditions for 2D coulomb interactions may have been met in this and other experiments.  Keldysh  has shown that this requires the dielectric constant of the film to be much greater than that of the substrate. In this study, substrate dielectric constants range from 1.5 for solid Xe to 15 for Ge underlayers. Hall measurements were done for the first superconducting film near the transition, and this yielded a high carrier concentration of the order of $`10^{23}cm^3`$, which is consistent with that reported in the literature , suggesting a high dielectric constant for the films. We have not done Hall measurements for films on the insulating side, since interpretation of data in the hopping regime is complicated. Vortices exist in these films, since they are random Josephson junction arrays,  due to the discrete nature of the array, which requires that the current take convoluted paths, consistent with a transition governed by percolation. Duality in JJ arrays has been observed before.

In conclusion, we have observed robust I-S transition in ultrathin Bi on several substrates. We have presented, for the first time, the I-S transition on solid inert gas underlayers. Although the substrate influences the $`T_c`$ as well as $`R_N`$ of the films, the transition itself is independent of the interaction potential between charges. An AL analysis indicates inhomogenous films, contrary to the R(T). I-V’s indicate a percolation type transition. A model similar to that proposed by Meir for the M-I transition in 2D merits further investigation. Further studies at lower temperatures are needed.

This work was supported by DST, Government of India. We thank Drs. Allen Goldman, C. J. Adkins for discussions. KDG thanks CSIR, New Delhi for the financial support through a Junior Research Fellowship.

FIGURE CAPTIONS

Figure 1 Evolution of R vs T of Bi on (a) 10 $`\mathrm{\AA }`$ Ge on amorphous quartz (b) bare quartz and (c) solid Xe condensed on amorphous quartz. The substrate temperature during deposition in all cases was maintained at 15 K.

Figure 2 Plots of resistance vs. thickness at different temperatures showing the crossover thickness of about $`26\mathrm{\AA }`$ for Bi films on Ge. The crossover occurs within $`\pm 3\mathrm{\AA }`$ of this value for all substrates.

Figure 3 The parameter $`g_{exp}`$ which characterizes the amplitude of the fluctuations as a function of film thickness. It asymptotically approaches the Aslamazov-Larkin value.

Figure 4 I-V of the superconducting films on quartz. The hysteresis loops show the dissipation due to resistances shunting the junctions of the random Josephson junction array. The first film is $`40\mathrm{\AA }`$, and the thickness increment is $`10\mathrm{\AA }`$.

Figure 5 Fits of the conductance of the insulating films (taken at 10 K) and the normal state conductance of the superconducting films (also taken at 10 K) to 2D percolation function, for Bi films on Ge. Only the critical thicknes in each regime is different, as discussed in the text.