text:
# 1 Preamble

## 1 Preamble

It is a real pleasure to be back in Singapore and have the opportunity to address this meeting. About a decade ago, I gave a similar talk here, as a last-minute substitute for Leon Lederman . Since that time I have for the most part exited theoretical physics and entered experimental physics. In particular I have been promoting new detector techniques for exploring the world of strong interactions. This led to my colleague Cyrus Taylor, a string theorist by training, and I leading a small test/experiment in the Fermilab Tevatron Collider. The results of this experiment were modest . But the experience was in many ways extremely rewarding and enriching, and in other ways frustrating. However, as the new century begins, I have emerged from the experimental trenches and am entering the world of retirement. I now do have the opportunity to again take a look at the big picture, after a decade of time off. And the invitation to give this talk provided a most opportune way to organize my own thoughts and to try to express them.

## 2 The Big Picture

Twentieth century physics featured several great syntheses, The first was the extraordinary synthesis by Planck of thermodynamics with Maxwell electrodynamics, giving birth to quantum theory. It was quickly followed by the synthesis of classical mechanics with electrodynamics by Einstein, giving birth to special relativity. And in the 1920s, with the decisive synthesis of Newtonian mechanics with the “old” quantum mechanics by Heisenberg and Schrodinger, there also emerged quantum electrodynamics (or QED), the synthesis of Maxwell electrodynamics with quantum mechanics.

All this happened in less than three decades. Much of the remaining history for the century belongs to the experiments, which built upon these foundations and in particular drove the development of particle physics. Many particles were discovered. The strong and weak forces themselves needed to be discovered before there was any opportunity to understand their role. Eventually things became understood well enough that by the 1970s there was the possibility of a synthesis of weak and electromagnetic forces into a common structure: the SU(2) $`\times `$ U(1) electroweak component of the Standard Model. And the strong force is now successfully described by the generalization of QED called quantum chromodynamics (QCD), based on the exact SU(3) color symmetry of quarks and their force carriers, the gluons.

All these forces (including gravitation) are described at short distances by the same class of fundamental theories, the so-called gauge theories. The force law at short distances is in all cases essentially inverse-square. Standard Model forces are proportional to a variety of conserved charges, while gravity couples to energy-momentum. So as we enter the twenty-first century, there is a strong anticipation that these four forces have a common origin and that further synthesis is on the way.

There are quite specific clues to build upon. One clue is the pattern of fundamental building blocks of matter: the quarks and leptons. They are exhibited explicitly as building blocks in Fig. 1. This is a construction which has technical meaning only when viewed beyond the Standard Model, when the SU(3) $`\times `$ SU(2) $`\times `$ U(1) Standard Model symmetry group is embedded in the much larger symmetry group of ten-dimensional rotations, SO(10). A clumsy, reducible representation of the Standard Model group becomes an elegant, single sixteen-dimensional fundamental spinor representation of SO(10). In that context the 5-dimensional cube which is depicted in Fig. 1 has a definite mathematical meaning.

A second clue, pointing in the same direction, has to do with the fact that the coupling strengths or “charges” associated with the three kinds of Standard Model “gauge” forces vary slowly with distance scale. It happens that they converge (or very nearly converge) to a common value at a very high mass scale, 1015 GeV, give or take a factor 10, as shown in Fig. 2. It is at this scale that synthesis of these three forces can be expected to occur. This anticipated synthesis is known as Grand Unification.

Even one or two generations ago, it was the dream of every theorist to come to an understanding of the value of the pure number 1/137 of QED, a number which characterizes the intrinsic strength of the electromagnetic force at the quantum level. This is no longer the case. The 137 evolves to 128 through “vacuum polarization” effects; cf. Fig. 2. At this point, at a mass scale of about 100 GeV, the electroweak synthesis takes place. Thereafter it is the coupling strength of a mixture of photon and electroweak boson which, together with weak-boson and gluon coupling strengths, evolve from values of about 1/60, 1/30, and 1/10 respectively to the common value of about 1/40 at the grand-unification scale. So the 137 has been divided by a factor of about 3 1/2, and the question is now “Why 1/40?”

There is another synthesis taking place at present, that of cosmology and particle physics. Twentieth century astronomy and astrophysics has its own rich history. But with the emergence of Big Bang cosmology in the last forty years, there has also emerged an increasingly strong interdependence of cosmology and particle physics. The high temperatures present during the early epochs of the Big Bang demand a good understanding of particle physics in any theoretical description, while the empirical evidence which constrains theories of the early history of the universe also constrains the theories of particle physics.

So there is now an increasingly strong interconnection of the largest distances with the smallest, and of the longest time scales with the shortest. This is illustrated in Fig. 3, which can truly be called the Big Picture. In that figure every effort has been made to omit the superfluous. What is left is what I personally believe to be the key landmarks necessary to comprehend when moving on to the next syntheses.

Nothing of the Big Picture in Fig. 3 existed a century ago. That it exists now is a tribute to the extraordinary scientific progress made in that period. Progress in physical science has three components: technological, experimental, and theoretical. These are interconnected, but I believe that the order of importance is as stated. Without technological advances, experimental technique stagnates. And without the validations and unanticipated discoveries that comes from advances in experiment, the finest creations of theoretical physics languish as exercises in natural philosophy or in higher mathematics, and are of little worth as physical theory.

## 3 Technology

In particle physics the name of the game is energy. Throughout the twentieth century there has been exponential growth in the attainable center-of-mass energy available for particle collisions, beginning with a few electron-volts in early vacuum tubes and ending with the trillions of electron volts in the Fermilab Tevatron Collider . On average this amounts to a doubling time of about 2-1/2 years.

This pace is unlikely to be equaled in the coming century. The slowing of the pace has already been apparent for the last decade or two. The new machines are big and expensive, and take a long time to build and to fully exploit. But this does not mean an end to the field. There is plenty of room for further expansion of the possibilities, throughout the coming century.

Modern colliding beam machines are circular storage rings, within which some combination of counter-rotating beams of electrons, positrons, protons, and/or antiprotons collide with each other. The biggest electron-positron collider is the LEP ring at the CERN laboratory  in Geneva, Switzerland, 27 kilometers in circumference, and now running at a cms energy of about 200 GeV. This represents the limit of possibility for this kind of ring. The electrons and positrons emit so much synchrotron radiation, and the rate grows so rapidly with increasing energy, that it is impractical to contemplate similar machines at much higher energies. This is not the case for the heaver protons, where ten times that energy is attained in the Fermilab proton-antiproton collider already. And the Large Hadron Collider (LHC), under construction at CERN in the LEP tunnel itself , will increase this value sevenfold. Sooner or later, a similar synchrotron-radiation limit will be reached, however, especially given that superconducting magnets are needed to bend the protons, and that superconducting systems and intense beams of x-rays tend not to peacefully coexist. A rough estimate of where the ultimate limit for circular proton rings occurs puts the number at a few hundred TeV per beam. This is between one and two orders of magnitude larger than the LHC, and implies that at least one more very big collider is technically feasible , and perhaps affordable on a world scale sometime in the future.

Such a machine would be very big, probably over a thousand kilometers in circumference. It would require a lot of (underground) real estate, and for a long time it has seemed to me that this region of the planet, with Down Under so close at hand, is a natural focus when dreaming about this possibility. Given the premise that the twenty-first century will witness not only an economic but also scientific and cultural blossoming within Southeast Asia, such a project in this region might be an appropriate and locally beneficial way of entering this fundamental field of science. The time scale for even considering such an initiative is not at all immediate, but could be as short as two or three decades. And as we shall see, there are scientific considerations as well to deal with. What is done in the far future depends in an important way on what will be learned from the experiments performed in the nearer future.

When thinking about higher energy electron-positron collisions, the technique of choice is to accelerate the beams within straight-line, linear accelerators, as done in my home institution SLAC at Stanford, where collisions with 50 GeV electrons and positrons per beam have been attained . Another factor of 10 to 20 can be contemplated using relatively straightforward extensions of existing techniques. The main issue is to do this in an energy-efficient, economical way. A great deal of study for the “Next Linear Collider” (NLC) is being carried out, with a large amount of international collaboration. And some kind of NLC is a leading candidate, perhaps the leading candidate, for the next large facility beyond the LHC needed to further push back the high energy frontier .

However there is a relatively new, competitive idea being studied as well, namely to collide muons and antimuons (the unstable, heavier second-generation copies of the electrons and positrons) with each other within circular storage rings (cf. Fig. 4). At first sight this idea looks crazy, but after careful scrutiny it becomes less crazy, albeit difficult and adventurous . One physics advantage of muons over electrons and positrons is that resonant production of a Higgs boson is much larger and possibly observable. Another advantage is that muons do not emit so much synchrotron radiation. However many of them do decay en route to the collision point, and their decay products create a serious nuisance. An extremely intense source of primary muons is required.

But a general advantage of a muon collider facility is that there are many side-benefits. One can easily see that each component of the facility can—and should—support other physics programs. A very intense source of protons of tens of GeV is required, and this by itself is a good device for studying properties of the hadrons, including e.g. the rare decays of kaons. And the muons are accelerated in a recirculating linear accelerator which is a more powerful and sophisticated version of a facility, CEBAF, now being used to study hadron structure via electron scattering . Perhaps the most promising secondary application is utilization of the intense beams of neutrinos created by the decaying muons. Just in the last few years, with the emergence of the evidence for neutrino mass and mixing from non-accelerator experiments, there has been an escalating interest within the particle accelerator community in creating a “neutrino factory” using muon-collider technology. Such a program would provide useful neutrino physics as well as providing proof-of-principle evidence that the idea of creating intense, bright muon beams really can be made to work .

So while consideration of an NLC, with center-of mass energy of up to 1–2 TeV, will be high on the options list of future facilities beyond the LHC, the muon colliders, which might reach to a few TeV in the center of mass were all to go well, will also be under consideration. It does have to be said that the muon-collider technology is new and untested, so that it is reasonable to expect that if that route is followed it will be considerably further into the century before one could contemplate high energy muon collisions being attained. In either case, the reason that such lepton colliders are competitive with the much higher energy $`pp`$ colliders, such as the LHC, is the efficient conversion of all the energy in each event into interesting physics. The advantage of higher energy in $`pp`$ colliders is mitigated by the complexity of the collisions and the relative rarity of the collisions which produce the highest-priority “new physics”.

In addition to the energy frontier, there will be many other frontiers remaining within particle physics. There are good reasons for producing interesting particles with the highest intensities possible. Accelerator facilities which specialize in production of a single kind of particle are already in abundance: we already see $`B`$ factories, $`Z`$ factories, $`K`$ factories, and $`\varphi `$ factories. And as mentioned above, we probably will see neutrino factories in the future. Once the Higgs sector, the prime target of experimental physics nowadays, is found, there will be an irresistible urge to create a Higgs factory as well, probably utilizing some kind of lepton-lepton collider. It is likely that the “factory” programs will have great longevity.

Another frontier is that of non-accelerator facilities, utilizing for example nature’s own particle beams, be they cosmic rays or neutrinos from the sun. When the ideas of grand unification emerged (Fig. 2), there also emerged the prediction of proton decay. This was a real turning point for the field of non-accelerator particle physics, because thereafter the large-scale investment in huge detectors became commonplace, and the level of sophistication relative to previous, traditional cosmic ray research escalated considerably. Even today the investment in non-accelerator particle physics is a relatively small fraction of the total, so that important measurements may be expected in the future to be approached on a scale larger than done at present. The recent advances in neutrino physics, especially in Japan with SuperKamiokande , bear witness to the value of making a large, sound investment in a good detector when the potential benefits warrant it.

But the non-accelerator experiments will forever be very difficult and fraught with more uncertainty that the more highly controllable accelerator experiments. So there will be strong motivation to push the high energy frontier well beyond what we have talked about so far. It is here that very difficult barriers appear.

If one wants to attain center-of-mass energies well beyond 1000 TeV, there seems to be very little choice but to do it with linear acceleration. If we ask to do this within a reasonable distance, say 100 km, this implies an average acceleration gradient of at least 10 GeV per meter, or 1 eV per Angstrom. High powered lasers can in principle provide forces of such a magnitude. But such forces, if present in matter, are strong enough to pull electrons out of metals, or in general destroy the orderly chemical structure of just about any material. There is a general rule, Lawson’s theorem, that states that free electromagnetic waves do not make a good accelerator, essentially because the electric vector is at right angles to the Poynting vector.

So a good accelerator will almost certainly have a material structure close to the beam. or perhaps a plasma within the beam. And this material structure might well be damaged or destroyed by the laser pulse or other source of the accelerating field every time the device is pulsed. I envisage this as the insertion of an accelerating structure into the beam, instead of the present practice of insertion of a beam into the accelerating structure.

The notion of an accelerator which is destroyed by the beam during every pulse is not necessarily a hopeless one, especially if the transverse dimensions of the machine are kept very small, not much larger than the beam itself. The small transverse dimensions are probably a necessity anyway on grounds of the energy budget. One cannot afford to fill a large cavity with a lot of energy, as presently done, especially when it does not directly contribute to the acceleration of the beam. So it could be that the accelerating structure is in the form of a sequence of structured tapes, or edges of structured rotating disks, which move through the beam region. The transverse scale might be anything from microns to millimeters.

And there is a new technology, nanotechnology, which although not viable now, might in the future evolve to the point that accurate, practical, and inexpensive (i.e. disposable!) acceleration microstructures might eventually be fabricated. Nevertheless, the trouble list associated with such a line of thinking is impressively formidable. A huge obstacle is that of maintaining the brightness of the beam, i.e. keeping the beam size small. The typical unit of periodicity (cell length) of such a “miniaturized” linear accelerator might be envisaged to be in the range of millimeters at most. So in a 100 km machine, there are at least 108 such cells. And the phase-space volume of the beam (emittance) cannot increase on average by much more than 1 part in 108 per cell or it will grow unacceptably large by the end of the machine. So the reproducibility of these extremely violent acceleration mechanisms must be maintained to a very high level of accuracy. I am sure this is only one of a number of similar problems.

There is in addition another great difficulty, which is motivation. It is a folk theorem in the trade that a project does best if there is strong physics motivation for it. The sense of urgency creates focus, drives the project forward, and encourages everyone to contribute that extra level of creative effort to make it successful. An R&D program on very high gradient linear accelerators will naturally be restricted to short prototypes for a long time, and therefore not be highly physics driven. And the primary physics demand is not only for high energy but also for an extremely high collision rate. Typically, for each factor of ten of energy increase, one needs to increase the luminosity, or collision rate, by a factor of order one hundred. But there are secondary physics topics which can be addressed. For example, were 10 GeV electron or proton beams to be produced, economically, on a tabletop in the basement of some university physics department, some physics uses for them might well be found. This is especially true for proton beams, where many physics applications do not require the relatively large intensities that electron-beam physics requires.

At present, there is actually considerable activity in the field of high gradient linear acceleration, not only theoretical but increasingly experimental . It seems to me that there should be a high level of attention paid to this problem, even though it does not look to be immediately practical, and even though super-high gradients do not seem to be needed for at least a couple of generations of machines, in order to push the energy frontier forward. Sometime in the future the field will have to face up to this ultimate problem, and doing the homework now would seem to be a prudent strategy.

## 4 Experiments

In the realm of accelerator-based particle physics, the technology of particle detectors is highly advanced. In the future, as prognosticated in the previous section, refinements and extensions of the existing detector technologies can be expected to occur, especially with regard to rate capability, pattern recognition, resolution, and management of increasing data volume. But I at least do not perceive any need for a fundamental change of approach. The most difficult frontier is the providing of the higher energy collisions themselves, not in detection and analyzing the collision products.

Perhaps the most interesting experimental frontier is sociological. The size and complexity of experimental collaborations continues to grow. At present we see, at the LHC, collaboration sizes approaching two thousand, with participation of hundreds of collaborating institutions around the globe. And the time scale for construction and full exploitation of a single detector for a single experimental program rivals the full career length of the participating physicists. When I was here in Singapore the last time, I devoted quite a lot of time to this issue . The social problems are still there. But there remains a large, enthusiastic young generation within the big detector collaborations, and I believe that the system, although very different from what it was two or three decades ago, still allows a highly creative and stimulating scientific environment.

Another singular feature of modern experimental particle physics is its professionalism. In the past two decades, the standards applied to all aspects of the experiments, especially with regard to the management and statistical analysis of the data, have soared. This is I think largely due to the size of the collaborations: they are large enough to contain experts in all the relevant subspecialties. I think these standards are well above what exist in other fields of science, and that participants in big experiments, even if they leave the field of particle physics at some point, are superbly prepared to apply that expertise in other scientific or technological applications.

Indeed, it must not be forgotten that experimental particle physics not only depends upon technological progress, but that it contributes to it. For example, it was the need for efficient communication and for transmission of very large volumes of information amongst the highly dispersed international community of collaboration members that gave birth to the World Wide Web at CERN . It is a perfect example of how research in basic science is not only an intellectual endeavor worthy of support by society for its own sake, but also an engine that drives economic progress in ways that create fundamental changes which are not programmable in advance.

It therefore seems to me that it is very important for Singapore and this region to not only consider the enhancement of its participation in basic science in the areas of theory, as represented in this meeting, but also to consider participation in experimental and accelerator physics. It is especially easy nowadays to enter the field via the large international experimental collaborations and thereby gain immediate access to the mainstream of the field. The benefits grow in proportion to the investment, and I would urge the creation of conditions that might allow participation of especially the younger generation in particle-physics experimentation.

## 5 Beyond the New Standard Model

We now turn to the theoretical situation. I believe, as do many others, that the most crucial problem we face has to do with understanding the mechanism by which the quarks, leptons, and electroweak force-carriers (the gauge bosons $`W`$ and $`Z`$) get their mass. This problem is characterized by a mass scale at the edge of measurability at present, of order 100 to 1000 GeV. It was well defined already twenty years ago, and the efforts of many experiments, which have included the discovery and measurement of the properties of the $`W`$, $`Z`$, and the sixth, remarkably massive top quark, have simply sharpened the basic issues.

But the situation regarding the Standard Model has not remained static, thanks to the newest generation of neutrino experiments. They provide strong evidence that neutrinos also have mass and undergo “mixing”, similar to the way the three generations of quarks are mixed by the electroweak interactions. This is an extremely significant discovery, although not really a revolutionary one. It is in fact a very welcome one, one which replace the Old Standard Model by a New Standard Model. The 20 or so free parameters of the Old Standard Model are now increased by at least 7 (3 neutrino masses and 4 mixing angles) and perhaps 12 new parameters (masses for the 3 $`N`$’s discussed below plus two other mixing angles), depending upon what one chooses to include in the count. This by itself may not seem like progress. But the reason that neutrino mass is not all that unwelcome has to do with the Grand Unification perspective we mentioned earlier. In the Old Standard Model, there are 15 spin-1/2 building blocks per generation: the sixteenth degree of freedom $`N`$ in Fig. 1 is not included. The natural GUT symmetry group to consider is SU(5), for which the 15 building blocks fill two multiplets: a $`\mathrm{𝟓}`$ and a $`\overline{\mathrm{𝟏𝟎}}`$. Neutrino mass, in particular the very small neutrino mass observed, occurs in the SO(10) extension when the extra degrees of freedom $`𝑵_𝒊`$ (one for each generation of fermions) possess very large masses of order of the 1015 GeV GUT scale. The ordinary neutrinos mix with the $`𝑵`$’s, and quantum mechanical level repulsion drives their masses to very small values in a natural way. And with the $`𝑵`$’s present, the pattern of the building blocks (Fig. 1) becomes simpler, as mentioned before, being described by a single 16-dimensional irreducible spinor representation of SO(10). So the New Standard Model is to the Old Standard Model as SO(10) is to SU(5): a little bigger, and also a little prettier. And the transition from “Old” to “New” should be regarded as a very strong clue in dealing with the question of how to synthesize the strong and electroweak forces at the GUT scale.

Nevertheless, the problem of how the masses of quarks, leptons, $`𝑾`$, $`𝒁`$, and neutrinos, are generated still must be faced. There are many ideas on how the problem of mass is to be solved. The most economical way requires only one extra particle to exist, the famous Higgs boson. In many ways, this simplest of scenarios, called the “desert” scenario, works the best. Consistency is maintained all the way to the GUT scale of 1015 GeV or so if and only if the mass of the Higgs boson lies in a narrow window: 160 $`\mathbf{\pm }`$ 20 GeV. Nevertheless, theorists remain very disquieted by the “desert” scenario because there appears a quadratic divergence in the Higgs mass which must be subtracted away, and it seems very unnatural to set the physical Higgs mass to a small value at the electroweak scale of 100 GeV or so, rather than at the GUT scale of 1015 GeV.

However, this quadratic divergence problem bears great similarity to the problem of the small value of the cosmological constant. It too suffers from power-law divergences. And the cosmological constant problem cannot be solved with a straightforward appeal to supersymmetry, as is commonplace for the Higgs problem. So a serious “solution” to this “hierarchy” problem is to ignore it, acknowledging that its resolution will require a much deeper level of understanding, but recognizing that in the meantime renormalization (subtracting out the infinity) suffices to remove the problem from all known phenomenology.

But angst over the “hierarchy problem” has in the past 25 years spawned alternative approaches, the most important of which are phenomenological supersymmetry and “technicolor”. Both are characterized by the introduction of many new particles as well as new forces . If either is correct, there will be an extraordinarily rich experimental program for the future.

The evidence from precision measurements of electroweak processes, especially at LEP, seems to favor a Higgs sector of relatively low mass, below 250 GeV . This in turn is more in line with the supersymmetry option (“Minimal Supersymmetric Standard Model”, or MSSM), or the “desert” scenario than with technicolor, although nothing is strictly ruled out. The MSSM is expecially rich in new particles and new parameters. Each known particle has its superpartner, differing in spin by 1/2, most of which having masses below a few TeV. There are about 100 extra parameters to be determined through experiment. The Higgs sector is bigger, and it is expected that at least one of the Higgs’ lies below 130 GeV.

From the experimental point of view, these two alternatives, MSSM and “desert”, stand in stark contrast to one another. If MSSM is correct, then—once the energy scale has increased sufficiently to produce this cornucopia of new particles and phenomena—it will be full employment for experimentalists (as it already has been for the MSSM theorists).

On the other hand, suppose none of that is found and only the single Higgs boson is seen, with a mass of 160 GeV as predicted by the “desert” scenario (cf. Fig. 5). It will be more difficult to motivate new, expensive facilities at higher energies if no clear landmarks for new phenomena exist.

But the MSSM and “desert” scenarios are extreme cases. Yet another way of dealing with the hierarchy problem is simply not to have a hierarchy at all, but to have many new mass scales for new physics between the electroweak and GUT scales. In Fig. 6 is plotted versus mass $`𝒎`$ the number of (2-component) spin 1/2 fermions possessing mass no larger then $`𝒎`$. At the GUT or Planck scale many theories, e.g. superstrings, end up with many hundreds of fermions. For example, three E(8) generations adds up to 744 fermions. At present energies we have $`\mathbf{}`$ 48. In both the “desert” scenario and the MSSM nothing much is supposed to happen across all those orders of magnitude. But perhaps there are “oases” of new physics all across the desert, which gradually release new degrees of freedom. The hierarchy problem becomes irrelevant. But, just like the “desert” scenario for experimentalists, the “oasis” scenario is a plague for theorists, because the vision of the GUT scale becomes obscured by everything which is in between, most of which is beyond experimental access.

The bottom line is simple. The next generation of experiments is sure to be a singularly important turning point for the field. The importance of the Higgs and new particle searches cannot be overrated.

## 6 Boundaries of Knowledge and Theories of
Everything

For the last two decades, the subfield of theoretical particle physics with the most vitality, and with the most powerful intellectual force applied to it, is undeniably that of string theory. It is a most ambitious subfield, with the often claimed goal to be no less than a “theory of everything”, one which addresses the “unsolved” problem of synthesizing quantum mechanics with general relativity .

While I have no problem with what people do, I do have a problem with the rhetoric. In my opinion, a “theory of everything” is not a subfield of physical science, where a theory requires validation by experiment, but rather is a subfield of “natural philosophy”, which includes such fields as mathematics, philosophy, and religion, and which allows speculations and investigations unfettered by the constraints of experimentation and of the scientific method.

I also question the assertion that we presently have no quantum field theory of gravitation. It is true that there is no closed, internally consistent theory of quantum gravity valid at all distance scales, But such theories are hard to come by, and in any case, are not very relevant in practice. But as an open theory, quantum gravity is arguably our best quantum field theory, not the worst. Feynman rules for interaction of spin-two gravitons have been written down, and the tree-diagrams (no closed loops) provide an accurate description of physical phenomena at all distance scales between cosmological scales, down to near the Planck scale of 10-33 cm. The divergent loop diagrams can be renormalized at the expense of an in-principle infinite number of counterterms appended to the Einstein-Hilbert action. However their effects are demonstrably small until one probes phenomena at the Planck scale of distances and energies .

One way of characterizing the success of a theory is in terms of bandwidth, defined as the number of powers of ten over which the theory is credible to a majority of theorists (not necessarily the same as the domain over which the theory has been experimentally tested). From this viewpoint, quantum gravity, when treated—as described above—as an effective field theory, has the largest bandwidth; it is credible over 60 orders of magnitude, from the cosmological to the Planck scale of distances. The runner-up is QCD, which loses credibility at the GUT scale. Above that scale QCD arguably gets synthesized into a new improved theory, in a way perhaps similar to the way QED gets synthesized at the electroweak scale. Indeed of the three theories, QED as formulated by Dirac and Heisenberg and renormalized by Feynman, Schwinger, and Tomonaga, has the worst bandwidth because it is already modified in an essential way at the electroweak scale.

In the old days QED was considered by Landau and others as an inconsistent (closed) theory, because the coupling constant $`𝜶`$ grows at short distances and eventually, at an incredibly short distance, blows up. This would in principle limit the bandwidth of QED to a mere 100 orders of magnitude or so. What a disaster! But in fact other physics intervenes. It is interesting that this inconsistency does not happen in QCD. The strong coupling constant only blows up in the infrared, where—with the help of experimental evidence—it is concluded that the theory remains consistent. Nevertheless, while QCD does enjoy the (unique) status of an experimentally relevant and logically consistent quantum field theory with infinite bandwidth, in practice it probably does not matter that this is the case, because almost certainly new physics will intervene at the GUT scale, if not sooner.

While quantum gravity may have splendid bandwidth, it still remains the case that at the Planck scale the effective field theory formalism totally falls apart. It is here of course that one finds the arena appropriate to the Theories of Everything. It is to be sure a valid and important arena. And it will be wonderful indeed if a successful theory can be put together at that scale. The trouble with doing so is that there is precious little guidance from experiment. What has always been typical for progress in physical science is a painful, slow progression, one step at a time, with guidance from experiment at most of the steps. Success with a theory of everything would be something very different and extraordinary—that of a theory found almost completely using arguments of symmetry and/or esthetics.

But no matter what, there are already many beneficial consequences of the theory-of-everything program. In its present state I see it as a theory of theories–the study of deep connections between different beautiful theories, some of which might conceivably be relevant to real physical phenomena. These connections will I am sure be covered in other talks. Certainly the theoretical phase-space of ideas has been greatly enlarged, for example in the considerations of supersymmetries, of extra space-time dimensions, of black hole phenomena, and even of more efficient ways for calculating Feynman diagrams. I would be very surprised if the future, improved theory does not contain some of the ideas spawned by the superstring revolution, even if superstrings have nothing to do with anything.

## 7 Macroscopic Quantum Gravity

Given the argument that quantum gravity is a good theory because it has large bandwidth, I now worry about whether I believe it. The issue is only whether the quantum-gravity phenomena not covered by perturbative Feynman-diagram calculations (which I consider safe territory) rest on solid foundations. This boils down to the question of black hole horizons and Hawking radiation, a subject which involves a nontrivial application of quantum gravity at distance scales large compared to the Planck scale. (The physics near a true gravitational singularity will also be nontrivial, but will be characterized presumably by physics at the Planck scale.) The potential problem that concerns me is that the black hole horizon (Schwarzschild radius) is a region characterized by very large complexity and very large bandwidth.

By this I mean the following. In Schwarzschild coordinates, place stationary observers a very small distance $`𝒉`$ above the horizon. Their job is to survey the local environment, as well as communicate with neighbors. As the horizon is approached, the spacing of such observers must decrease as $`\sqrt{𝒉}`$; otherwise they will not be able to send light signals to neighboring stations; the photons will fall into the black hole before getting there. So the number of such stations must be proportional to the black hole area, and depend inversely upon the height above the horizon. This result no doubt has something to do with black hole entropy, and is what I mean by increasing complexity as the horizon is approached.

The increase of bandwidth is related to the fact that, because of gravitational redshift, the surveyor’s clock rate decreases toward zero as the horizon is approached, implying infrared sensitivity. The surveyor will see a divergent ratio of frequency of the light from his own Cesium clock, relative to light received from a distant Cesium-clock frequency standard. In addition the surveyor gets hot; he feels himself immersed in local Hawking radiation, with Hawking temperature which again diverges as the horizon is approached.

If our existing theoretical formalism has finite bandwidth, then the divergence in these quantities, which implies infinite bandwidth, may signal a sensitivity to new-physics phenomena at the horizon. Without an understanding of what the new physics is, there is no necessity that something discontinuous happens to, say, the nature of the vacuum as the horizon is crossed, but only that there is a reasonable possibility that this might occur. I realize that this viewpoint cuts strongly against conventional wisdom, because freely falling observers are not supposed to “see” the infinite bandwidth phenomena that the Schwarzschild surveyors see. And it is also a necessity that the horizon which is considered here is not the event horizon used by Penrose and Hawking in their global analyses, but rather an apparent, or “redshift” horizon. (See Fig. 7 for a description of the distinction I have in mind.)

While I am not yet sure of my ground and recognize that most experts do not share my doubts, I still find the Schwarzschild horizon a potential frontier, something akin to the frontiers posed by the Planck and cosmological distance scales. But maybe that is just my own shortcoming.

## 8 The Fate of QCD

As we have mentioned already, quantum chromodynamics (QCD), the theory of the strong force, is arguably the most comprehensive of the quantum field theories in use today, and in principle is a consistent closed theory with infinite bandwidth. In practice, there are two major branches of QCD, short-distance and long-distance. The former is dominated by a Feynman-diagram approach. It is basically perturbative in nature, although in practice large sets of diagrams need to be summed in order to attain the needed accuracy. And the short-distance limit has a host of applications and is of immediate relevance for all new-physics searches at the high energy frontier. For all these reasons there has been and will continue to be a large investment of effort in this area .

The large-distance limit, “soft QCD”, has to do with hadron structure and vacuum structure. The distance scale ranges from above 10-13 cm to a little below 10-14 cm, essentially the size scale associated with ordinary hadrons. Here perturbation theory cannot be reliably used, and therefore the theory is much more challenging. Many open questions remain, the most prominent being a full description of the phenomenon of quark confinement.

The vacuum structure of QCD is especially rich. At moderate distances there is the challenge of understanding the role of instantons. At large distances, there exists a “chiral condensate”, emergent from the spontaneous breaking of the approximate strong-interaction chiral symmetry. And the QCD phase diagram needs explication; at present there is a lot of progress in the theory . And the exploration of heavy ion collisions at CERN and soon at the RHIC ion-ion collider at Brookhaven  makes the experimental situation also a dynamic one.

At short distances hadrons are described in terms of the pointlike quark-gluon, “parton” degrees of freedom. Much is know about the momentum spectrum of these partons, viewed in reference frames where the parent hadron has very high momentum. But precious little is understood about the correlations between the partons. For example, how they are distributed in the transverse plane is still a serious issue, and even the multiplicity distribution of the partons is not established. And the nature of the low-momentum tail of the distribution, the “wee” partons of Feynman, remains an active and unresolved issue.

Ordinary collisions at the very high energies of the LHC are another frontier. So many “wee” partons participate simultaneously in the typical central proton-proton collision that the complexity of the dynamics, e.g. the number of relevant constituent collisions per proton-proton collision, exceeds what will be dealt with in gold-gold collisions at RHIC. Diffractive processes comprise another difficult and unresolved subfield, one which increases in prominence with increasing energy. It is a “shadowy” topic which probably involves large-distance QCD concepts in an essential way.

Even the venerable subject of hadron spectroscopy ought to have a rich future. A modern “electronic bubble chamber”, built to exceed the classic bubble chamber’s acceptance and resolution, could improve the statistics and data quality of all the old resonance-physics topics by a millionfold. Many anticipated resonant states of hadrons, especially those made primarily of gluons, remain to be discovered and carefully studied.

I prefer to label this whole subfield “Hadron Physics”, in analogy to atomic, molecular, and nuclear physics. The goals of those subfields are quite analogous to the goals of hadron physics—namely to study the internal structure of their building blocks, as well as to study their “vacua”, namely the extensive, condensed-matter structures built from those constituents.

These subfields, especially atomic and nuclear physics, used to be within the mainstream of elementary particle physics. As the energy scale of interest to particle physics increased, each evolved into a distinct discipline. And I think this is already happening to hadron physics. Relatively little attention is paid nowadays by mainstream particle physicists to the subfield of hadron physics. And a great deal of the subject matter is now appropriated by the nuclear physics community, even though what they are doing can hardly be called nuclear physics.

While this evolution is basically a natural one, there is a new problem which was not faced in the previous examples. Much of the experimental program of hadron physics needs to be done via high energy collisions, within the facilities built and exploited by particle physicists. It is not easy for hadron-physics initiatives to be recognized and to receive the necessary priority rating, whether from laboratory managements, program committees, funding agencies, or the existing peer review system, when a program dedicated to hadron structure is put in direct competition with major high-energy physics initiatives. I believe that in order for such hadron-physics initiatives to be viable, there must be institutional changes at all these levels, which recognize the complementary nature of hadron-physics research, and which provide a certain degree of guaranteed access to the high-energy facilities in return for an appropriate contribution to the operating costs of the facilities .

## 9 Remarks

In summary, elementary particle physics in the next century should continue to be full of progress and full of vitality. While there is a slowing of the pace, it is still the case that within the next decade we should already witness a major turning point, namely a much better understanding of the problem of mass and of the mechanism of electroweak symmetry breaking. The way in which that problem is answered will have much to do in shaping the nature of the experimental program beyond.

The vitality of theory nowadays is focused on the superstring ideas, which continue to generate new ways of looking at the fundamental problem of going beyond the standard model. The only problem with this activity is its remoteness from data. This in itself causes me no problem. But if it leads to an indifference of theorists toward the data-driven side of the field, then I will have a problem.

When new data appears at a relatively slow rate, ideas and interpretations tend to ossify, and it sometimes becomes harder not only to think in new ways, but also to maintain a diversity of approach, which is always an essential element of the scientific endeavor. The existence of the Standard Model does not imply the existence of a standardized anticipation of the future. The only thing that deserves institutionalization is doubt. This problem of maintaining diversity of approach afflicts both experiment and theory, and if I have any concern about how the field is developing, it is about this point I worry the most.

In this respect, the second-tier initiatives such as the “factories”, and new generations of non-accelerator facilities are important to encourage. And I would add QCD and hadron-physics initiatives to this list as well. There is very much to do, most of which is relatively accessible provided the resources are made available. And last but not least, the long range problem of reaching extremely high energies should not be neglected, implying good support for advanced accelerator R&D.

## 10 Acknowledgments

It is a pleasure to thank the organizers of this conference for the excellent program and fine hospitality extended to us all. I also thank Pisin Chen, Leon Lederman, Chris Quigg, and Leonard Susskind for critical comments and advice.


format: text

text:
# Charge noise analysis of an AlGaAs/GaAs quantum dot using transmission-type radio-frequency single-electron transistor technique

\[

## Abstract

Radio-frequency (rf)- operated single-electron transistors (SETs) are high-sensitivity, fast-response electrometers, which are valuable for developing new insights into single-charge dynamics. We investigate high-frequency (up to 1 MHz) charge noise in an AlGaAs/GaAs quantum dot using a transmission-type rf-SET technique. The electron capture and emission kinetics on a trap in the vicinity of the quantum dot are dominated by a Poisson process. The maximum bandwidth for measuring single trapping events is about 1 MHz, which is the same as that required for observing single-electron tunneling oscillations in a measurable current ($``$0.1 pA).

\]

A single electron transistor (SET) is a high-sensitivity electrometer that measures a small fraction of the elementary charge, $`e`$, on a small island . If the island, or quantum dot atom, shows well-defined eigenstates in a zero-dimensional confinement potential, quantum mechanical effects found in atomic physics can be reproduced with tunable parameters . The ability to control single-particle quantum states will be helpful for developing quantum logic gates . So the dynamical behavior of charge states need to be studied for those interested. Most of the experiments have been done by taking the dc response from an ac modulation. However, direct measurement of the ac response of a single-particle state has not been done due to technical difficulties. Recently, a radio-frequency (rf) operated SET technique for following the fast response of the charge has been proposed and demonstrated by R. J. Schoelkopf et al.  . A bandwidth greater than 100 MHz would be useful for studying single-electron dynamics such as single-electron tunneling oscillation  and coherent charge oscillation , as well as for various sensors .

In this letter, we describe an application of the rf-SET for detecting individual emission and capture events of a trap in a semiconductor. A modified rf-SET technique that measures the transmission of rf signals through a resonator, is also used for a quantum dot fabricated in an AlGaAs/GaAs heterostructure . The charge noise of the quantum dot is studied both for low-frequency $`1/f`$ noise and for random telegraph signals (RTSs) originating from a trap near the dot. For a specific RTS, the statistics of the electron capture and emission are given by the Poisson process.

The transmission-type rf-SET technique is shown schematically in Fig. 1(a). The SET, or quantum dot, is fabricated in an AlGaAs/GaAs modulation doped heterostructure using focused ion beam implantation and patterning of fine Schottky gates . The two gate voltages, $`V_L`$ and $`V_R`$, control the two tunneling barriers independently, and effectively lift the electrostatic potential of the dot. The SET is placed in an LC resonator (two inductors of 2$`L`$ and one capacitor of $`C`$). Other lumped elements allow measurement of the dc current and rf-transmission simultaneously. When the rf carrier signal $`V_ie^{i\omega t}`$ is supplied at the resonant frequency, $`\omega =\frac{1}{\sqrt{LC}}`$, to the resonator, a transmitted signal $`V_te^{i\omega t}`$ appears at the other end of the resonator. The transmitted signal is sensitive to the resistance, $`R(q)`$, of the SET , or the charge on the island, $`q`$, i.e., $`V_t/V_i=(14Q^2Z_0/R(q))`$ for $`R(q)4Q^2Z_0`$ and $`Q1`$, where $`Z_0`$ = 50 $`\mathrm{\Omega }`$ is the cable impedance and $`Q`$ is the quality factor of the resonator. The rf excitation voltage across the SET, $`v_{ex}`$, is given by $`v_{ex}=QV_i`$.

Compared with the reflection-type rf-SET originally demonstrated in Ref. REFERENCES, the transmission-type rf-SET has some advantages for convenient measurements. Since the incident and the transmitted signals are separated, a directional coupler necessary for the reflection measurement is not required (although two inductors are necessary for the transmission measurement). The transmission signal always shows a clear and sharp resonance signal when the frequency is swept, while the reflection signal does not particularly if the sample is highly resistive or Coulomb blockaded. Once the frequency is adjusted properly, both transmitted and reflected signals behave similarly if the sample resistance is high, $`R(q)4Q^2Z_0`$. High-sensitivity and high-frequency (ideally upto $`\omega /Q`$) operation of the rf-SET could also be applied to the transmission-type rf-SET.

Figure 1(b) shows typical Coulomb blockade oscillations measured by dc current (upper trace) and rf-transmission (lower trace). The traces are taken simultaneously with a dc bias voltage of $`V_{SD}`$ = 0.2 mV and an rf excitation of $`v_{ex}`$ $``$ 0.1 mV (rms for all ac amplitudes). The best charge sensitivity is obtained at the largest slope, $`|dV_t/dV_g|`$, denoted by the arrow labelled by $`\alpha `$ in the figure. We estimate the charge sensitivity, i.e., the minimal detectable charge on the island, by applying a sinusoidal modulation to the $`V_L`$ as follows. The inset of Fig. 2 shows the frequency spectrum of transmitted signal $`V_t`$ at the modulation frequency, $`f_{mod}`$ = 10 kHz. The signal intensity is linearly dependent on the modulation amplitude, $`V_{mod}`$, as shown in Fig. 2. We obtain a charge sensitivity of $`\delta q=3.6\times 10^5e/\sqrt{Hz}`$ at an excitation of $`v_{ex}`$ $``$ 0.4 mV. This is larger than the thermal noise and the shot noise, but is dominated by the noise of the amplifier.

Figure 3(a) shows the spectra of the transmitted signal. The thick trace measured at the best charge sensitivity ($`\alpha `$ in Fig. 1(a)) shows the SET noise, while the thin trace measured at zero conductance of the SET shows the noise of the measurement system. The SET shows $`1/f`$ noise below 1 kHz, similar to that reported in metal SETs . For frequencies above 1 kHz, the white noise is dominated by the HEMT amplifier. Better amplifiers and a higher resonator $`Q`$ should reduce the system noise.

We also find a RTS in our dot. The specific trap we study has a relatively fast switching time and is a good example for demonstrating the fast response of the rf-SET. Figure 1(c) shows another CB oscillation with a jump seen at about -710 mV, labelled $`\beta `$ in the figure, where the jump appears at the largest $`|dV_t/dV_g|`$ of the CB peak for the best sensitivity. The jump is caused by the emission and capture of an electron from a trap in the vicinity of the quantum dot. The trap may be a defect in the crystal or a potential hollow (maybe a quantum dot) unintentionally created during fabrication. This specific trap is located close to the left gate rather than the right gate (schematically shown in the left inset of Fig. 4(a)) .

The RTS noise is observed on a relatively short time scale for this particular trap, as shown in Fig. 4(a). The emission and capture statistics are derived from the RTS noise in the time domain. The duration time for the trap being empty (lower signal in the figure) corresponds to the capture rate, and the duration time for the trap being occupied (higher signal) corresponds to the emission rate. These durations are widely distributed, as shown in Fig. 4(b), and well characterized by an exponential dependence, $`\mathrm{exp}(t/\tau )`$, with time constant $`\tau `$. The mean ($`m`$) and the standard deviation ($`\sigma `$) of the durations are practically the same as $`\tau `$. These observations indicate a Poisson process, i.e., each emission and capture process is an independent event . The time constant for capture, $`\tau _c`$, and emission, $`\tau _e`$, are strongly dependent on the energy level of the trap, $`E_t`$, or on the gate voltage $`V_L`$ (see Fig. 4(c)). Simply assuming a constant attempt rate for both transitions, $`\tau _0^1`$, and a Fermi distribution, $`f_{FD}(E)(1+\mathrm{exp}(E/k_BT_{eff}))^1`$, in the source electrode, the capture and emission rates should be given by $`\tau _c^1=\tau _0^1f_{FD}(E_tE_F)`$ and $`\tau _e^1=\tau _0^1(1f_{FD}(E_tE_F))`$, respectively. The measured time-constants are reproduced with parameters $`\tau _0=`$ 12 $`\mu `$s and $`k_BT_{eff}`$ = 0.46 meV, as shown by the solid lines in Fig. 4(c). The effective temperature, $`T_{eff}`$, is comparable to the rf excitation voltage across the SET, $`v_{ex}`$ $``$ 0.4 mV. We find that $`T_{eff}`$ increases with $`v_{ex}`$.

The power spectrum of the RTS noise (Fig. 3(b)) is flat up to a few kHz and decreases above 5 kHz. The spectrum can be fitted with a Lorentzian curve (dashed line), $`(1+(2\pi f\overline{\tau })^2)^1`$. The fitting parameter, $`\overline{\tau }`$ = 16 $`\mu `$sec, is comparable to $`\tau _e\tau _c/(\tau _e+\tau _c)`$ from the time domain analysis, which also indicates a Poisson process.

The RTS noise has been studied in sub-micrometer Si metal-oxide-semiconductor structures , AlGaAs/GaAs narrow channels , and quantum dots . A higher sample resistance in general restricts the lower frequency range that can be measured. Thus, most of the measurements on SETs ($`R`$ $`>`$26 k$`\mathrm{\Omega }`$ in principle) are restricted to the $`1/f`$ noise region . Our observation of RTS in the quantum dot is consistent with those previously reported. The maximum bandwidth $`B`$, where the change in the charge, $`\mathrm{\Delta }q`$, is distinguished from the noise, $`\delta q`$, is given by $`B\frac{1}{8}(\mathrm{\Delta }q/\delta q)^2`$. This yields $`B`$ $``$ 1 MHz for the trap we investigated. We observe a duration as short as few $`\mu \mathrm{sec}`$ and rise/fall time shorter than 1 $`\mu \mathrm{sec}`$ (the right inset of Fig. 4(a)), when the bandwidth is set to 2.5 MHz. If the trap is replaced with another quantum dot connecting another set of leads , single electrons will tunnel through the dot with a frequency of $`I/e`$ (600 kHz at a driving current, $`I`$, of 0.1 pA). The rf-SET should resolve each tunneling event of such a current.

In summary, we have demonstrated a transmission-type rf-SET using a quantum dot for detecting charge kinetics of a trap in an AlGaAs/GaAs heterostructure. Our results are encouraging for the direct measurement of ac response from a single-electron state.

We thank D. G. Austing, R. Espiau de Lamaestre, A. Kanda, S. Komiyama, B. Starmark for their discussion and help.


text:
# Digital Sky Surveys: Software Tools and Technologies

## Digital Sky Surveys: Software Tools and Technologies

Large digital sky surveys, over a broad range of wavelengths, both from the ground and from space observatories, are becoming a major source of astronomical data. Some examples include the Sloan Digital Sky Survey (SDSS) and the Digital Palomar Observatory Sky Survey (DPOSS) in the visible, the Two-Micron All-Sky Survey (2MASS) in the near-infrared, the NRAO VLA Sky Survey (NVSS) and the Faint Images of the Radio Sky at Twenty centimeters (FIRST) in the radio. Many others surveys are planned or expected, in addition to the previously named surveys. While most surveys are exclusively imaging, large-scale spectroscopic surveys also exist. In addition, a number of experiments with specific scientific goals, e.g., microlensing surveys for MACHOs, searches for near-Earth asteroids, are generating comparable data volumes. Typical sizes of resulting data sets (as of the late 1990’s) are in the range of tens of Terabytes of digital information, with detections of many millions or even billions of sources, and several tens of parameters measured for each detected source.

This vast amount of new information presents both a great scientific opportunity and a great technological challenge: how to process, and calibrate the raw data; how to store, combine, and access them using modern computing hardware and networks; and how to visualize, explore and analyses these great data sets quickly and efficiently. This is a rapidly developing field, which often entails collaborative efforts between astronomers and computer scientists.

Broadly speaking, the steps along the way include: obtaining the data (telescope/observatory and instrument control software), processing the data into catalogs of detected sources and their observed properties, calibrating the data, archiving the results, and providing the tools for their scientific analysis and exploration.

### Survey Pipeline Software

The actual gathering and processing of the raw survey data encompasses many steps, which can often be performed using a software pipeline. The first step involves hardware-specific data acquisition software, used to operate the telescopes and the instruments themselves. In principle this is not very different from the general astronomical software used for such purposes, except that the sky surveying modes tend to require a larger data throughput, a very stable and reliable operation over long stretches of time, and greater data flows than is the case for most astronomical observing. In most cases, additional flux calibration data are taken, possibly with separate instruments or at different times. Due to the long amounts of time required to complete a survey (often many years), a great deal of care must be exercised to monitor the overall performance of the survey in order to ensure a uniform data quality.

The next step is the removal of instrumental effects, e.g., flat-fielding of CCD images, subtraction of dark current for infrared detectors. Other than the sheer size of the data, this process is extremely similar to the usual astronomical observational data reduction techniques.

At this point one does some kind of automated source detection on individual survey images, be it CCD frames, drift scans, photographic plate scans, their subsections, or whatever other “raw” image format is being generated by the survey. This process requires a good understanding of the noise properties, which determines some kind of a detection significance threshold: one wants to go as deep as possible, but not count the noise peaks. In other words, maximize the completeness (the fraction of real sources detected) while minimizing the contamination (the fraction of noise peaks mistaken for real sources). Typically one aims for the completeness levels of at least 90%, and contamination of less than 10% in the first pass, and typically the source catalogs are purified further at subsequent processing steps.

Most source detection algorithms require a certain minimum number of adjacent or connected pixels above some signal-to-noise thresholds for detection. The optimal choice of these thresholds depends on the power spectrum of the noise. In many cases, the detection process involves some type of smoothing or optimal filtering, e.g., with a Gaussian whose width approximates that of an unresolved point source. Unfortunately, this also builds in a preferred scale for source detection, usually optimized for the unresolved sources (e.g., stars) or the barely-resolved ones (e.g., faint galaxies), which are the majority. This is a practical solution, but with the obvious selection biases, with the detection of sources depending not only on their flux, but also on their shape or contrast: there is almost always a limiting surface brightness (averaged over some specific angular scale) in addition to the limiting flux. The subject of possible missing large populations of low surface brightness galaxies has been debated extensively in the literature. The truth is that a surface brightness bias is always present at some level, whether it is actually important or not. Novel approaches to source, or more accurately, structure detection involve so-called multi-scale techniques.

Once individual sources are detected, a number of structural parameters are measured for them, including fluxes in a range of apertures, various diameters, radial moments of the light distribution, etc. , from which a suitably defined, intensity-weighted centroid is computed. In most cases, the sky background intensity level is determined locally, e.g., in a large aperture surrounding each source; crowding and contamination by other nearby sources can present problems and create detection and measurement biases. Another difficult problem is deblending or splitting of adjacent sources, typically defined as a number of distinct, adjacent intensity peaks connected above the detection surface brightness threshold. A proper approach keeps track of the hierarchy of split objects, usually called the parent object (the blended composite), the children objects (the first level splits), etc. Dividing the total flux between them and assigning other structural parameters to them are nontrivial issues, and depend on the nature of the data and the intended scientific applications.

Object detection and parameter measurement modules in survey processing systems often use (or are based on) some standard astronomical program intended for such applications, e.g., FOCAS, SExtractor, or DAOPHOT, to mention just a few of the programs often used in 1990’s. Such programs are well documented in the literature. Even if custom software is developed for these tasks, the technical issues are very similar. It is generally true that all such systems are built with certain assumptions about the properties of sources to be detected and measured, and optimized for a particular purpose, e.g., detection of faint galaxies, or accurate stellar photometry. Such data may serve most users well, but there is always a possibility that a custom reprocessing for a given scientific purpose may be needed.

At this point (or further down the line) astrometric and flux calibrations are applied to the data, using the measured source positions and instrumental fluxes. Most surveys are designed so that improved calibrations can be reapplied at any stage. In some cases, it is better to apply such calibration after the object classification (see below), as the transformations may be different for the unresolved and the resolved sources. Once the astrometric solutions are applied, catalogs from adjacent or overlapping survey images can be stitched together.

Object classification, e.g., as stars or galaxies in the visible and near-IR surveys, but more generally as resolved and unresolved sources, is one of the key issues. Classification of objects is an important aspect of characterizing the astrophysical content of a given sky survey, and for many scientific applications one wants either stars (i.e. , unresolved objects) or galaxies; consider for example studies of the Galactic structure and studies of the large-scale structure in the universe. More detailed morphological classification, e.g., Hubble types of detected galaxies, may be also performed if the data contain sufficient discriminating information to enable it. Given the large data volumes involved in digital sky surveys, object classification must be automated, and in order to make it really useful, it has to be as reliable and objective as possible, and homogeneous over the entire survey.

In most cases, object classification is based on some quantitative measurements of the image morphology for the detected sources. For example, star-galaxy separation in optical and near-IR surveys uses th fact that all stars (and also quasars) would be unresolved point sources, and that the observed shape of the light distribution would be given by the point-spread function, whereas galaxies would be more extended. This may be quantified through various measures of the object radial shape, e.g., moments of the light distribution in various combinations. The problem of star-galaxy separation thus becomes a problem of defining a boundary in some parameter space of observed object properties, which would divide the two classes. In simplest approaches such a dividing line or surface is set empirically, but more sophisticated techniques use artificial intelligence methods, such as the Artificial Neural Nets or Decision Trees (artificial induction software). They require a training data set of objects for which the classification is known accurately from some independent observations. Because of this additional information input, such techniques can outperform the methods where survey data alone are used to decide on the correct object classifications.

There are several practical problems in this task. First, fainter galaxies are smaller in angular extent, thus approaching stars in their appearance. At the fainter flux levels the measurements are noisier, and thus the two types of objects become indistinguishable. This sets a classification limit to most optical and near-IR surveys, which is typically at a flux level a few times higher than the detection limit. Second, the shape of the point-spread function may vary over the span of the survey, e.g., due to the inevitable seeing variations. This may be partly overcome by defining the point-spread function locally, and normalizing the structural parameters of objects so that the unresolved sources are the same over the entire survey. In other words, one must define the unresolved source template which would be true locally, but may (and usually does) vary globally. Furthermore, this has to be done automatically and reliably over the entire survey data domain, which may be very heterogeneous in depth and intrinsic resolution. Additional problems include object blending, saturation of signal at bright flux levels, detector nonlinearities, etc.

The net result is that the automated object classification process is always stochastic in nature. Accuracies better than 90% are usually required, but accuracies higher than about 95% are generally hard to achieve, especially at faint flux levels.

In other situations, e.g., where the angular resolution of the data is poor, or where nonthermal processes are dominant generators of the observed flux, morphology of the objects may have little meaning, and other approaches are necessary. Flux ratios in different bandpasses, i.e. the spectrum shape, may be useful in separating different physical classes of objects.

### Survey Archive Software

Once all of the data has been extracted from the image pixels by the survey pipeline software, it must be persisted in order to facilitate scientific exploration. Basically, this data archiving can take one of two forms: raw data access through the web or some suitable storage and distribution medium (e.g., CD-ROMs or data tapes), or utilization of a database system designed specifically to handle a given digital sky survey archive. The former storage technique, while considerably less expensive, is certainly not optimal for any real analysis, especially for very large data volumes. On the other hand, using a database system provides significant advantages (e.g., powerful query expressions) limited primarily by how much a survey can afford. Currently, most surveys are accessible from the Internet, whether it is a simple ftp location or a full featured, web-accessible, query engine.

While certainly flexible and generally easy to establish, web accessible data is subject to limitations on bandwidth. Despite technological advances which promise to ease such concerns, the rapid growth in available data will continue to swamp available resources. To illustrate this bandwidth problem, for a typical building network (e.g., shared Ethernet at 10 Mbit/s), it would take nearly a week and half to sift through a Terabyte of data (which is the current benchmark for large sky surveys), and this assumes a dedicated bandwidth. Even at fast SCSI speeds (e.g., 100 Mbit/s), it would take approximately one day to merely move the data of interest.

Each archival center is presented with the challenging problem of storing and serving vast amounts of complex data. Currently the majority of the software written for these applications is in either C++ or Java, while the actual data is transferred via ASCII, FITS, or XML. Recently, many of the major data centers have begun to work on sharing knowledge and expertise in order to simplify the development process, as well as improve the overall efficacy of astronomical archives. This work is leading to standards which dictate how archives can communicate with each other, how archives can describe themselves (i.e. their metadata, or data that describes the data), how archives can transfer large amounts of dynamic information, and how sources in different archives can be cross-identified.

In general, the data processing flow is from the pixel (image) domain to the catalog domain (detected sources with measured parameters). This usually results in a reduction of the data volume by about an order of magnitude (this factor varies considerably, depending on the survey or the data set), since most pixels do not contain statistically significant signal from resolved sources. However, the ability to store large amounts of digital image information on-line opens up interesting new possibilities, whereby one may want to go back to the pixels and remeasure fluxes or other parameters, on the basis of the catalog information. For example, if a source was detected (i.e. cataloged) in one bandpass, but not in another, it is worth checking if a marginal detection is present even if it did not make it past the statistical significance cut the first time; even the absence of flux is sometimes useful information!

The two most common types of database management systems used within astronomy are relational-based (where data are manipulated as tables), and object-based (where data is manipulated individually as objects). Each of the two methods have working sites which demonstrate the technology in action(e.g., the 2MASS project uses Informix, while the GSCII project uses Objectivity). In general, relational systems offer more features including third-party add-ons, and powerful query mechanisms due to their dominant position in the business world. On the other hand, object-based systems have shown higher performance and better potential for scaling to extremely large data sets (e.g., CERN is developing an object-based persistence solution to multi-Petabyte archives).

Regardless of how the data is actually persisted, with the advent of the Internet, essentially all archives are Web accessible. This trend away from tabular or media based archiving produces datasets which are distributed in nature, allowing users and tools equal access to vast amounts of data, which in the past were nearly impossible to efficiently query. As a result, astronomical data is now able to be utilized in a more democratic fashion resulting in uses which the survey institutions did not even imagine.

### Scientific Analysis and Exploration

The end goal of digital sky surveys is to enable scientific investigations. In some cases, the goals are so specific that the best approach is to use analysis software designed and optimized to tackle them, e.g., as in the studies of the cosmic microwave background, in gravitational microlensing experiments, etc. However, most of the modern survey data sets are so information-rich, that a wide variety of different scientific studies can be done with the same data. This entails some general tools for the exploration, visualization, and analysis of large survey data sets.

The tools which will be utilized with the new datasets will need to employ cutting-edge techniques from computer science. For example, clustering techniques to detect rare, anomalous, or somehow unusual objects, e.g., as outliers in the parameter space, to be selected for further investigation (i.e. follow-up spectroscopy). Other examples include genetic algorithms to improve current detection and supervised classification methods, new data visualization and presentation techniques, which can convey most of the multidimensional information in a way more easily grasped by a human user, and the use of semi-autonomous AI or software agents to explore the large data parameter spaces and report on the occurrences of unusual instances or classes of objects.

The nature of sky surveys and the resulting data sets dictates the kinds of science one may wish to do with them. For example, most surveys would cover a large solid angle, but not go as deep as the typical “pointed observations”. This, in turn, drives the kinds of data analysis software needed for their scientific exploration. Broadly speaking, the kinds of astronomical investigations for which surveys are well suited include the following:

Combining surveys done at different wavelengths, for a more panchromatic view of the universe. Typically, optical IDs are needed for most of the follow-up work. An obvious example are the optical identifications of radio or x-ray sources. A typical basic software requirement is efficient and accurate matching of sources detected in different wavelengths. If positional accuracies and source densities in matched surveys are comparable this is a relatively straightforward task; however, if either of these two conditions is not satisfied, more sophisticated algorithms may be needed to assign matching probabilities to multiple candidates. These techniques can generally be optimized for certain classes of sources by utilizing as much a priori knowledge as possible.

Studies of the large-scale structure (if one detects galaxies in copious numbers), or studies of the Galactic structure (if one detects stars in copious numbers). The sheer numbers of detected sources make the Poissonian fluctuations unimportant, but systematic errors may limit the results. Accurate and uniform flux calibrations and source classifications are essential for such applications, and possible biases should be modeled. In addition, the sheer size of most surveys allows for further subdivision of various analysis (e.g., by morphological or spectral type) for more specific results.

Searches for rare types of objects or groups of objects, whose frequency of occurrence is so low that huge input source catalogs are necessary in order to find them. This “needle in a haystack” search is facilitated by the vast number of sources in current surveys (i.e. a one in a million object would appear a thousand times in a billion source survey). Typically such rare objects are found as outliers in some parameter space, e.g., colors: the bulk of “ordinary” sources (normal stars, galaxies, etc. ) would form well-defined clusters in the parameter space of observables, and some rare or peculiar types of objects may be found away from them. Unsupervised classification and cluster analysis techniques may be especially useful for this task. This data-mining also facilitates significantly higher efficiencies in performing follow-up spectroscopy of unusual objects. We note that rare objects may belong to known classes (e.g., distant quasars, brown dwarfs, etc.), which can be known to occupy some a priori determined portion of the given parameter space. Alternatively, there may be previously unknown or unexpected types of objects discovered in this manner.

### Future Directions

Astronomy finds itself at an interesting time, as several, large area, digital sky surveys are currently underway. In the future, the number of such surveys, as well as the wavelength coverage, will only continue to increase, providing temporal as well as spatial coverage of the sky. This flood of data necessitates a new approach to data handling – a virtual observatory (see Figure 1). Individual surveys are important in their own right, but the federation of multiple, cross-wavelength digital surveys provides a tremendous opportunity to truly quantify the origins of stars, galaxies and the universe itself.

This future situation provides the opportunity to adopt a new research paradigm for studying the heavens, as the ability to perform cutting edge research will not be restricted to those fortunate enough to have access to the best facilities. Instead, anyone who has the diligence and ability to sift through the avalanche of data can perform novel science.

### Further Readings

The field of large digital sky surveys, the related software methodologies, and the new astronomy they enable are developing very rapidly as of this writing (late 1999). There are thus no standard texts or reviews, and whatever articles do exist tend to become obsolete very quickly. Probably the best way to find more up to date information is through the World Wide Web (and the Interested Reader will know how). While many web sites can be somewhat ephemeral, we suggest one which will provide a number of useful links in this field, and which we hope will be active throughout the useful shelf life of this Encyclopedia: http://www.digital-sky.org

Another possibility is conference proceedings on this and related subjects, e.g., the series of volumes on “Astronomical Data and Software Systems”, published by the Astronomical Society of the Pacific in their Conference Series. A number of useful papers can also be found in the proceedings of IAU Symposium 179, “New Horizons From Multi-Wavelength Sky Surveys”, eds. B.J. McLean et al. , Dordrecht: Kluwer Academic Publ. (1998).

An example of a particular software system for processing of digital sky surveys is described by Weir, N., et al. 1995, Publ. Astron. Soc. Pacific 107, 1243.

S. George Djorgovski
Robert J. Brunner