text:
# The Optical/Infrared Astronomical Quality of High Atacama Sites. II. Infrared Characteristics

## 1 Introduction

Atmospheric molecules are responsible for band absorption of cosmic radiation. In the near and mid–IR, the most important absorbers are the triatomic molecules H2O, CO2 and O3. Between absorption bands, partially transparent windows appear, making astronomical observations possible in the near and mid–IR from the ground. The atmosphere is virtually opaque between 50 and 200 $`\mu `$m from all sites on the ground, making that wavelength region the exclusive territory of airborne and space observatories. Water vapor also dominates absorption in the far IR and submm region; atmospheric windows appear near 200, 350, 450, 600, 750 and 870 $`\mu `$m; in this spectral region transparency is significant only in dry, high altitude sites (Tokunaga 1998; Wolfe & Zissis 1978). H2O molecules have a short residence time (a few days) in the atmosphere and their concentration is highly variable. Thus, the characterization of potential astronomical sites for operation in the IR and submm spectral regimes depends very strongly on measurements of atmospheric water vapor.

In a companion paper (Giovanelli et al. 2000; Paper I), we have discussed the desirability of ascertaining the quality of high altitude sites for astronomical research in the optical and infrared parts of the spectrum. Paper I describes briefly the Atacama desert region and, as part of a site survey campaign, the results of optical seeing measurements. Here, we review the results of measurements carried out in the Andean plateau known as the Llano de Chajnantor, in the Atacama desert of northern Chile, aimed at gauging the atmospheric transparency and emissivity in the infrared. The main characteristics of the atmospheric water vapor in the region are discussed: its absolute amount, annual, seasonal and diurnal variations, as well as its vertical distribution. The latter information provides indications of the relative infrared transparency at sites at varying elevation above the plateau, even of those currently inaccessible for deployment of testing equipment.

In Section 2 we briefly describe the data utilized for the analysis; in Section 3, statistics of the total precipitable water vapor (PWV, defined as the atmospheric column density of water vapor) are given, and in Section 4 the vertical distribution of PWV is analyzed, with estimates of its value expected for candidate sites for an optical/IR telescope. Finally, in Section 5 we estimate atmospheric transparency, emissivity and telescope sensitivities in the infrared, for a range of possible water vapor conditions.

## 2 Data Sources

We make use of two kinds of data in this report:

* Atmospheric opacities measured with automated tipping radiometers at 225 GHz and at 183 GHz. These instruments are operated by the National Radio Astronomy Observatory (NRAO) , the European Southern Observatory (ESO) and the Onsala Space Observatory. Data taken since April 1995 can be found at http://www.tuc.nrao.edu/mma/sites/Chajnantor and http://alma.sc.eso.org. We shall refer to summaries of relevant data products as given at those sites.
* Radiosonde data from balloon launches initiated in October 1998, jointly operated by Cornell University, NRAO, ESO and the Smithsonian Astrophysical Observatory. We will summarize results from radiosondes launched between October 1998 and August 2000. Details on individual launches can be found at http://www.astro.cornell.edu/atacama/sondedata.html and at the above web sites of NRAO (Radford 2000) and ESO (Otarola 2000). A more detailed analysis of the full radiosonde data set will be presented in a forthcoming paper, by the ‘radiosonde consortium’.

Tipping radiometers yield values of the atmospheric opacity, from which estimates of PWV above the radiometer location can be inferred. Radiosondes, on the other hand, provide vertical profiles of atmospheric parameters (temperature, pressure, relative humidity, wind speed and direction); from those, the water vapor density distribution and thus the PWV above any altitude can be inferred. Several summits, which are potentially attractive sites for infrared telescopes, have no access roads and thus installation and access to equipment at those locations is impractical. The free atmosphere parameters obtained from radiosonde launches provide a rough approximation of conditions that would be found at those summits. In the remainder of this paper, we shall always refer to PWV at zenith.

The sites for radiometer data taking and radiosonde launches are within one km from each other, near latitude S 23 01’, longitude W 67 46’, at an altitude near 5000 m above mean sea level. It should be kept in mind that, depending on wind speed and direction, a sonde flight samples an oblique path through the atmosphere, rather than a vertical one. In strong wind. a sonde profile typically spans tens of km in horizontal range. However, since most of the atmospheric water vapor is generally distributed within a few km from the ground, the effective sampling of most of the PWV corresponds to a horizontal range of only a few km from the launch site. Given the prevalent wind direction (see Paper I for details), sondes usually fly towards the East over unpopulated terrain, with ground elevation ranging between 4500 m and 5800 m above mean sea level.

In this paper, we shall use data from 108 radiosonde launches distributed as follows: 8, 10 and 16 launches in October, November and December 1998 respectively; 1, 5 and 33 in February, March and November 1999 respectively; 11, 18 and 6 in June, July and August 2000 respectively. Thirty were launched in nighttime hours (UT 01 to 11 hours), 65 in daytime (UT 12 to 21 hours) and 13 in the early evening (UT 22 to 00 hours). For each sonde profile we integrate the water vapor to obtain PWV above the plateau level of 5050 m, PWV above the elevation of 5400 m (equivalent to the summit of Cerro Honar) and PWV above 5750 m (equivalent to the summit of Cerro Chascón). We also compute the height above the plateau at which water vapor density drops to $`1/e`$ of the value at the plateau level, $`h_e`$, an indication of the exponential scaleheight of the water vapor layer. Since the water vapor distribution is most often not close to an exponential, we also measure ($`h_{1/2}`$), the height at which PWV drops to 50% of the value at the plateau level. Finally, we note the location of temperature inversions, which often are seen trapping large fractions of the total PWV below them. The derivation of these parameters and a detailed presentation of the sonde data will be given in a ‘consortium paper’, as mentioned above.

## 3 Precipitable Water Vapor Above 5000 m

The conversion from radio opacities to PWV can be obtained from models of the atmosphere and assumptions on the water vapor line shapes. These are usually fairly uncertain, especially in the case of 225 GHz observations (e.g. see Holdaway et al. 1996). The conversion is more reliable in the case of 183 GHz data, for the nearby water line is stronger and the “wet” contribution to the opacity larger. Delgado et al. (1999) give a reliable conversion relation directly from antenna temperature to PWV from 183 GHz observations, and have also derived a conversion relation from $`\tau _{225}`$ to PWV, which we use:

$$\tau _{225}=0.0435(PWV)+0.0068$$
(1)
Based on preliminary comparisons, Equation 1 yields good agreement with PWV derived from radiosonde data for values of PWV$`<3`$ mm; for higher values, however, PWV may be overestimated by Eqn. 1. Fortunately, the latter cases are of relatively less interest in our case.

The most extensive data base of radiometric opacities at the Chajnantor Plateau is that derived from the NRAO data base (Radford 2000). The historical quartiles between April 1995 and October 2000 are respectively 0.036, 0.060 and 0.114, which convert to PWV values of respectively 0.67, 1.22 and 2.46 mm. The PWV quartiles of 108 radiosonde flights between October 1998 and August 2000 are: 0.71, 1.04 and 1.85 mm. The difference between radiosonde and tipping radiometry values of PWV is not very surprising, for variations in the median PWV from year to year are large and the time interval for sonde launches is less than two years. Moreover, the radiosonde launches underemphasize the Summer season, when PWV is much larger than in the rest of the year. The value of PWV inferred from radiometry provides a more reliable long–term estimate for the Chajnantor Plateau, although it should be kept in mind that the epoch of radiometry measurements includes the exceptional El NiñoLa Niña episode of 1997–98, the most extreme on record (McPhaden 1999), which may bias high the PWV median.

Seasonal variations in PWV are conspicuous. In the months of January and February, PWV is several times higher than in the winter months. This is the result of moist Amazon air flowing from the NE, a condition locally referred to as “Bolivian Winter” and often accompanied by precipitation above 4000 m elevation. Inspection of the 225 GHz radiometry record indicates that the median PWV at Chajnantor between March and early December is less than 1 mm.

The diurnal cycle also affects PWV, which tends to be higher in the afternoon hours, followed by a rapid decrease after sunset. Figure 1 displays the diurnal variation of PWV above 5000 m, above 5400 m and above 5750 m, respectively top to bottom, as derived from radiosonde data (unfilled circles). Median values over 3 hr intervals are indicated as large, filled squares, connected by a solid line. The dashed line displays the median values of PWV, as obtained from 225 GHz opacities at 5000 m. Local midnight takes place at 04:31 UT. The atmosphere appears to be driest in the late part of the night and early part of the morning. The diurnal cycle of PWV lags in phase the sunlight cycle by about 4 hours, and has amplitude of about 20% about the median value.

## 4 Vertical Distribution of Precipitable Water Vapor

Table 1 displays quartile values of PWV above 5000, 5400 and 5750 m for four different groupings of the radiosonde flights: in addition to the daytime and nighttime sets described in Section 2, we also use the combined set of 108 sondes (“All”) and that of 32 sondes launched between UT 05 and UT 13 hrs. The latter corresponds to the UT interval in which the minimum of the diurnal cycle is seen. We note that with respect to the value of PWV at the plateau level overall, PWV drops by 30% in the first 400 m, and by a factor of 2 in the first 750 m above the plateau. That decrease may be even steeper during nighttime hours, suggesting that the effective thickness of the water vapor layer decreases at night. While the absolute values of PWV listed in Table 1 may be lower than historical values, as indicated by the comparison with 225 GHz opacities and discussed in the preceding Section, the variation of PWV with elevation is reliable.

Whether the decrease in PWV with height will be that of the free atmosphere indicated in Table 1, or lower, will depend on the local topography, the area of the summits, the direction of the wind and other factors. However, it is clear that by accessing higher ground above the plateau, significant gains in terms of infrared transparency and background emission can be obtained.

The thickness of the water vapor layer can be measured by $`h_e`$ or $`h_{1/2}`$. For the set of 108 radiosondes, we find the following quartile values:

| $`h_{1/2}`$ | : | 571, | 836, | 1083 | m |
| --- | --- | --- | --- | --- | --- |
| $`h_e`$ | : | 836, | 1135, | 1504 | m |

Examining the relative humidity records from stations at different elevations at the Japanese testing site of Rio Frio, a region to the south of the Chajnantor Plateau, Holdaway et al.  (1996) found indications that the scaleheight of the water vapor layer may vary with the diurnal cycle, between values of approximately 2 and 1 km, the low value being attained at night. Figure 2 yields support for that suggestion. Values of $`h_e`$ and $`h_{1/2}`$ for each in the set of 108 radiosonde flights are displayed against time of day. Median values are identified by large filled squares. A diurnal oscillation in the thickness of the water layer, of amplitude near 25% about its median value, may be present. The cycle appears to be in phase with that of the solar illumination cycle. The statistical significance of this result is low, the lack of sonde launches between UT 04 and 09 hrs being particularly important in this respect. We next present corroborating evidence regarding this potentially important result.

In Figure 3 we show the vertical distribution of the water vapor density in the Chajnantor median atmosphere (similar plots for the temperature distribution and the wind speed are shown in Paper I). The thick solid line tracks the median water vapor density profile over the plateau between April and December, from 108 radiosonde profiles. The distribution is well approximated by an exponential with scaleheight $`h_e=1.135`$ km, as discussed above. The median profiles for day and night radiosonde launches do however depart significantly from exponential behavior: during night, more water vapor is found at lower elevations, i.e. the effective thickness of the water layer decreases. As pointed out in teh previous section, median PWV between UT 01 hr and 11 hr (night) is about the same as between UT 12 hr and 21 hr (day), although the median value between UT 05 hr and 13 hr is significantly lower.

On any given time, the departures of the water vapor distribution from an exponential can be quite severe. Figures 4, 5 and 6 display data for three different radiosonde launches. In panel (a) of each figure, the air temperature and the dewpoint temperature are displayed; in panel (b), wind speed and direction are shown; in panel (c), the relative humidity and the water vapor pressure are given, while in panel (d) we have the water vapor density and PWV. The temperature and water vapor profiles of Figure 4 are unusually smooth. The water vapor density distribution appears exponential, with a scaleheight of about 1420 m. Profiles such as that tend to be the exception. More often, temperature inversions are present, and the water vapor distribution is raggedly uneven. In the case of Figure 5, for example, two prominent temperature inversions take place near altitudes of 6600 and 8200 m. The atmospheric water vapor lies mostly under the lower of the two inversion layers, and the vertical distribution of the water vapor density is far from exponential in shape. In Figure 6, an inversion layer is present near 5300 m, and half of all water vapor above the Chajnantor Plateau is packed in the lowest 400 m of atmosphere. Were circumstances such as that displayed in Figure 6 frequent, peaks that rise several hundred m above the plateau would be highly attractive candidate sites for infrared observations. We have reviewed all radiosonde profiles and selected those for which temperature inversion layers can be identified. In Figure 7, we plot the altitude of the lowest (in case that more than one is discernible) inversion visible in the temperature profile, separately for day and night radiosonde flights. The fraction of temperature inversions taking place below 500 m above the plateau level is far larger during the night than during the day, by an amply significant margin. The nightly decrease in the thickness of the water vapor layer appears to be related to the development of temperature inversions close to the ground.

The occurrence and altitude of temperature inversions is of importance not only for their impact on the PWV, and thus the infrared transparency, of a site, but also for the quality of astronomical seeing. Evidence exists (Hufnagel 1978) that in those layers large values of the refractive index structure constant $`C_n^2`$ occur. The seeing disk of stellar images, $`\theta _{hpfw}C_n^2𝑑z`$, can thus be strongly affected. A site above the inversion layer would hence yield better astronomical image quality.

## 5 Atmospheric Transparency in the Infrared

### 5.1 Atmospheric Transparency and Thermal Background

Figure 8 displays curves of atmospheric transparency at zenith, computed for a site at an altitude of 5000 m and different column densities of water vapor above head, for the window between 5 $`\mu `$m and 1 mm. In the spectral region between 50 and 200 $`\mu `$m the atmosphere is quite opaque from all ground sites: even if PWV were as low as 100 $`\mu `$m, transparency would not reach 40%. However, the window near 200 $`\mu `$m may be of interest for observations of the very important spectral line of $`C^+`$. For low values of PWV, numerous atmospheric windows appear longwards of 10 $`\mu `$m, notably near 20, 24, 32, 34, 38, 42 and 46 $`\mu `$m.

The main deleterious effects of the atmosphere on astronomy consist in reduced transparency to cosmic photons from a given line of sight, seeing degradation of optical and infrared images and an increase of the diffuse background, against which the generally weak signals of cosmic sources need to be discriminated. The effective temperature of the most relevant atmospheric layers is between 200 and 300 K, so the atmospheric thermal emission peaks near 15 $`\mu `$m. Near the centers of the absorption bands where the atmosphere is opaque, a maximum background flux is reached and, of course, no cosmic photons get through. In spectral regions of partial transparency, the atmospheric background radiation can be approximated by that of a blackbody attenuated by a factor which is the complement of the transparency. If opacity is distributed over a wide range of elevations, this assumption may be flawed; it is however a good approximation for the opacity produced by water vapor, which is found very close to the ground, and is our main concern in this paper. In the optical and near IR, the principal source of atmospheric background is the scattering of solar radiation, during the day. At night, however, airglow in the near IR results from transitions between vibrational states of the OH- radical, discharging energy stored during daytime from solar radiation, through the dissociation of ozone. Thus OH- airglow originates at heights of 70 to 90 km and affects all ground based locations, independently on altitude.

In addition to the atmospheric background, astronomical observations need to contend with cosmic backgrounds, such as that produced by the thermal emission and the scattering of solar photons by interplanetary dust, the interstellar dust emission, the cosmological microwave background radiation and the thermal emission of the telescope and detectors themselves.

We next compute background levels for various atmospheric conditions; in doing so we follow the approach of Thronson et al. (1995).

Let’s first consider the “near field” thermal background. In the atmospheric windows of interest the atmosphere is optically thin, so that the specific intensity of the emitted radiation can be approximated by

$$I_\nu =\tau _\nu B_\nu =\frac{2\tau _\nu (h\nu ^3/c^2)}{exp(h\nu /kT)1}$$
(2)
where $`B_\nu `$ is Planck’s function and $`\tau _\nu `$ the optical depth. Thus the number of photons per unit time. originating with the thermal background of temperature $`T`$ and striking the telescope within the solid angle subtended by a point source $`\mathrm{\Omega }_{ps}`$, is

$$=I_\nu 𝒜\mathrm{\Omega }_{ps}\mathrm{\Delta }\nu /h\nu $$
(3)
where $`𝒜`$ is the collecting area of the telescope and $`\mathrm{\Delta }\nu `$ is the detection bandwidth. Converting to photoelectrons per second and assuming a detector gain of 1

$$_e=2.22\times 10^{15}\frac{𝒯QE}{}\frac{1e^\tau }{\lambda _\mu [exp(1.44\times 10^4/\lambda _\mu T)1]}[\mathrm{e}\mathrm{s}^1]$$
(4)
where $`𝒯`$ and $`QE`$ are respectively the system transmission and quantum efficiency over the band, $`\lambda _\mu `$ the wavelength in $`\mu `$m, and $``$$`=\nu /\mathrm{\Delta }\nu =\lambda /\mathrm{\Delta }\lambda `$. Equation 4 is derived assuming that the telescope is diffraction limited at all wavelengths, so that $`\mathrm{\Omega }_{ps}`$ is the solid angle subtended by the ring corresponding to the first diffraction null and $`𝒜\mathrm{\Omega }_{ps}`$ is then $`3.7\lambda _\mu ^2`$. This assumption makes the background estimate independent on telescope aperture, allowing an intercomparison of different sites or atmospheric conditions. At the shorter wavelengths, at which the seeing angle is larger than the diffraction limit of the aperture, such an assumption is incorrect, and $`\mathrm{\Omega }_{ps}`$ should be replaced with the solid angle which encircles 90% of the light within the seeing disk.

Different sources of thermal emission yield an additive contribution to $`_e`$. For the atmospheric thermal background, we approximate $`\tau `$ with the complement to one of the transparency as displayed in Figure 8. For the thermal emission of the telescope, we replace $`\tau `$ in Eqn. 4 with a constant emissivity $`ϵ_{tel}`$.

As for the cosmic backgrounds, following Tokunaga (1998) and using Equation 4, we approximate the zodiacal emission of dust by that of a blackbody with $`T=275`$ K attenuated by $`ϵ=7.1\times 10^8`$; the sunlight scattered by the interplanetary particles by a blackbody of $`T=5800`$ K and $`ϵ=3\times 10^{14}`$; the interstellar dust emission by a blackbody of $`T=17`$ K and $`ϵ=10^3`$; and the cosmic microwave background by a blackbody with $`T=2.73`$ K and $`ϵ=1`$.

Figure 9 illustrates the background $`_e`$ for an Atacama telescope with $`ϵ_{tel}=0.05`$, estimated for the very best atmospheric conditions of $`PWV=0.2`$ mm and a seeing of 0.35” at 0.5 $`\mu `$m(as shown in Table 1, PWV=0.2 mm is the value for the lowest quartile at a high elevation site, and seeing near 0.35” may have similar frequency at such a site; the chosen combination of values may occur in 5–20% of the time, depending on how correlated seeing and PWV are; of eight reliable events of simultaneous radiosonde and seeing measurements, the four with PWV$`<1`$ mm yield an average seeing of 0.51” at 0.5 $`\mu `$m, while the other four, with PWV$`>1`$ mm, have average seeing of 0.84”; more data is needed to confirm this possible correlation). In the same figure, we also display estimates of the background for a space telescope passively cooled (e.g. the Next Generation Space Telescope, NGST), a cryogenically cooled space telescope (e.g. the Space Infrared Telescope Facility, SIRTF), and a stratospheric, telescope airborne at 12 km altitude (above which PWV is assumed to be 10 $`\mu `$m, $`ϵ_{tel}=0.15`$, and image quality of 2” at 1 $`\mu `$mfor SOFIA (this may be an overestimate of image quality: while the optical quality of the instrument is high, pointing jitter and seeing produced by the turbulent air flow near the airplane may produce worse values). We have assumed a telescope temperature of 260 K for Atacama, 240 K for SOFIA, 60 K for NGST and 5.5 K for SIRTF. The temperature of the emitting atmospheric layers was assumed to be 255 K for Atacama and 230 K for SOFIA. For simplicity, we assume $`𝒯QEG/=1`$, which overestimates the bandwidths that are possible in some of the atmospheric windows. Telescope and atmospheric thermal backgrounds dominate in the case of all ground–based instruments, while in the case of space instruments, even if not cryogenically cooled, the zodiacal emission dominates. Between 5 and 15 $`\mu `$m, for example, the NGST will have a background about six orders of magnitude lower than that for an Atacama telescope. While airborne at high altitude, the SOFIA telescope itself will produce substantial thermal emission, and lower image quality than can be achieved from the ground. Note that the background in good Atacama conditions and for Sofia are comparable. SOFIA will, however, have access to atmospheric windows inaccessible from the ground.

### 5.2 Comparison of Broad Band Sensitivities

The signal–to–noise ratio $`𝒬`$ for a single integration of time $`t`$ can in general be expressed as (Thronson et al. 1995)

$$𝒬=\frac{𝒮t}{[(𝒮+_e+𝒟)t+R^2+N_ct^2]^{1/2}}$$
(5)
where $`𝒮`$ is the source signal, $`𝒟`$ is the detector dark current, $`R`$ its readout noise and $`N_c`$ is the signal produced by source confusion. A given signal, such as $`S`$, relates to the flux density via

$$𝐒(\mathrm{mJy})e^\tau =6.63\times 10^5\frac{}{QE𝒯𝒜}𝒮$$
(6)
We adopt the following detector performance characteristics for the 3–40 $`\mu `$m range: $`QE0.50`$, $`𝒟10`$ $`e^{}s^1`$ and $`R50`$ electrons rms (Thronson et al. 1995; Stacey 2000; Brandl & Pirger 2000), and source confusion does not become an issue shortwards of 50 $`\mu `$m, as we shall discuss in the next Section. It is thus safe to assume that in the near and mid–IR the thermal background dominates for broad band observations, so that the minimum detectable signal, for an exposure of time $`t`$ achieving signal–to–noise $`𝒬`$, according to Equation 5 is

$$𝒮_{min}𝒬[_e/t]^{1/2}.$$
(7)
We use Equations 4 and 7 to estimate limiting fluxes for point sources, assuming $`𝒯`$$`=0.4`$ and, as indicated above, $`QE=0.5`$. The broadest bandwidths usable are, of course, limited by atmospheric absorption. Sensitivities are computed for $`𝒬=5`$ and exposures of $`t=10^4`$ sec. We underscore the fact that the chosen bands are those accessible from the ground: SOFIA, SIRTF and NGST will access spectral regions inaccessible from the ground. The results are plotted in Figure 10, for a telescope at Atacama with a 15 m diameter, NGST of 6 m diameter, SIRTF with a 0.85 m diameter and SOFIA with a diameter of 2.4 m. In the case of the Atacama telescope, we have assumed that the image size is seeing–limited at 0.35” at 0.5 $`\mu `$m. Note that with an adaptive optics system of moderate to good efficiency, the near infrared sensitivity can improve by one order of magnitude, beyond the limit in Figure 10, provided that it does not significantly increase the telescope emissivity. The implementation of an adaptive secondary would thus be desirable and that of a multiconjugate system would raise more serious emissivity concerns. The point source sensitivity plotted in Figure 10 for J band is equivalent to 26th magnitude; with an AO system that limit would approach 29th magnitude. The telescope labels are the same as for Figure 9. The SIRTF sensitivities have been obtained from the SIRTF website (http://ssc.ipac.caltech.edu/sirtf/Mission) and links referred therein. The NGST sensitivities have been plotted for the same bandpasses and $``$ as for the Atacama telescope, although it should be clear that the whole near IR spectrum will be accessible to NGST. For comparison, we also plot the flux density curves of the brown dwarf Gliese 229B at the source’s distance of 5.7 pc and at 1 pc (Matthews et al. 1996).

The thermal background does not always constitute the main limitation at submillimeter wavelengths. Source confusion can become important. The sky source density at far IR and submm wavelengths is relatively uncertain, but it is unlikely to be a concern even for large integrations ($`10^4`$ s) for large aperture telescopes (diameter greater than 10 m). Assuming $`𝒯=0.4`$, $`QE=0.5`$ (a quantum efficiency which can be approached by detectors with modern bolometers) and optimal atmospheric conditions as can be found in Atacama (PWV=0.2 mm), an observation with Q$`=5`$ and $`t=10^4`$ s can reach a sensitivity on the order of 1 mJy at all the atmospheric windows between 300 $`\mu `$m and 1 mm, and about 3 mJy for those between 200 and 300 $`\mu `$m.

### 5.3 Comparison of Spectroscopic Sensitivities

Suppose the flux density contributed at a given frequency by a spectral line is S. The integrated flux over the whole line is $`S𝑑\lambda `$. If the spectral line is unresolved, the flux associated with the line is $`S\mathrm{\Delta }\lambda =S\lambda /`$, where S is now the mean line flux over the spectral channel which contains the line. If we measure the integrated flux in W m-2, then the signal $`S`$ in electrons per second can be converted to

$$F(\mathrm{W}\mathrm{m}^2)=2.0\times 10^{19}\frac{S}{\lambda _\mu AQE𝒯}$$
(8)
In spectroscopic observations, the high $``$ reduces the impact of the high thermal background, as illustrated by Equation 4. For observations with $``$$`=10^5`$, for example, the thermal background contribution, which is inversely proportional to $``$, will be several orders of magnitude lower than in the case of photometric observations. Consider Equation 5: $`_e`$ can become comparable with or smaller than $`𝒟`$, in which case the observation will be limited by detector noise. For values of $`𝒟10`$ e- s-1 and $`R50`$ electrons, the background can easily become negligible with respect to $`𝒟`$ in space–based instruments. The readout noise on the other hand is likely not to play a role if exposures exceed a few hundred seconds, because then $`R^2<𝒟t`$. We will assume here that $`R^2`$ and the source confusion term in the denominator of Equation 5 are negligible.

The minimum detectable integrated line flux for an integration time $`t`$, with a signal–to–noise ratio $`𝒬`$, can be written, in analogy to Equation 7, as

$$F_{min}(\mathrm{W}\mathrm{m}^2)=2.0\times 10^{19}\frac{𝒬}{\lambda _\mu AQE𝒯}\left[\frac{^{}}{t}\right]^{1/2}$$
(9)
where $`^{}`$ is $`_e+𝒟`$. Note that, for a space–based instrument, near 5 $`\mu `$$`_e5\times 10^3`$ e- s-1 for $``$$`=10^5`$, so $`_e𝒟`$, and the instrument is detector–limited. In Figure 11, we show sensitivity limits computed using Equation 9, for $``$$`=10^5`$, $`𝒟`$=10, $`QE=0.5`$, $`𝒯`$=0.4 and $`𝒬`$=5, except for SIRTF, for which $``$ is only 600.

## 6 Conclusions

The water vapor distribution above the Chajnantor plateau appears to have the following properties:

* The median PWV above the plateau at 5000 m is $`1.2`$ mm. Variations from year to year can be as large as 50% of that value.
* Seasonal variations in PWV are also large: median values for the 8 months from April to November are 30% lower than the yearly median, while median values for the Summer months may be more than twice as large.
* PWV varies by about $`\pm 20`$% with the daily cycle, with a phase lag about 4 hours behind that of sunlight: minimum PWV occurs between midnight and noon, and maximum occurs at sunset.
* The distribution of the water vapor of the median atmosphere over Chajnantor is well aproximated by an exponential, with a scaleheight of 1.13 km. At any given time, however, the water vapor distribution can depart from an exponential shape, more dramatically when temperature inversions occur.
* The thickness of the water vapor layer appears to vary in phase with the solar illumination cycle by about $`\pm 25`$%; minimum is near local midnight.

The implications of these effects for summits in the vicinity of the Chajnantor Plateau are:

* The median PWV above 5400 m elevation drops by one third with respect to the value measured at the plateau; above an elevation of 5750 m it drops by a factor of 2.
* Median PWV in winter nights at a summit near 5400 m, such as Cerro Honar, may be as low as 0.5–0.6 mm, if conditions near the summit approach those of the free atmosphere. The lowest quartile of PWV may approximate 0.35–0.40 mm.
* Median PWV in winter nights at a summit near 5700 m, such as Cerro Chascón may approximate 0.40 mm, and the lowest quartile near 0.20–0.25 mm.
* Low elevation temperature inversions are more likely during nighttime. During those episodes, much of the PWV is trapped below the inversion layer, and mountain peaks are offered a nearly dry atmosphere, possibly endowed with high quality seeing.

The conclusions listed above rely on still relatively sparse data from 108 radiosondes and should be considered as preliminary. However, these results suggest that exceptional possibilities for ground–based IR and submm astronomical observations exist in the Llano de Chajnantor region. The combination of low water vapor content and high quality seeing allow for low atmospheric background in the near and mid–IR. At a telescope on a summit in the vicinity of the Chajnantor plateau, numerous atmospheric windows would appear in the mid IR, up to about 50 $`\mu `$m. In the far IR and submillimeter regime, the 350 $`\mu `$m, 450 $`\mu `$m, 600 $`\mu `$m, 750 $`\mu `$m and 870 $`\mu `$m windows reach exceptional transparency, while two useful windows appear near 200 $`\mu `$m. Broad band observations with a 15 m class telescope at such a site would be close in sensitivity to those made with SIRTF in the mid–IR, offering a superb synergy match for follow–up observations of the SIRTF surveys. The near and mid–IR performance of such a ground–based telescope in high resolution spectroscopic mode would be exceptional and could only be exceeded by that of a space telescope of comparable aperture. The relative ease and cost–effectiveness of operation in the Atacama advises that serious attention be given to the Atacama sites for the installation of the next generation of large infrared telescopes.

Acknowledgements: The support and assistance of Joe Veverka, Yervant Terzian, Bob Richardson, Jennifer Yu and Bryan Isacks of Cornell U.; Martha Haynes of Cornell U. and Associated Universitites, Inc.; Robert L. Brown, Eduardo Hardy and Geraldo Valladares of NRAO; the use of radiosonde data obtained through a collaboration with NRAO (Simon Radford and Bryan Butler), ESO (Angel Otarola) and SAO (Ray Blundell and Scott Paine); access to 225 MHz radiometry data of NRAO (Simon Radford); don Tomás Poblete Alay and the staff of La Casa de Don Tomás are thankfully acknowledged. This study was made possible by a grant of the Provost’s Office of Cornell University and the National Science Foundation grant AST–9910136.


format: text

text:
# Are Quantum-Cascade Lasers really quantum?

\[

## Abstract

The first global simulation of semiconductor-based quantum-cascade lasers is presented; Our fully three-dimensional approach allows to study in a purely microscopic way —without resorting to phenomenological parameters— the current-voltage characteristics of state-of-the-art unipolar nanostructures. Based on the proposed theoretical scheme, we are able to give a definite answer to the long-standing controversial question: is charge transport in quantum-cascade lasers mainly coherent or incoherent? Our analysis clearly shows that a proper inclusion of carrier-phonon as well as carrier-carrier scattering within a semiclassical framework gives excellent agreement with experimental results.

\]

Since the seminal paper of Esaki and Tsu , semiconductor-based nanometric heterostructures have been the subject of an impressive theoretical and experimental activity, due to their high potential impact in both fundamental and applied research . One of the main fields of research focuses on exploiting “band-gap engineering”, namely the splitting of the bulk conduction band into several subbands, to generate and detect electromagnetic radiation in the infrared spectral region, as already envisioned by Kazarinov and Suris .

Unipolar coherent-light sources like quantum-cascade lasers (QCLs) , are usually modelled in terms of three-levels systems . Their theoretical description is thus often grounded on purely macroscopic models: the only relevant physical quantities are the various carrier concentrations within the different energy levels, whose time evolution is dictated by proper sets of coupled rate equations. This simplified three-level picture does not take into account the existence of transverse or “in-plane” degrees of freedom and thus provides no information on the microscopic processes governing carrier dynamics within the three-dimensional (3D) multi-subband structure. As pointed out in , such a macroscopic modeling can only operate as an a posteriori fitting procedure: The best-fit phenomenological parameters obtained through a comparison with experiments do strongly depend on details of the 3D hot-carrier distribution and therefore on the device operation conditions.

QCLs are complex devices, whose core is a multi-quantum-well (MQW) structure made up of repeated stages of active regions sandwiched between electron-injecting and collecting regions. When a proper bias is applied, an electron cascade along the subsequent quantized-level energy staircase takes place. For a detailed understanding of the basic physical processes involved, two main issues need to be addressed: (i) the nature of the hot-carrier relaxation within the device active region; (ii) the nature —coherent versus incoherent— of the physical mechanisms governing charge transport through injector/active-region/collector interfaces.

Point (i) has been recently addressed in , where the macroscopic treatment of hot-carrier relaxation within the device active region previously mentioned has been compared to a fully kinetic description, based on the following set of equations :

$$\frac{d}{dt}f_{𝐤\nu }=\left[g_{𝐤\nu }\mathrm{\Gamma }_{𝐤\nu }f_{𝐤\nu }\right]_{\mathrm{i}/\mathrm{l}}+\underset{𝐤^{}\nu ^{}}{}\left[P_{𝐤\nu ,𝐤^{}\nu ^{}}f_{𝐤^{}\nu ^{}}P_{𝐤^{}\nu ^{},𝐤\nu }f_{𝐤\nu }\right].$$
(1)
Here, the first two terms describe —still on a partially phenomenological level— injection/loss (i/l) of carriers with wavevector $`𝐤`$ in subband $`\nu `$, while the last ones describe intra- as well as inter-subband in- and out-scattering processes ($`𝐤\nu 𝐤^{}\nu ^{}`$) within the device active region only. The Boltzmann-like structure of (1) allows for a partially stochastic solution; to this aim, the commonly used Monte Carlo (MC) method  has been adopted. The simulated experiments in  show that, as far as the active region is concerned, the quantum-cascade process is mainly determined by carrier-optical phonon emission; Moreover, the values of effective interlevel rates strongly depend on the shape of the in-plane charge distributions as well as on Pauli-blocking effects.

However, the microscopic analysis in , being limited to the device active region only, does not allow to answer point (ii): which are the basic physical mechanisms governing charge transport through injector/active-region/collector interfaces? This issue is intimately related to the long-standing controversial question : is charge transport in quantum-cascade lasers mainly coherent or incoherent?

To provide a definite answer to this fundamental question, in this Letter we present the first global simulation —injector plus active region plus collector— of semiconductor-based QCL structures. To this end, the partially phenomenologic model in (1) has to be replaced by a fully microscopic description of the whole MQW core structure. This is not a trivial task. It requires: (i) a proper description of all the 3D electron states within the active as well as injector/collector regions; (ii) a fully microscopic simulation scheme to “close the circuit”.

As a starting point, we evaluate —within a self-consistent Schrödinger-Poisson approach in an envelope-function framework — the set of 3D single-particle electron states $`\{𝐤\nu \}`$ corresponding to a single QCL stage, e.g., active region plus collector/injector in the presence of the applied bias (see Fig. 1). Given such carrier states, we consider the ideal MQW structure obtained as infinite repetition of this QCL periodicity region (see Fig. 1). Within such extended scheme, the time evolution of the carrier distribution function $`f_{𝐤\alpha }`$ is governed by the following Boltzmann-like equation:

$$\frac{d}{dt}f_{𝐤\alpha }=\underset{𝐤^{}\alpha ^{}}{}\underset{s}{}\left[P_{𝐤\alpha ,𝐤^{}\alpha ^{}}^sf_{𝐤^{}\alpha ^{}}P_{𝐤^{}\alpha ^{},𝐤\alpha }^sf_{𝐤\alpha }\right].$$
(2)
Here, $`\alpha (\lambda ,\nu )`$ denotes the generic electron state in our MQW structure, i.e., the $`\nu `$-th state of the $`\lambda `$-th stage, while $`s`$ labels the different scattering mechanisms considered, e.g., carrier-phonon, carrier-carrier, etc.

To “close the circuit”, we impose periodic boundary conditions limiting the inter-stage ($`\lambda ^{}\lambda `$) scattering to just nearest-neighbor coupling ($`\lambda ^{}=\lambda \pm 1`$). In view of the translational symmetry, we are allowed to simulate carrier transport over the central —i.e., $`\lambda =0`$— stage only. Every time a carrier in state $`\nu `$ undergoes an inter-stage scattering process (i.e., $`0,\nu \pm 1,\nu ^{}`$, the electron is properly reinjected into the central region ($`0,\nu 0,\nu ^{}`$) and the corresponding electron charge $`\pm e`$ will contribute to the current through the device . The proposed global-simulation scheme allows to study in a purely microscopic way —without resorting to phenomenological i/l parameters— the current-voltage characteristics of state-of-the-art unipolar nanostructures.

At this point a few comments are in order. Contrary to Eq. (1), the Boltzmann-like equation in (2) —describing an infinite system— does not contain explicit i/l terms, which are automatically accounted for within the proposed re-injection scheme. Moreover, the electric field does not enter our transport equation via the usual drift term ; the corresponding bias is directly taken into account in evaluating the single-particle states $`\{𝐤\nu \}`$ .

It is well-known that the application of the present semiclassical transport theory to MQW structures at high fields becomes questionable. Indeed, according to (2), in the absence of scattering processes ($`P_{𝐤\alpha ,𝐤^{}\alpha ^{}}=0`$) we have no current. This paradoxical result is due to the neglect of phase coherence between our localized states. A proper treatment of coherent transport —including resonant-tunneling effects— would require a density-matrix formalism . However, in the presence of a strong scattering dynamics current is mainly determined by incoherent hopping processes between localized states; moreover, such phase-breaking processes lead to a very fast decay of off-diagonal density-matrix elements. In this regime coherent-transport effects play a minor role and the semiclassical limit is recovered. It is then clear that the nature —coherent versus incoherent— of charge transport in QCLs is primarily dictated by the role played by incoherent scattering processes.

In order to discriminate between the two different regimes, we have applied the above global-simulation scheme to state-of-the-art QCL structures. In particular, as prototypical device, we have considered the diagonal-configuration QCL in , schematically depicted in Fig. 1, in which the proposed simulation scheme is also sketched. Here, the energy levels and probability densities of various electron states within the simulated stage ($`\lambda =0`$) are also plotted: They correspond to the device active region (full line, $`\nu =1,2,3`$ according to the standard notation) as well as to the collector region (dashed line, $`\nu =`$ A,B,C,D,E). The nearest stages $`\lambda =\pm 1`$ are partially shown for clarity as well.

In order to properly model phase-breaking hopping processes, in addition to carrier-optical phonon scattering, all various intra- as well as intersubband carrier-carrier interaction mechanisms have been considered. To this end, the well-established MC method for the simulation of intercarrier scattering in quantum-well systems  has been extended to our MQW biased structure. Within such approach —based on the Born approximation— the effect of nonequilibrium multisubband screening of the bare two-body Coulomb potential is taken into account within the standard random-phase approximation . Other scattering mechanisms, not included in the simulation, are expected to play a very minor role : The interaction with acoustic phonons —which in the absence of carrier-carrier scattering would be essential to get ergodicity— does not affect energy relaxation significantly due to its quasi-elastic nature and small coupling constant; Carrier-impurity scattering is also expected to have very little impact on vertical-transport phenomena.

As a first step, we have investigated the relative weigth of the carrier-carrier and carrier-phonon competing energy-relaxation channels. The time evolution of the carrier population in the various subbands as well as of the total current density in the presence of carrier-phonon scattering are depicted in Fig. 2. Parts (a) and (b) refer, respectively, to simulated experiments without and with two-body carrier-carrier scattering. In our “charge conservation” scheme, we start the simulation assuming the total number of carriers to be equally distributed among the different subbands; then the electron distribution functions evolve according to Eq. (2) and a steady-state condition is eventually reached, leading to the desired $`32`$ population inversion. As we can see, the inclusion of intercarrier scattering has significant effects; It strongly increases inter-subband carrier redistribution, thus reducing the electron accumulation in the lowest energy level A and optimizing the coupling between active region and injector/collector (the populations of subbands 3 and B get equal). This effect comes out to be crucial in determining the electron flux through the MQW structure. Indeed, Fig. 2(c) shows the simulated current density across the core region of our QCL device, obtained with (full line) and without (dashed line) the inclusion of carrier-carrier scattering. These simulations have been performed at the threshold operating parameters estimated in . The current density in the presence of both electron-phonon and electron-electron scattering mechanisms is about 4 kA/cm2: This value is in good agreement  with the experimental results in .

The results plotted in fig. 2 clearly demonstrate that the electron-phonon interaction alone is not able to efficiently couple the injector subbands to the active region ones: While carrier-phonon relaxation well describes the electronic quantum cascade within the bare active region , carrier-carrier scattering plays an essential role in determining charge transport through the full core region. This can be ascribed to two typical features of carrier-carrier interaction —compared to the case of carrier-phonon—: (i) this is a long-range two-body interaction mechanism, which also couples non-overlapping single-particle states \[see Fig. 1\]; (ii) the corresponding scattering process at relatively low carrier density is quasielastic, thus coupling nearly resonant energy levels, like states 3 and B.

This result is extremely important: it tells us that the presence of a coherent-tunneling dynamics —often invoked to account for nearly resonant inter-level processes between active region and injector/collector— is not necessary: The charge transport in QCL structures can be explained quantitatively in terms of incoherent intercarrier-assisted tunneling processes, and thus within a purely semiclassical transport theory. This allows us to answer the long-standing controversial question on the nature —coherent versus incoherent— of charge transport in QCLs previously mentioned: for the typical structures considered, energy-relaxation and dephasing processes are so strong to destroy any phase-coherence effect; as a result, the usual semiclassical description is in excellent agreement with experimental results.

Since in the proposed MC simulation scheme, the current density across the whole structure is an output of the simulation, the current-versus-voltage behavior of our semiconductor quantum device can be obtained within our fully microscopic description, without resorting to any phenomenological external parameter. Figure 3 presents the simulated voltage-current characteristics for the MQW structure of Fig. 1. As we can see, the current density increases with the applied field, due to the increased coupling between subband 3 and various lower-energy injector states, in which carriers tend to relax and accumulate. The agreement between our theoretical results and the measured dependence shown in Ref.  is extremely good. Discrepancies between simulated and measured values are well within the measurement uncertainties and the complexity of the real device structure.

In summary, we have presented the first global simulation of semiconductor-based QCLs. The proposed theoretical scheme allows for a fully microscopic evaluation of the current-voltage characteristics of prototypical QCL structures. The excellent agreement between our semiclassical simulations and recent experimental results confirms the incoherent nature of charge transport in such unipolar quantum devices.

This work has been supported in part by the European Commission through the Brite Euram UNISEL project (Contract No. BE97-4072).


text:
# 1 Introduction

## 1 Introduction

It is known that at asymptotically large momentum transfer certain exclusive hadronic reactions are calculable within the framework of perturbative QCD (pQCD) due to asymptotic freedom. However the applicability of pQCD to exclusive processes remains controversial at the largest momentum transfers probed in the laboratory. The quark-counting scaling laws of Brodsky and Farrar  tend to agree remarkably well with data. This apparently indicates that a finite, minimal number of quarks are being probed. However, the helicity conservation selection rules of Lepage and Brodsky tend not to agree with data . This indicates that the Brodsky-Lepage factorization scheme fails, independent of further details. Furthermore the formalism fails to correctly predict the magnitude of the proton electromagnetic form factor . This suggests that the basic assumption in this formalism, namely that these processes are dominated by short distance, may not be true at laboratory energies. As a consequence, the agreement of the scaling laws with data becomes rather mysterious.

Compared to exclusive processes in free space, it has been shown  that the corresponding processes in a nuclear medium will be theoretically cleaner. Large quark separations will tend not to propagate in the strongly interacting nuclear medium. Configurations of small quark separations, on the other hand, which coincide with the perturbatively calculable region, will propagate with small attenuation. This phenomenon, called nuclear filtering, is the complement of the idea called color transparency . In its original rendition, color transparency  was based on having large momentum transfer $`Q^2`$ select short distance, then free to propagate easily through a passive nuclear probe. Nuclear filtering uses the nuclear medium in an active way toward the same purpose, and may be more efffective.

## 2 Exclusive Processes in Free Space

Let us briefly review the framework for calculation of exclusive hadronic processes within pQCD. We take the pion electromagnetic form factor as an example. The short-distance formalism crucially depends on the assumption that the process can be factorized in a perturbatively calculable hard scattering piece and the soft distribution amplitude. Given this assumption, the pion electromagnetic form factor  at momentum transfer $`q^2=Q^2`$ can be written as

$$F_\pi (Q^2)=𝑑x_1𝑑x_2\varphi (x_2,Q)H(x_1,x_2,Q)\varphi (x_1,Q),$$
(1)
where $`\varphi (x,Q)`$ are the distribution amplitudes which can be expressed in terms of the pion wave function $`\psi (x,\stackrel{}{k}_T)`$ as

$$\varphi (x,Q)=^Qd^2k_T\psi (x,\stackrel{}{k}_T).$$
(2)
Here $`x`$ is the longitudinal momentum fraction and $`\stackrel{}{k}_T`$ the transverse momentum carried by the quark; $`\psi `$ is the light-cone Bethe-Salpeter amplitude. The factorization is a good approximation provided the external photon momentum $`Q^2`$ is much larger than all other physical scales. The hard scattering is then evaluated with on-shell quarks carrying neglible $`k_T`$.

The formalism predicts that the cross section for exclusive processes $`d\sigma /dt`$, where $`t`$ is the momentum transfer squared, scales like $`1/t^{n2}`$ up to logs, where $`n`$ is the total number of elementary partons participating in the process. The underlying reason for the power law is scale invariance of the fundamental theory. Further logarithmic dependence is given by QCD scaling violations. In making these assertions one asumes that $`t`$ is asymptotically large. The dominant contribution to this scattering arises from the valence quark, since every additional parton leads to an additional suppression factor of $`1/t`$. Physically the scattering probes the short distance part of the hadron wave function. Dominance by the short distance wave functions leads to several predictions such as helicity conservation, color transparency  etc.

The successes and failures of this scheme are well known. Calculation of electromagnetic form factors using this factorization scheme has been criticised by several authors . The basic problem is that the momentum scales of the exchanged gluons tend to become rather small, and the applicability of pQCD becomes doubtful. The normalization of form factors is largely unknown; use of asymptotic distribution amplitudes tends to give small normalizations compared to data. Form factor magnitudes can be enhanced by use of model distribution amplitudes  which peak closer to the end-points, namely $`x0,1`$, which then exacerbates the problem of small internal momentum transfers.

To investigate this problem, there exists an alternate factorization which does not neglect the $`k_T`$ dependence of the hard scattering. The method, which we call “impact parameter factorization”, was first used by Botts and Sterman  to deal with the Landshoff pinch regions of proton-proton scattering. These regions lie outside the assumptions of the quark-counting formalism, and cannot be described by its factorization scheme. Impact parameter factorization was first applied to electron-beam ($`\gamma ^{}`$ initiated) experiments in Ref. , in order to accomodate color transparency and nuclear filtering. Li and Sterman  developed the method for free-space $`\gamma ^{}`$ initiated experiments such as the proton’s electromagnetic form factor. A consistent feature of this formalism involves attention to the transverse spatial coordinates of quarks and attendant Sudakov effects.

For the case of pion form factor  the starting point is,

$$F_\pi (Q^2)=𝑑x_1𝑑x_2𝑑\stackrel{}{k}_{T1}𝑑\stackrel{}{k}_{T2}\psi ^{}(x_2,\stackrel{}{k}_{T2},P_2)H(x_1,x_2,Q^2,\stackrel{}{k}_{T1},\stackrel{}{k}_{T2})\psi (x_1,\stackrel{}{k}_{T1},P_1),$$
(3)
where it is again assumed that the process is factorizable into hard scattering and soft hadronic wave functions $`\psi (x,\stackrel{}{k}_T,P)`$. The calculation is simplified by dropping the $`k_T`$ dependence in the quark propagators in hard scattering kernel $`H`$, in which case only the combination $`\stackrel{}{k}_{T1}+\stackrel{}{k}_{T2}`$ appears in the calculation. The authors  work in configuration space where this can be written as

$$F_\pi (Q^2)=𝑑x_1𝑑x_2\frac{d^2\stackrel{}{b}}{(2\pi )^2}𝒫(x_2,b,P_2,\mu )\stackrel{~}{H}(x_1,x_2,Q^2,\stackrel{}{b},\mu )𝒫(x_1,b,P_1,\mu ),$$
(4)
where $`𝒫(x,b,P,\mu )`$ and $`\stackrel{~}{H}(x_1,x_2,Q^2,\stackrel{}{b},\mu )`$ are the Fourier transforms of the wave function and hard scattering respectively; $`\stackrel{}{b}`$ is conjugate to $`\stackrel{}{k}_{T1}+\stackrel{}{k}_{T2}`$, $`\mu `$ is the renormalization scale and $`P_1`$, $`P_2`$ are the initial and final momenta of the pion.

Sudakov form factors are obtained by summing the leading and next to leading logarithms using renormalization group (RG) techniques. The wave function at small $`b`$ is often approximated by the distribution amplitude $`\varphi (x,1/b)`$.

The resulting form factor using asymptotic as well as CZ  distribution amplitudes is shown in fig. 1. A remarkable fact is that the correct asymptotic $`Q^2`$ behavior is seen beyond the scale of about $`Q=1`$ GeV, irrespective of the choice of wave function. In contrast to the Brodsky-Lepage factorization, the $`k_T`$ dependence of the hard scattering is not neglected, and hence this $`Q^2`$ dependence does not follow trivially. Instead, the $`Q^2`$ dependence is a detailed dynamical prediction of the theory, and depends on the relative size of intrinsic $`k_T^2`$ and $`x_1x_2Q^2`$. The $`Q^2`$ dependence of the prediction is comparatively robust, since it is weakly dependent on the details of the distribution amplitude.

We note that the normalization of the theoretical result falls below the experimental data for both choices of distribution amplitude. It is good to keep in mind that the distribution amplitudes are not known with much exactitude, so perhaps the models might be improved. Indeed the theoretical normalization of the form factor is comparatively murky, because it strongly depends on such model-dependent details. Moreover, the large difference between theory and experiment at high momenta should be interpreted with caution, since there may be large systematic errors in the experimental extraction of the form factor which are not shown in the figure. Further theoretical issues in this extraction have also been raised in Ref. .

Furthermore, the leading order pQCD amplitudes calculated may not give a very reliable estimate of the normalization. Li and Sterman argue that roughly 50% of the contribution can be regarded as perturbative, since it is obtained from the region where $`\alpha _s/\pi <0.7`$. It may be that higher order contributions in $`\alpha _s`$ are not negligible, and the leading order predictions for the normalization of the form factor cannot be regarded as accurate.

We are left with the following interesting situation: Although the basic Brodsky-Lepage factorization survives at asymptotic $`Q^2`$, the method is sensitive to end-point singularities, and one may need to go to higher orders in $`\alpha _s`$ in order to obtain an accurate prediction for the form factor normalizations. Meanwhile the predicted $`Q^2`$ dependence of the impact parameter factorization appears to be quite robust, and less strongly dependent on the theoretical uncertainties such as the choice of distribution amplitude.

### 2.1 The Proton Electromagnetic Form Factor

The improved impact parameter factorization has also been applied to the proton Dirac form factor $`F_1^p(Q^2)`$. The calculation is considerably more complicated compared to the pion due to the presence of three valence quarks. Here also it is necessary to use distribution amplitudes which peak close to the end points if one wishes to fit the experimental normalization of the form factor. The results  of the calculation using KS  and CZ  distribution amplitudes and $`c=1`$ and 1.14 are shown in fig. 2, where $`c`$ is a parameter which determines the long and short distance factorization scale.

The natural agreement of $`Q^2`$ dependence of the pQCD calculations should be contrasted to data fits obtained using soft overlap models . In such models the $`Q^2`$ dependence depends on the details of the model wave function. Soft overlap model predictions at high momentum have a tendency to fall more strongly than experimental data.

### 2.2 Hadron-Hadron Exclusive Processes

To a rough approximation, the empirically fit power-laws for hadron-hadron exclusive processes tend to agree well with the quark-counting scaling law. However when data is examined in detail, one finds contradictions, such as the violation of helicity conservation selection rules and oscillations  around the overall power law momentum dependence of $`d\sigma /dt`$. It is not commonly appreciated that these signals of processes beyond the quark-counting model are observed for almost every process tested so far.

### 2.3 Free Space Data: $`\pi `$-P to $`\pi `$-P at Fixed Angle

In Fig. 3a we show results $`d\sigma /dt`$ for $`\pi p\pi ^{}p^{}`$ scattering at $`90^o`$. An overall momentum dependence $`|t|^8`$ has been factored out of the cross section . The free space cross section is fit in terms of interfering short and long distance amplitudes using a three component (solid curve) and two component model (dashed curve). One amplitude is obtained from the independent scattering diagrams , where the quarks scatter at large transverse separations. Leading logarithmic summation calculable in pQCD and related by analyticity to Sudakov effects generates the oscillations . In the three component model we also include a subleading contribution: details are given in Ref. . We note that the oscillations around the overall power dependence are not a small effect. The model incorporating the independent scattering diagrams fits the free space data very well with $`\chi ^2/\mathrm{degree}\mathrm{of}\mathrm{freedom}=1.97`$. The Brodsky-Lepage short distance model gives $`\chi ^2/\mathrm{degree}\mathrm{of}\mathrm{freedom}=99`$ and is objectively ruled out.

### 2.4 Free Space Data: $`\gamma `$-P to $`\pi `$-N at Fixed Angle

The experimental data  for $`\gamma p\pi ^+n`$ also shows fluctuations around the overall power behavior as seen in Fig. 4a. This is quite interesting, since in this case the Landshoff pinch amplitudes have been theoretically shown to be subleading at large $`Q^2`$ . However at medium $`Q^2`$ even the subleading amplitudes can give significant contributions. We again model the fluctuations in terms of interfering soft and hard contributions. The free space fit is shown in Fig. 4a.

## 3 Nuclear Targets and Color Transparency

Color transparency is a natural prediction of pQCD. If exclusive processes at large momentum transfer are actually dominated by short distance, then interaction of hard-struck hadrons with other hadrons is predicted to be small. The hadron interaction cross section $`\sigma b^2`$, where $`b`$ is the transverse separation between quarks in a color singlet state, vanishes as $`b1/Q0`$. Experimentally color transparency is measured by observing quasi exclusive reactions on nuclear targets such as $`eAe^{}p(A1)`$ and comparing it with the corresponding free space process $`epe^{}p^{}`$. The experimental results are reported in terms of transparency ratio $`T`$, defined as

$$T=\frac{d\sigma _{\mathrm{nuclear}}}{Ad\sigma _{\mathrm{free}\mathrm{space}}}.$$

At asymptotically large $`Q^2`$, one has asymptotically short distance, and color transparency ratios are unity. This is actually as much as one can say with quark-counting factorization. To say more one must incorporate information about the transverse, or impact parameter separation, hence the use of “impact parameter factorization”.

There are two ways, then, to get to short distance: asymptotically large $`Q^2`$, which we have seen above is not a realistic feature of the laboratory, or by nuclear filtering. The nucleus essentially acts as a transverse-separation filter  which preferentially attentuates the soft amplitudes. It follows that the nucleus is a cleaner medium to study exclusive processes than free space: which is quite a surprising prediction.

Several experiments indicate that color transparency  and nuclear filtering  have been observed at large nuclear number $`A`$. The first color transparency experiment of Carroll et al  convincingly showed that interference effects in proton-proton scattering were filtered away in nuclear targets. Attenuation of the Landshoff-induced oscillations, seen in free-space , in the nucleus reproduces the oscillating transparency . In contrast, other models based on a classically expanding cross section  do not fit the data. Extraction of the attenuation cross section in nuclear targets show values significantly below the Glauber theory values . The FNAL E-665 experiment  also proved consistent with filtering effects .

Electron beam experiments remain controversial, with few signals of interesting $`Q^2`$ dependence . A basic feature of $`\gamma ^{}`$-initiated reactions is that most events are knocked out from the back side of the nucleus. The resolving power of such experiments to measure the size of propagating states is rather modest, unless one has very high experimetal precision. The $`A`$ dependence is a particularly useful tool  to measure effective attenuation cross sections. O’Neill et al  showed that effective attenuation cross sections extracted from $`A(e,e^{}p)`$ SLAC data were smaller than Glauber theory calculations by a statistically significant amount. However, the precision of the data  was insufficient to establish a large effect, and model dependence in the choice of the normalization of hard scattering is another complication. Reports on new $`(e,e^{}p)`$ beam experiments from CEBAF are expected shortly.

### 3.1 New Color Transparency Calculations

We have recently calculated color transparency ratio for several processes which include $`\pi A\pi ^{}p(A1)`$, $`\gamma A\pi N(A1)`$, $`eAe^{}p(A1)`$ and $`eAe^{}\pi A`$. For the case of $`eAe^{}p(A1)`$ and $`eAe^{}\pi A`$ we did detailed impact-parameter calculations using leading order pQCD including the Sudakov effects. The more complicated processes $`\pi A\pi ^{}p(A1)`$, $`\gamma A\pi N(A1)`$ were modelled in terms of interfering long and short distance amplitudes following the earlier work on proton proton quasi elastic scattering in nuclear medium . The magnitude of short and long distance amplitudes were obtained phenomenologically by making a fit to the corresponding free space process.

In Fig. 5 we show the $`Q^2`$ dependence of the transparency ratio for electroproduction of pions using the CZ and asymptotic distribution amplitudes . The scale of $`Q^2`$ ranging up to 5 GeV2 may benefit from explanation. At the exclusive production point, the relativistic boost factor of a pion is given by $`\gamma =Q^2/(2m_\pi ^2)25(Q^2/\mathrm{GeV}^2)`$. Since even a 1 GeV pion is highly relativistic, we may suppose that the perturbative calculations may well apply in the comparatively small $`Q^2`$ regime. These calculations show a rather striking rise with $`Q^2`$ of the transparency ratio, which should be easily observable experimentally. The fact of a rise does not depend much on the distribution amplitude, but the slope of the rise does: we discuss the reasons shortly when we review the proton. For these calculations we used $`\mathrm{\Lambda }_{QCD}=200`$ MeV. We adjusted the value of the parameter $`k`$, defined by $`\sigma _{\mathrm{attenuation}}=kb^2`$, so that the predicted results for proton (discussed later) are in agreement with the SLAC data . This selects the value of $`k`$ to be approximately equal to 10.

Results for the $`Q^2`$ dependence of the proton transparency ratio  from various models are shown in Figs. (6, 7). A standard model for the distribution amplitude, the KS model, was used to generate Fig. 6. One sees that the calculation with the KS model has a rather flat $`Q^2`$ dependence. At first this result was surprising, assuming short-distance dominance and the expectation of a rapidly increasing function of $`Q^2`$, but in retrospect the result appears quite natural. As in the earlier discussion of the pion, these results depend on the distribution amplitude model, which can be categorized into two types. The KS model is an end-point dominated distribution amplitude, which is known to produce its dominant contributions from long-distance components of the quark wave functions. For this reason use of the KS wave function in the free-space form factor has led to many questions of theoretical consistency, mentioned earlier. Precisely the same lack of a dominant short-distance contribution is responsible for the calculated flat dependence on $`Q^2`$. Turning to Fig. 7, which compares the CZ and KS models, both of which are end-point dominated, one sees nearly identical flat $`Q^2`$ behavior. This indicates that the details of the model do not matter so long as they are end-point models. The figure also shows the dependence on the factorization scale parameter $`c`$. Rather interestingly, a substantial dependence on $`c`$ of form factors in free space drops out in the transparency ratio.

If the transparency ratio is flat with $`Q^2`$, does it mean that color transparency has not been observed? No, because important contributions to the free-space denominator are being filtered away by a large nucleus, depleting the numerator, making the interpretation of the $`Q^2`$ dependence ambiguous. Fortunately experiments also have the $`A`$ dependence, which can be expressed as an empirically defined effective attentuation cross section $`\sigma _{eff}`$. In the standard Glauber theory of $`pA`$ scattering this quantity is flat with energy in the region of interest and has a value of about 40 mb. Observation of $`\sigma _{eff}`$ substantially below the Glauber value would be a signal of color transparency. We evaluated our predictions as if they were experimental data, extracted $`\sigma _{eff}`$, and found the value to be about 31 mb at $`Q^2=4`$ GeV2 and slowly decreasing with energy as shown in fig. 8. This value is substantially below the Glauber value: so color transparency can be observed in $`e,e^{}p`$ measurements of sufficiently high precision.

Finally in fig. 3b,c we show results for the transparency ratio for the process $`\pi A\pi ^{}p(A1)`$. Rather interesting oscillations are predicted for the transparency ratio. These predictions can be tested in future experiments. The results for the transparency ratio for $`\gamma A\pi ^+n(A1)`$ are shown in Fig 4b,c,d. We again predict fluctuations in the transparency ratio which can be tested in the near future at CEBAF .

Acknowledgements: Work was partially supported by DOE grant number 85ER40214.


text:
# A Decision Tree of Bigrams is an Accurate Predictor of Word Sense

## 1 Introduction

Word sense disambiguation is the process of selecting the most appropriate meaning for a word, based on the context in which it occurs. For our purposes it is assumed that the set of possible meanings, i.e., the sense inventory, has already been determined. For example, suppose bill has the following set of possible meanings: a piece of currency, pending legislation, or a bird jaw. When used in the context of The Senate bill is under consideration, a human reader immediately understands that bill is being used in the legislative sense. However, a computer program attempting to perform the same task faces a difficult problem since it does not have the benefit of innate common–sense or linguistic knowledge.

Rather than attempting to provide computer programs with real–world knowledge comparable to that of humans, natural language processing has turned to corpus–based methods. These approaches use techniques from statistics and machine learning to induce models of language usage from large samples of text. These models are trained to perform particular tasks, usually via supervised learning. This paper describes an approach where a decision tree is learned from some number of sentences where each instance of an ambiguous word has been manually annotated with a sense–tag that denotes the most appropriate sense for that context.

Prior to learning, the sense–tagged corpus must be converted into a more regular form suitable for automatic processing. Each sense–tagged occurrence of an ambiguous word is converted into a feature vector, where each feature represents some property of the surrounding text that is considered to be relevant to the disambiguation process. Given the flexibility and complexity of human language, there is potentially an infinite set of features that could be utilized. However, in corpus–based approaches features usually consist of information that can be readily identified in the text, without relying on extensive external knowledge sources. These typically include the part–of–speech of surrounding words, the presence of certain key words within some window of context, and various syntactic properties of the sentence and the ambiguous word.

The approach in this paper relies upon a feature set made up of bigrams, two word sequences that occur in a text. The context in which an ambiguous word occurs is represented by some number of binary features that indicate whether or not a particular bigram has occurred within approximately 50 words to the left or right of the word being disambiguated.

We take this approach since surface lexical features like bigrams, collocations, and co–occurrences often contribute a great deal to disambiguation accuracy. It is not clear how much disambiguation accuracy is improved through the use of features that are identified by more complex pre–processing such as part–of–speech tagging, parsing, or anaphora resolution. One of our objectives is to establish a clear upper bounds on the accuracy of disambiguation using feature sets that do not impose substantial pre–processing requirements.

This paper continues with a discussion of our methods for identifying the bigrams that should be included in the feature set for learning. Then the decision tree learning algorithm is described, as are some benchmark learning algorithms that are included for purposes of comparison. The experimental data is discussed, and then the empirical results are presented. We close with an analysis of our findings and a discussion of related work.

## 2 Building a Feature Set of Bigrams

We have developed an approach to word sense disambiguation that represents text entirely in terms of the occurrence of bigrams, which we define to be two consecutive words that occur in a text. The distributional characteristics of bigrams are fairly consistent across corpora; a majority of them only occur one time. Given the sparse and skewed nature of this data, the statistical methods used to select interesting bigrams must be carefully chosen. We explore two alternatives, the power divergence family of goodness of fit statistics and the Dice Coefficient, an information theoretic measure related to pointwise Mutual Information.

Figure 1 summarizes the notation for word and bigram counts used in this paper by way of a $`2\times 2`$ contingency table. The value of $`n_{11}`$ shows how many times the bigram big cat occurs in the corpus. The value of $`n_{12}`$ shows how often bigrams occur where big is the first word and cat is not the second. The counts in $`n_{+1}`$ and $`n_{1+}`$ indicate how often words big and cat occur as the first and second words of any bigram in the corpus. The total number of bigrams in the corpus is represented by $`n_{++}`$.

### 2.1 The Power Divergence Family

\[Cressie and Read, 1984\] introduce the power divergence family of goodness of fit statistics. A number of well known statistics belong to this family, including the likelihood ratio statistic $`G^2`$ and Pearson’s $`X^2`$ statistic.

These measure the divergence of the observed ($`n_{ij}`$) and expected ($`m_{ij}`$) bigram counts, where $`m_{ij}`$ is estimated based on the assumption that the component words in the bigram occur together strictly by chance:

$`m_{ij}={\displaystyle \frac{n_{i+}n_{+j}}{n_{++}}}`$
Given this value, $`G^2`$ and $`X^2`$ are calculated as:

$`G^2=2{\displaystyle \underset{i,j}{}}n_{ij}\mathrm{log}{\displaystyle \frac{n_{ij}}{m_{ij}}}`$
$`X^2={\displaystyle \underset{i,j}{}}{\displaystyle \frac{(n_{ij}m_{ij})^2}{m_{ij}}}`$
\[Dunning, 1993\] argues in favor of $`G^2`$ over $`X^2`$, especially when dealing with very sparse and skewed data distributions. However, \[Cressie and Read, 1984\] suggest that there are cases where Pearson’s statistic is more reliable than the likelihood ratio and that one test should not always be preferred over the other. In light of this, \[Pedersen, 1996\] presents Fisher’s exact test as an alternative since it does not rely on the distributional assumptions that underly both Pearson’s test and the likelihood ratio.

Unfortunately it is usually not clear which test is most appropriate for a particular sample of data. We take the following approach, based on the observation that all tests should assign approximately the same measure of statistical significance when the bigram counts in the contingency table do not violate any of the distributional assumptions that underly the goodness of fit statistics. We perform tests using $`X^2`$, $`G^2`$, and Fisher’s exact test for each bigram. If the resulting measures of statistical significance differ, then the distribution of the bigram counts is causing at least one of the tests to become unreliable. When this occurs we rely upon the value from Fisher’s exact test since it makes fewer assumptions about the underlying distribution of data.

For the experiments in this paper, we identified the top 100 ranked bigrams that occur more than 5 times in the training corpus associated with a word. There were no cases where rankings produced by $`G^2`$, $`X^2`$, and Fisher’s exact test disagreed, which is not altogether surprising given that low frequency bigrams were excluded. Since all of these statistics produced the same rankings, hereafter we make no distinction among them and simply refer to them generically as the power divergence statistic.

### 2.2 Dice Coefficient

The Dice Coefficient is a descriptive statistic that provides a measure of association among two words in a corpus. It is similar to pointwise Mutual Information, a widely used measure that was first introduced for identifying lexical relationships in \[Church and Hanks, 1990\]. Pointwise Mutual Information can be defined as follows:

$`MI(w_1,w_2)=log_2{\displaystyle \frac{n_{11}n_{++}}{n_{+1}n_{1+}}}`$
where $`w_1`$ and $`w_2`$ represent the two words that make up the bigram.

Pointwise Mutual Information quantifies how often two words occur together in a bigram (the numerator) relative to how often they occur overall in the corpus (the denominator). However, there is a curious limitation to pointwise Mutual Information. A bigram $`w_1w_2`$ that occurs $`n_{11}`$ times in the corpus, and whose component words $`w_1`$ and $`w_2`$ only occur as a part of that bigram, will result in increasingly strong measures of association as the value of $`n_{11}`$ decreases. Thus, the maximum pointwise Mutual Information in a given corpus will be assigned to bigrams that occur one time, and whose component words never occur outside that bigram. These are usually not the bigrams that prove most useful for disambiguation, yet they will dominate a ranked list as determined by pointwise Mutual Information.

The Dice Coefficient overcomes this limitation, and can be defined as follows:

$`Dice(w_1,w_2)={\displaystyle \frac{2n_{11}}{n_{+1}+n_{1+}}}`$
When $`n_{11}=n_{1+}=n_{+1}`$ the value of $`Dice(w_1,w_2)`$ will be 1 for all values $`n_{11}`$. When the value of $`n_{11}`$ is less than either of the marginal totals (the more typical case) the rankings produced by the Dice Coefficient are similar to those of Mutual Information. The relationship between pointwise Mutual Information and the Dice Coefficient is also discussed in \[Smadja et al., 1996\].

We have developed the Bigram Statistics Package to produce ranked lists of bigrams using a range of tests. This software is written in Perl and is freely available from www.d.umn.edu/~tpederse.

## 3 Learning Decision Trees

Decision trees are among the most widely used machine learning algorithms. They perform a general to specific search of a feature space, adding the most informative features to a tree structure as the search proceeds. The objective is to select a minimal set of features that efficiently partitions the feature space into classes of observations and assemble them into a tree. In our case, the observations are manually sense–tagged examples of an ambiguous word in context and the partitions correspond to the different possible senses.

Each feature selected during the search process is represented by a node in the learned decision tree. Each node represents a choice point between a number of different possible values for a feature. Learning continues until all the training examples are accounted for by the decision tree. In general, such a tree will be overly specific to the training data and not generalize well to new examples. Therefore learning is followed by a pruning step where some nodes are eliminated or reorganized to produce a tree that can generalize to new circumstances.

Test instances are disambiguated by finding a path through the learned decision tree from the root to a leaf node that corresponds with the observed features. An instance of an ambiguous word is disambiguated by passing it through a series of tests, where each test asks if a particular bigram occurs in the available window of context.

We also include three benchmark learning algorithms in this study: the majority classifier, the decision stump, and the Naive Bayesian classifier.

The majority classifier assigns the most common sense in the training data to every instance in the test data. A decision stump is a one node decision tree\[Holte, 1993\] that is created by stopping the decision tree learner after the single most informative feature is added to the tree.

The Naive Bayesian classifier \[Duda and Hart, 1973\] is based on certain blanket assumptions about the interactions among features in a corpus. There is no search of the feature space performed to build a representative model as is the case with decision trees. Instead, all features are included in the classifier and assumed to be relevant to the task at hand. There is a further assumption that each feature is conditionally independent of all other features, given the sense of the ambiguous word. It is most often used with a bag of words feature set, where every word in the training sample is represented by a binary feature that indicates whether or not it occurs in the window of context surrounding the ambiguous word.

We use the Weka \[Witten and Frank, 2000\] implementations of the C4.5 decision tree learner (known as J48), the decision stump, and the Naive Bayesian classifier. Weka is written in Java and is freely available from www.cs.waikato.ac.nz/~ml.

## 4 Experimental Data

Our empirical study utilizes the training and test data from the 1998 SENSEVAL evaluation of word sense disambiguation systems. Ten teams participated in the supervised learning portion of this event. Additional details about the exercise, including the data and results referred to in this paper, can be found at the SENSEVAL web site (www.itri.bton.ac.uk/events/senseval/) and in \[Kilgarriff and Palmer, 2000\].

We included all 36 tasks from SENSEVAL for which training and test data were provided. Each task requires that the occurrences of a particular word in the test data be disambiguated based on a model learned from the sense–tagged instances in the training data. Some words were used in multiple tasks as different parts of speech. For example, there were two tasks associated with bet, one for its use as a noun and the other as a verb. Thus, there are 36 tasks involving the disambiguation of 29 different words.

The words and part of speech associated with each task are shown in Table 1 in column 1. Note that the parts of speech are encoded as n for noun, a for adjective, v for verb, and p for words where the part of speech was not provided. The number of test and training instances for each task are shown in columns 2 and 4. Each instance consists of the sentence in which the ambiguous word occurs as well as one or two surrounding sentences. In general the total context available for each ambiguous word is less than 100 surrounding words. The number of distinct senses in the test data for each task is shown in column 3.

## 5 Experimental Method

The following process is repeated for each task. Capitalization and punctuation are removed from the training and test data. Two feature sets are selected from the training data based on the top 100 ranked bigrams according to the power divergence statistic and the Dice Coefficient. The bigram must have occurred 5 or more times to be included as a feature. This step filters out a large number of possible bigrams and allows the decision tree learner to focus on a small number of candidate bigrams that are likely to be helpful in the disambiguation process.

The training and test data are converted to feature vectors where each feature represents the occurrence of one of the bigrams that belong in the feature set. This representation of the training data is the actual input to the learning algorithms. Decision tree and decision stump learning is performed twice, once using the feature set determined by the power divergence statistic and again using the feature set identified by the Dice Coefficient. The majority classifier simply determines the most frequent sense in the training data and assigns that to all instances in the test data. The Naive Bayesian classifier is based on a feature set where every word that occurs 5 or more times in the training data is included as a feature.

All of these learned models are used to disambiguate the test data. The test data is kept separate until this stage. We employ a fine grained scoring method, where a word is counted as correctly disambiguated only when the assigned sense tag exactly matches the true sense tag. No partial credit is assigned for near misses.

## 6 Experimental Results

The accuracy attained by each of the learning algorithms is shown in Table 1. Column 5 reports the accuracy of the majority classifier, columns 6 and 7 show the best and average accuracy reported by the 10 participating SENSEVAL teams. The evaluation at SENSEVAL was based on precision and recall, so we converted those scores to accuracy by taking their product. However, the best precision and recall may have come from different teams, so the best accuracy shown in column 6 may actually be higher than that of any single participating SENSEVAL system. The average accuracy in column 7 is the product of the average precision and recall reported for the participating SENSEVAL teams. Column 8 shows the accuracy of the decision tree using the J48 learning algorithm and the features identified by a power divergence statistic. Column 10 shows the accuracy of the decision tree when the Dice Coefficient selects the features. Columns 9 and 11 show the accuracy of the decision stump based on the power divergence statistic and the Dice Coefficient respectively. Finally, column 13 shows the accuracy of the Naive Bayesian classifier based on a bag of words feature set.

The most accurate method is the decision tree based on a feature set determined by the power divergence statistic. The last line of Table 1 shows the win-tie-loss score of the decision tree/power divergence method relative to every other method. A win shows it was more accurate than the method in the column, a loss means it was less accurate, and a tie means it was equally accurate. The decision tree/power divergence method was more accurate than the best reported SENSEVAL results for 19 of the 36 tasks, and more accurate for 30 of the 36 tasks when compared to the average reported accuracy. The decision stumps also fared well, proving to be more accurate than the best SENSEVAL results for 14 of the 36 tasks.

In general the feature sets selected by the power divergence statistic result in more accurate decision trees than those selected by the Dice Coefficient. The power divergence tests prove to be more reliable since they account for all possible events surrounding two words $`w_1`$ and $`w_2`$; when they occur as bigram $`w_1w_2`$, when $`w_1`$ or $`w_2`$ occurs in a bigram without the other, and when a bigram consists of neither. The Dice Coefficient is based strictly on the event where $`w_1`$ and $`w_2`$ occur together in a bigram.

There are 6 tasks where the decision tree / power divergence approach is less accurate than the SENSEVAL average; promise-n, scrap-n, shirt-n, amaze-v, bitter-p, and sanction-p. The most dramatic difference occurred with amaze-v, where the SENSEVAL average was 92.4% and the decision tree accuracy was 58.6%. However, this was an unusual task where every instance in the test data belonged to a single sense that was a minority sense in the training data.

## 7 Analysis of Experimental Results

The characteristics of the decision trees and decision stumps learned for each word are shown in Table 2. Column 1 shows the word and part of speech. Columns 2, 3, and 4 are based on the feature set selected by the power divergence statistic while columns 5, 6, and 7 are based on the Dice Coefficient. Columns 2 and 5 show the node selected to serve as the decision stump. Columns 3 and 6 show the number of leaf nodes in the learned decision tree relative to the number of total nodes. Columns 4 and 7 show the number of bigram features selected to represent the training data.

This table shows that there is little difference in the decision stump nodes selected from feature sets determined by the power divergence statistics versus the Dice Coefficient. This is to be expected since the top ranked bigrams for each measure are consistent, and the decision stump node is generally chosen from among those.

However, there are differences between the feature sets selected by the power divergence statistics and the Dice Coefficient. These are reflected in the different sized trees that are learned based on these feature sets. The number of leaf nodes and the total number of nodes for each learned tree is shown in columns 3 and 6. The number of internal nodes is simply the difference between the total nodes and the leaf nodes. Each leaf node represents the end of a path through the decision tree that makes a sense distinction. Since a bigram feature can only appear once in the decision tree, the number of internal nodes represents the number of bigram features selected by the decision tree learner.

One of our original hypotheses was that accurate decision trees of bigrams will include a relatively small number of features. This was motivated by the success of decision stumps in performing disambiguation based on a single bigram feature. In these experiments, there were no decision trees that used all of the bigram features identified by the filtering step, and for many words the decision tree learner went on to eliminate most of the candidate features. This can be seen by comparing the number of internal nodes with the number of candidate features as shown in columns 4 or 7.<sup>1</sup><sup>1</sup>1For most words the 100 top ranked bigrams form the set of candidate features presented to the decision tree learner. If there are ties in the top 100 rankings then there may be more than 100 features, and if the there were fewer than 100 bigrams that occurred more than 5 times then all such bigrams are included in the feature set.

It is also noteworthy that the bigrams ultimately selected by the decision tree learner for inclusion in the tree do not always include those bigrams ranked most highly by the power divergence statistic or the Dice Coefficient. This is to be expected, since the selection of the bigrams from raw text is only measuring the association between two words, while the decision tree seeks bigrams that partition instances of the ambiguous word into into distinct senses. In particular, the decision tree learner makes decisions as to what bigram to include as nodes in the tree using the gain ratio, a measure based on the overall Mutual Information between the bigram and a particular word sense.

Finally, note that the smallest decision trees are functionally equivalent to our benchmark methods. A decision tree with 1 leaf node and no internal nodes (1/1) acts as a majority classifier. A decision tree with 2 leaf nodes and 1 internal node (2/3) has the structure of a decision stump.

## 8 Discussion

One of our long-term objectives is to identify a core set of features that will be useful for disambiguating a wide class of words using both supervised and unsupervised methodologies.

We have presented an ensemble approach to word sense disambiguation \[Pedersen, 2000\] where multiple Naive Bayesian classifiers, each based on co–occurrence features from varying sized windows of context, is shown to perform well on the widely studied nouns interest and line. While the accuracy of this approach was as good as any previously published results, the learned models were complex and difficult to interpret, in effect acting as very accurate black boxes.

Our experience has been that variations in learning algorithms are far less significant contributors to disambiguation accuracy than are variations in the feature set. In other words, an informative feature set will result in accurate disambiguation when used with a wide range of learning algorithms, but there is no learning algorithm that can perform well given an uninformative or misleading set of features. Therefore, our focus is on developing and discovering feature sets that make distinctions among word senses. Our learning algorithms must not only produce accurate models, but they should also shed new light on the relationships among features and allow us to continue refining and understanding our feature sets.

We believe that decision trees meet these criteria. A wide range of implementations are available, and they are known to be robust and accurate across a range of domains. Most important, their structure is easy to interpret and may provide insights into the relationships that exist among features and more general rules of disambiguation.

## 9 Related Work

Bigrams have been used as features for word sense disambiguation, particularly in the form of collocations where the ambiguous word is one component of the bigram (e.g., \[Bruce and Wiebe, 1994\], \[Ng and Lee, 1996\], \[Yarowsky, 1995\]). While some of the bigrams we identify are collocations that include the word being disambiguated, there is no requirement that this be the case.

Decision trees have been used in supervised learning approaches to word sense disambiguation, and have fared well in a number of comparative studies (e.g., \[Mooney, 1996\], \[Pedersen and Bruce, 1997\]). In the former they were used with the bag of word feature sets and in the latter they were used with a mixed feature set that included the part-of-speech of neighboring words, three collocations, and the morphology of the ambiguous word. We believe that the approach in this paper is the first time that decision trees based strictly on bigram features have been employed.

The decision list is a closely related approach that has also been applied to word sense disambiguation (e.g., \[Yarowsky, 1994\], \[Wilks and Stevenson, 1998\], \[Yarowsky, 2000\]). Rather than building and traversing a tree to perform disambiguation, a list is employed. In the general case a decision list may suffer from less fragmentation during learning than decision trees; as a practical matter this means that the decision list is less likely to be over–trained. However, we believe that fragmentation also reflects on the feature set used for learning. Ours consists of at most approximately 100 binary features. This results in a relatively small feature space that is not as likely to suffer from fragmentation as are larger spaces.

## 10 Future Work

There are a number of immediate extensions to this work. The first is to ease the requirement that bigrams be made up of two consecutive words. Rather, we will search for bigrams where the component words may be separated by other words in the text. The second is to eliminate the filtering step by which candidate bigrams are selected by a power divergence statistic. Instead, the decision tree learner would consider all possible bigrams. Despite increasing the danger of fragmentation, this is an interesting issue since the bigrams judged most informative by the decision tree learner are not always ranked highly in the filtering step. In particular, we will determine if the filtering process ever eliminates bigrams that could be significant sources of disambiguation information.

In the longer term, we hope to adapt this approach to unsupervised learning, where disambiguation is performed without the benefit of sense tagged text. We are optimistic that this is viable, since bigram features are easy to identify in raw text.

## 11 Conclusion

This paper shows that the combination of a simple feature set made up of bigrams and a standard decision tree learning algorithm results in accurate word sense disambiguation. The results of this approach are compared with those from the 1998 SENSEVAL word sense disambiguation exercise and show that the bigram based decision tree approach is more accurate than the best SENSEVAL results for 19 of 36 words.

## 12 Acknowledgments

The Bigram Statistics Package has been implemented by Satanjeev Banerjee, who is supported by a Grant–in–Aid of Research, Artistry and Scholarship from the Office of the Vice President for Research and the Dean of the Graduate School of the University of Minnesota. We would like to thank the SENSEVAL organizers for making the data and results from the 1998 event freely available. The comments of three anonymous reviewers were very helpful in preparing the final version of this paper. A preliminary version of this paper appears in \[Pedersen, 2001\].