text:
# On the nature of a secondary main sequence turn-off in the rich LMC cluster NGC 1868

## 1 Introduction

NGC 1868 is a rich Large Magellanic Cloud (LMC) cluster located at $`\alpha =5^h14^m`$ and $`\delta =63^o57^{}`$ (J2000), approximately 6 away from the LMC’s centre. Its age has been estimated both from ground-based photometry and spectroscopy, often yielding discrepant results covering the $`310^8yrs`$ $`\stackrel{<}{}`$$`\tau `$ $`\stackrel{<}{}`$$`10^9yrs`$ range (Flower et al. 1980, Hodge 1983, Bica & Alloin 1986, Chiosi et al. 1986, Olszewski et al. 1991, Corsi et al. 1994 and references therein). Most age estimates come from the interpretation of optical colour-magnitude diagrams (CMDs), usually based on the positions of either the main-sequence turn-off (MSTO) or the clump of red giants (RC) or both. However, none of the CMDs available until now have been deep enough to allow probing the main sequence at magnitudes as faint as $`V22`$ with small photometric errors (see the web page on www.ast.cam.ac.uk/STELLARPOPS/LMCdatabase for a detailed list of references on NGC 1868 and other rich LMC clusters in the HST Cycle 7 program GO7307).

In this paper we investigate a possible second, fainter and therefore older, MSTO at $`V21`$, based on a deep CMD of NGC 1868 built from HST/WFPC2 data. This second population of stars may be the result of a strong interaction with another cluster, or of the capture of a lower-mass, older cluster by NGC 1868. In fact, there is evidence for a growing number of clusters in the LMC which may have undergone strong interactions or mergers (Kontizas et al. 1993, Dieball & Grebel 1998, Leon et al. 1999, Dieball, Grebel & Theis 2000). Confirmation of a merger event in a cluster’s history, however, requires detailed photometric and/or spectroscopic data. Sagar et al (1991), based on a CMD of NGC 2214 with two apparent supergiant branches, suggested that it was made up of two distinct populations. However, Lee (1992), Bhatia & Piotto (1994) and Banks et al (1995), based on larger and more accurate photometry, found a single population in the CMD of NGC 2214. Given that signatures of mergers or strong interactions remain observable in clusters for at least 1 Gyr (de Oliveira, Bica & Dottori 2000), NGC 1868, if confirmed as a merger product, may be a good laboratory for studying the dynamical effects and the final products of such events. Hence, it is essential to investigate closely the possibility that it is made up of two distinct populations.

Other explanations for a feature similar to a MSTO exist. Unresolved binaries, for instance, are usually brighter and redder than the main sequence of single stars. Therefore, an enhanced fraction of unresolved binaries could result in such a feature. Sudden changes in stellar structure, such as the onset of convective envelopes, may lead to CMD features such as the Böhm-Vitense gap, that may also mimic a MSTO (Böhm-Vitense 1970). These possibilities are also considered in the analysis of our deep NGC 1868 CMD, as well as of other similar CMDs.

The paper outline is as follows: in §2 we describe the HST/WFPC2 data used, both on NGC 1868 and on control field areas. We also discuss the process of removing contamination by field stars from the on-cluster CMD. In §3 we test the statistical significance of the candidate secondary MSTO by means of bootstrapping realizations. In §4 we use isochrones and star counts in selected CMD areas to extract useful information about this presumed secondary population in NGC 1868. In §5 we explore the possibility that the candidate MSTO is related to field stars, making use of CMDs obtained in control field areas. We also explore the alternative mechanisms that could yield to features similar to a MSTO at that position. Our main conclusions are presented in §6.

## 2 The data

The WFPC2 on-cluster data are described in more detail in Santiago et al. (2001). The off-cluster data used for field star subtraction were reduced and analyzed by Castro et al. (2001). In both cases the images were combined and calibrated using the standard pipeline procedure (Holtzman et al. 1995a,b). Photometry was carried out using DAOPHOT tasks and is described in detail by Santiago et al (2001) and Castro et al (2001). Zero-points, CTE and aperture corrections were also applied, again following standard procedures (Santiago et al 2001, de Grijs et al 2002a,b).

Figure 1 shows the resulting CMDs. The left panel shows stars belonging to the on-cluster images. Two such images were taken, one with the Planetary Camera (PC) at the centre of NGC 1868 and the other at its half-light radius. They are the CEN and HALF fields as defined by Santiago et al. (2001). The CMD shown represents the final sample, including stars from both HALF and CEN fields and with no repeats. The total on-cluster solid angle is approximately 7.5 arcmin. As explained in Santiago et al. (2001), the stars in the region common to the HALF and CEN fields had two independent photometric measurements and were thus used to determine uncertainties as a function of $`m_{555}`$ magnitudes and $`m_{555}m_{814}`$ colour. A total of 731 stars were found in the overlap region between the HALF and CEN fields. These empirical error determinations are an essential part of our upcoming analysis. The uncertainties were computed in bins 0.5 mag wide in both $`m_{555}`$ and $`m_{814}`$. In each magnitude bin we computed the mean value and standard deviation ($`\sigma `$) of the distribution of magnitude differences. The typical uncertainty of a single measurement was then assumed to be $`\sigma /\sqrt{2}`$. The photometric uncertainty analysis is described in more detail in Kerber et al (2002).

The right panel in Figure 1 shows stars in the off-cluster area, located 7.3’ away from the centre of NGC 1868 and previously studied by Castro et al (2001). This off-cluster field corresponds to a single WFPC2 field, covering about 5 arcmin. Both CMDs include only objects that were classified as stellar sources in the classification schemes presented in Santiago et al (2001) and Castro et al (2001). No additional cleaning of remaining spurious objects or background galaxies was made, resulting in some objects located well away from the cluster main-sequence (MS), the red giant branch (RGB) and the RC. Most of them should be contaminating unresolved background galaxies or faint stars in our Galaxy.

The on-cluster stars display a clear MS that terminates at $`m_{555}19.2`$. Previous works find the MS termination at $`V=19.219.3`$ (Corsi et al 1994, Brocato et al 2001). This is also roughly the magnitude of the RC ($`m_{555}m_{814}0.8`$), which contains He burning stars and is largely dominated by the cluster stars. In both panels, a clear RGB stretches downwards from the RC position to the bottom of the subgiant branch (SGB) at $`m_{555}22`$, $`m_{555}m_{814}0.8`$. These stars mostly belong to the old ($`\tau `$ $`\stackrel{>}{}`$$`10Gyrs`$) LMC field star population (Holtzman et al. 1997, Castro et al. 2001, Smecker-Hane et al 2002). Besides the main cluster and old field turn-offs, there are two additional features that may be interpreted as MSTOs in the CMD: one has mostly field stars with $`m_{555}20.3`$ and had already been identified by Castro et al. (2001) as a $`\tau 2Gyrs`$ field population; the other is fainter, at $`m_{555}21`$, $`0.4<m_{555}m_{814}<0.5`$ and consists entirely of stars in the on-cluster data. It is this latest feature that we concentrate on, as a candidate NGC 1868 subpopulation. The MSTOs described here are indicated in Figure 1.

### 2.1 Removing contaminating field LMC stars

In order to clean the left panel of Figure 1 from background field contamination, we statistically remove the control off-cluster CMD from the on-cluster one. Several procedures for doing so have been considered. One method is to match the off-cluster stars to corresponding stars in the on-cluster image using the estimated probabilities that two stars could be independent photometric measurements of each other. In order to estimate these matching probabilities, we use the empirically determined photometric uncertainties in both images and assume a Gaussian error distribution. Cluster stars are then randomly removed according to their probability of matching any of the off-cluster stars. In this approach a reliable estimate of the photometric uncertainties is very important. As pointed out previously, we used the independent magnitude measurements for the stars in common between the CEN and HALF fields to empirically estimate standard deviations ($`\sigma `$) in the error distribution at different magnitude bins (Santiago et al 2001, Kerber et al 2002).

An alternative way of removing contaminating field stars is to bin both on-cluster and off-cluster data in magnitude (or in magnitude and colour) and to subtract the histogram of the latter from that of the former. At each bin, a subset of the on-cluster stars, corresponding to the subtracted histogram, is randomly selected. The two approaches have been tested and yield similar results. As our main concern is to assess the existence of a second, older population superposed to the dominant NGC 1868 population, the branch of evolved stars is critical, since it provides the best opportunity to single out stars from each individual population. Therefore, field subtraction was carried out separately for the MS and for the SGB/RGB locus. This latter is marked on Figure 1.

A more detailed discussion about the issue of field stars removal is presented in an upcoming paper (Kerber et al. 2002).

The cleaned CMD is shown in Figure 2. The CMD is now cut-off at the faint end ($`m_{555}>24`$) as this region is not relevant to the current analysis. Below this limit, sampling incompleteness would only further complicate field star subtraction. We also cut it at bright magnitudes ($`m_{555}<19`$) to avoid regions where saturation effects start to take place ($`m_{555}19`$ and $`m_55517.8`$ for the HALF and CEN fields, respectively). The statistical removal of field stars has depleted the subgiant branch in the range $`21`$ $`\stackrel{<}{}`$$`m_{555}`$ $`\stackrel{<}{}`$$`22`$. A residual number of SGB stars brighter than $`m_{555}21`$ but much fainter than the cluster RC remains. As this excess of SGB stars reltive to the field is located at brighter magnitudes than the candidate second MSTO, it is a first evidence for the reality of this feature. As for the secondary MSTO itself, at $`m_{555}21`$, it was left untouched, which reflects the absence of field stars close to it. This again supports the reality of the second population in NGC 1868.

## 3 Statistical significance of the candidate secondary MSTO

In this section we will assess whether the feature we tentatively identify as a secondary MSTO in the CMD of NGC 1868 is statistically significant. More specifically we address the question of whether a similar feature could originate from a single main sequence through a random realization of photometric errors in the data. We explore this possibility by means of Monte-Carlo realizations of the cleaned NGC 1868 CMD. In each realization we collapse the cluster MS onto a fiducial line and redistribute the data, using the measured photometric uncertainties and assuming Gaussian error distributions in magnitude and colour.

The main sequence fiducial line is determined by taking the median $`m_{555}m_{814}`$ colour at different magnitude bins. We apply a 3$`\sigma `$ clipping to the colour distribution in order to eliminate outliers and iterate until convergence. We initially define this MS line in bins spaced by 0.2 mag. A cubic spline interpolation is then used to fill this line in a much narrower ($`\mathrm{\Delta }m_{555}=0.02`$) binning. Each true MS star is then randomly assigned to one of the points along the MS fiducial line, using a two-dimensional Gaussian probability function whose standard deviations are given by the star’s measured magnitude and colour uncertainties. We then redistribute the data back onto the CMD plane, the magnitude and colour of each artificial star again being a random realization of the same Gaussian distribution of errors.

One hundred such Monte-Carlo simulations were carried out. A subset with 11 of the resulting CMDs is shown in Figure 3. The actual data are shown in the lower-right panel for comparison. The MS fiducial line is shown as a white line for guidance. The artificial CMDs have similar width as the data but are unable to reproduce the apparent turn-off at $`m_{555}21`$, or the residual of the old field population turn-off ($`m_{555}22`$), whose stars are farther to the red of the MS and much more clumped together than any set of points in any of the simulated CMDs. The simulations also fail to reproduce the main cluster MSTO at $`m_{555}19`$ and $`m_{555}m_{814}0.3`$. This proves that scatter due to photometric errors alone cannot account for these features seen in the data. Notice that these experiments do not explicitly account for the effect of unresolved binaries on the CMD. The fact that the distribution of true stars is skewed redwards from the MS fiducial line is likely caused by unresolved binaries. Incorporation of unresolved binaries would help spread out the simulated stars towards redder colours, but is unlikely to reproduce the features mentioned above, especially if we consider that part of the effect is already incorporated in the position of the MS fiducial line. We discuss the effect of binaries in more detail in §5.2.

## 4 The candidate NGC 1868 second population

Assuming that the second MSTO at $`m_{555}21`$ is real and belongs to the cluster, one wishes to determine what are the main characteristics of this population in NGC 1868.

In Figure 4 we again show the cleaned NGC 1868 CMD. Superimposed to the data we show three Padova isochrones (Girardi et al. 2000). In plotting the isochrones, we assume a distance modulus of $`mM=18.5`$ to the LMC (Panagia et al. 1991). The data were extinction corrected assuming E(B-V) =0.04 as indicated. Conversion from E(B-V) to E(555-814) was done as described by Holtzman et al (1995a,b). The $`[Fe/H]`$ and age (in Myrs) of each isochrone are shown in the figure. The \[-0.4,800\] isochrone fits the main NGC 1868 population. The other isochrones are attempts to fit the second turn-off and SGB. The younger and more metal-rich isochrone with \[Fe/H\] = -0.4 and an age of 2500 Myrs provides the best fit to both turn-off and SGB regions.

Using star counts and this best fitting isochrones, we can estimate the mass fraction of this candidate older NGC 1868 population relative to the dominant and younger one. There are 11 stars in the range $`2.5`$ $`\stackrel{>}{}`$$`M_{555}`$ $`\stackrel{>}{}`$$`1.5`$ ($`21.0`$ $`\stackrel{>}{}`$$`m_{555}`$ $`\stackrel{>}{}`$$`20.0`$) and located along the SGB. Besides these, 13 stars are located within the CMD area whose limits are: $`20.85m_{555}21.3`$, $`m_{555}m_{814}0.6`$ and $`m_{555}5.333(m_{555}m_{814})+18.55`$. This corresponds to the region where the second MSTO is clearly detached from the MS. Assuming that all such $`N_{old}=24`$ stars are in fact associated with an older population belonging to NGC 1868 and using the best fitting isochrone, we infer the masses that correspond to the limiting range in absolute magnitudes: they are $`m_{min}1.36m_{}`$ for the basis of MSTO position and $`m_{max}1.49m_{}`$ for $`M_{555}=1.5`$ ($`m_{555}=20`$) along the SBG. As for the dominant cluster population we find $`N_{young}`$ = 732 stars along the cluster MS with $`M_{555}<2`$ ($`m_{555}<20.5`$) and fainter than the MS termination (we cut it at $`m_{555}=19`$). These limits correspond to $`1.63`$ $`\stackrel{<}{}`$$`m/m_{}`$ $`\stackrel{<}{}`$$`2.10`$ for the best fitting isochrone to the NGC 1868 upper MS (\[Fe/H\],age(Myrs) = -0.4,800). Assuming the cluster’s present day mass function (PDMF) to be a power-law with fixed slope $`\alpha `$ within the mass ranges considered, the mass ratio will be given by

$$\frac{M_{old}}{M_{young}}=\frac{N_{old}}{[m_{max,old}^{1\alpha }m_{min,old}^{1\alpha }]}\frac{[m_{max,young}^{1\alpha }m_{min,young}^{1\alpha }]}{N_{young}}$$

Assuming a Salpeter value for the MF ($`\alpha =2.35`$), and the limiting mass values quoted above for each of the subpopulation, we then infer $`M_{old}/M_{young}=0.06`$. For $`\alpha =1.5`$ we have $`M_{old}/M_{young}=0.08`$ and for $`\alpha =3.2`$, $`M_{old}/M_{young}=0.05`$. The mass ratios are also fairly insensitive to adopting the (\[Fe/H\],age(Myrs)) = (-0.7,3500) Padova isochrone for the secondary population; in this case, the old population mass range would be $`1.22<m/m_{}<1.29`$ and $`M_{old}/M_{young}`$ would change by no more than 50% for any choice of $`\alpha `$. The mass ratio estimates are summarized in Table 1.

In addition to the PDMF slope and the model mass-luminosity relation, another source of uncertainty in the mass ratio quoted above is the residual contamination by field stars to the $`N_{old}`$ counts in the SGB region. Previous experiments with the field star subtraction (see §2.1) methods have led to variations of up to $`35\%`$ in the resulting value of $`N_{old}`$. If we assume this to be the uncertainty caused by field contamination, a similar relative error is expected to propagate into $`M_{old}/M_{young}`$. Notice that this is a conservative reasoning, since residual contamination by field LMC stars should also affect the younger population. We should also point out that this $`35\%`$ uncertainty is larger than the Poisson fluctuation expected for the $`N_{old}`$ counts.

An important question is how the stars in this candidate secondary population are distributed throughout the cluster. Differences in spatial distribution between the two populations could indicate a recent merger, in which the two systems have not yet attained dynamical equilibrium. Figure 5 shows the projected distribution of the stars located at the second turn-off; they are the same 13 stars used in estimating the $`M_{old}/M_{young}`$ ratio earlier in this section. The 11 more evolved stars along the SBG were left out of Figure 5 because they represent just a statistical excess relative to field counts; hence, they are not necessarily true members of the older population. The contours represent circles around the cluster centre and the attached labels show the percentage of all cluster stars located inside them. We note that, within the fluctuations expected by the small numbers, the projected distribution of these stars is consistent with that of the entire cluster.

## 5 Alternative interpretations

We have so far been interpreting the CMD feature at $`m_{555}21`$ and $`m_{555}m_{814}0.4`$ as a second MSTO associated with NGC 1868. In this section we discuss other possible interpretations.

### 5.1 Is it a population associated with the LMC field?

One obvious alternative is that the feature in the NGC 1868 CMD is in fact a MSTO, but associated with stars belonging to the general LMC field population rather than to the cluster itself. Notice however that this is unlikely, since most contaminating field stars have been statistically removed from the on-cluster CMD. Furthermore, Figure 5 shows that the stars located at the position of the second turn-off are concentrated towards the cluster centre, a strong evidence in favour of a cluster origin for them. On the other hand, the star formation history in the LMC is known to be complex, especially in the last few Gyrs, when the bar was formed (Gallagher et al 1996, Elson et al 1997, Smecker-Hane et al 2002). Thus, small scale variations in the field CMDs are possible, making field subtraction more uncertain.

One way to investigate possible MSTOs in field populations is to search for similar features in the 7 CMDs of field stars studied by Castro et al (2001). These are fields located 7.3’ from the target clusters of the GO7307 HST project. In fact, Castro et al have visually identified possible turn-offs, with ages in the range $`2`$ $`\stackrel{<}{}`$$`\tau `$ $`\stackrel{<}{}`$$`4`$ Gyrs, in several of these fields. In addition to those, turn-offs associated with an old ($`>10`$ Gyrs) population were found in all field CMDs. In some cases, visual inspection of the CMDs revealed a broadening in the MS for $`m_{555}`$ $`\stackrel{<}{}`$$`21.5`$, indicative of continuous star formation in the LMC.

In order to quantify these visual impressions we tried to identify features in the field CMDs containing stars that are highly detached from the MS line, similar to the presumed NGC 1868 MSTO. One obvious such locus is the SGB/RGB region, which is always present in the CMDs of field LMC stars. It thus has to be eliminated from the analysis a priori. We must also avoid contamination from background galaxies and faint stars belonging to the Galaxy. Unresolved galaxies should be limited to faint magnitudes, $`m_{555}`$ $`\stackrel{>}{}`$$`22.5`$, as brighter ones are visibly extended in our WFPC2 images. From counts of faint compact galaxies, one expects $`50`$ contaminating galaxies within the $`22.5m_{555}25`$ range in our on-cluster field (Abraham et al 1996). As for foreground stars, a similar number is expected in the entire observed CMD (Santiago et al 1996). Together both types of contaminating sources contribute with a few percent of the total sample and will be spread out in the CMD.

We therefore proceeded as follows. For each field a fiducial line representing the CMD MS was defined in the same way as described in §3. A low-order polynomial was fit to the MS fiducial line of most fields in order to smooth out the wiggles caused by noise in the median $`m_{555}m_{814}`$ value at each $`m_{555}`$ bin. For NGC 1805 and NGC 1818, which have much larger numbers of MS stars than the other fields, especially at the bright end ($`20.5<m_{555}<19`$), the raw MS line was used. Evolved stars, possible background galaxies and faint stars belonging to the Galaxy were then eliminated by cutting out all CMD objects beyond $`\pm 5\sigma `$ from the MS line, where $`\sigma `$ is the empirically determined standard deviation in the $`m_{555}m_{814}`$ colour distribution at each magnitude. For the remaining stars, we computed the number of standard deviations by which each star is detached from the MS fiducial line, $`n_{sig}=\mathrm{\Delta }(m_{555}m_{814})/\sigma `$.

In Figure 6, we show the stars whose $`n_{sig}`$ values fall at the 95% position or beyond in each LMC field. The number of stars in each panel varies from 30 to 130. These stars would be $`>2\sigma `$ (and $`<5\sigma `$) events of a Gaussian error distribution in colour. The MS fiducial line and the $`\pm 5\sigma `$ lines are also shown for guidance. Assuming that the photometric uncertainties have been adequately measured over the entire $`m_{555}`$ range and that these stars just reflect the high tail of the error distribution, we would expect them to follow the distribution of MS stars. Therefore, the shape of the distribution of such stars may reveal features that are not accounted for by errors alone. As an example, a clumped distribution of such stars at some magnitude range, as compared to the smooth distribution of MS stars, would indicate the existence of features such as a MSTO.

There is a common pattern in the distribution of these 5% reddest stars in the 7 LMC fields. In all cases, the majority of them are within the range $`20<m_{555}<22`$. This can be also seen by the frequency distribution of these stars as a function of $`m_{555}`$, which is shown in Figure 7. In all fields, large $`n_{sig}`$ stars show a distinct peak in their magnitude histogram when compared to the distribution of normal MS stars. This peak is usually broad and covers the $`m_{555}`$ range mentioned above. Examples are the fields close to NGC 1805, NGC 1818 and NGC 1868. NGC 1805 and NGC 1818 are close to star formation regions and their neighbouring field includes stars of different magnitudes (masses) leaving the MS towards the RGB. As for the field close to NGC 1868, a 2 Gyr MSTO is seen in its CMD (Castro et al 2001), contributing with stars brighter than $`m_{555}=20.5`$.

In some cases, on the other hand, high $`n_{sig}`$ field stars show a narrow and very tall peak. This is the case of the field stars in the neighbourhood of NGC 1831, which display a distinct peak at $`m_{555}21`$; this is consistent with the position of the $`4Gyrs`$ turn-off identified in this field by Castro et al (2001). Other examples of high peaks in the distribution of detached field stars include the fields close to Hodge 14 and NGC 2209. These peaks are at least $`0.5`$ mag fainter than that of NGC 1831 and are close to the basis of the SGB of the old LMC field populations. Inspection of the stars contributing to these peaks in fact confirms that they are dominated by SGB stars which were not successfully removed using the $`5\sigma `$ colour cut-off.

Figures 8 and 9 show, respectively, the CMD and $`m_{555}`$ histogram for the stars in the on-cluster, background cleaned NGC 1868 data whose $`n_{sig}`$ values are at the 95% position or beyond. These figures follow the same conventions as Figures 6 and 7. A total of 201 stars contribute to Figures 8 and 9. The larger number of stars allowed a magnitude binning in Figure 9 narrower than in Figure 7. Notice that the on-cluster histogram of high $`n_{sig}`$ stars as a function of magnitude now shows a large and narrow peak at a bright magnitude, $`m_{555}19.5`$, which is close to the dominant NGC 1868 MSTO. A second peak is seen in the range $`20.5<m_{555}<22`$, similar to those found in the field CMDs. This latter peak is well centered where the candidate secondary MSTO was found. Apart from the bright peak, however, the distribution of high $`n_{sig}`$ stars as a function of $`m_{555}`$ in the cluster data is not markedly different from those of field stars.

Thus, we conclude that a field origin for the second turn-off in NGC 1868 may not be ruled out. However, the differences in shape between this feature and those known to be associated with field LMC stars, plus the fact that the NGC 1868 CMD had previously been removed of field stars, argue against this interpretation. Furthermore, the suspected secondary turn-off in NGC 1868 is detached enough from the MS that the two reddest stars contributing to it actually are located beyond the $`+5\sigma `$ line shown in Figure 8 (see Figure 2), and therefore do not contribute to the histogram in Figure 9.

### 5.2 Selective unresolved binarism and Am stars

As mentioned in §3, there are other causes, besides a MS turn-off, that could lead to a set of stars being significantly detached from the MS towards redder colours.

Unresolved binarism is one such possibility. The presence of a secondary star, whose flux is added to the primary, will shift the position of the system towards brighter magnitudes in a CMD, relative to the position of the primary. If both are MS stars, the secondary will be redder, thus also affecting the colour of the system. Unresolved binaries are certainly present in any CMD and should be distributed all over the MS and among evolved stars. Their main effect is to make the MS broader with a red tail in the colour distribution. The feature seen in Figures 1 and 2, at $`m_{555}21`$, could be explained only if there is a specific increase in the fraction of cluster binaries at an absolute magnitude $`M_{555}2.5`$.

The apparent second MSTO occurs at the luminosity typical of peculiar A stars (Ap, Am, etc). Due to their chemical anomalies, Am stars in particular tend to be redder than normal A stars by about 0.05 mag as a result of increased line blanketing. This effect alone is too small to account for the feature in question but certainly goes in the right direction. As many Am stars are also known to be binaries (Carquillat et al 2001, Debernardi et al 2001), a combination of blanketing and enhanced binarism might accommodate the observed feature in the CMD of NGC 1868. However, more stringent constraints on the frequency of Am stars and on their binary fraction must be placed in order to test this possibility. Another problem with this interpretation is that global binary fraction estimates tend to be smaller in rich clusters than in the general field (Mayor et al 1996, Coté et al 1996, Elson et al 1998).

### 5.3 The Böhm-Vitense gap

The onset of a convective envelope is known to occur for MS stars of spectral type A or later. This change in energy transport is expected to occur abruptly for stars with $`T_{eff}7500K`$, or $`(BV)0.22`$ (Böhm-Vitense 1958). As a result, stars cooler than this critical effective temperature would become redder than the slightly hotter ones with fully radiative envelopes (Böhm-Vitense 1970). The resulting discontinuity in the CMD is the so-called Böhm-Vitense gap and is thought to have an amplitude of $`\mathrm{\Delta }(BV)0.10`$. Recent surveys combining precise photometry and distance measurements from astrometry have shown that a gap does exist and is located not too far from where expected by theory. But the evidence is often weak or subject to selection effects. Newberg & Yanny (1998) studied bright and nearby field stars with available photometry plus distances from Hipparcos and found a gap at $`0.2<(BV)<0.3`$ in the CMD of field stars of luminosity class V only. In other words, the gap is masked out by evolved stars contaminating the apparent MS in their CMD. Rachford & Caterna (2000) detected a gap at $`(BV)0.35`$ in most nearby open clusters they studied, but they argue that this gap may be unrelated to the onset of envelope convection.

We have carried out the same gap detection analysis as Rachford & Caterna (2000). We computed $`\chi ^2`$ values as defined by those authors for gap candidates within the $`0.15<m_{555}m_{814}<0.60`$ colour range and with widths varying from 0.02 to 0.10 mag. Figure 10 shows the $`m_{555}m_{814}`$ frequency distribution, not only for NGC 1868, but also for 3 other rich LMC clusters in our sample, each one with more than 5000 stars in their CMD (Santiago et al 2001). The clusters are paired according to age, NGC 1805 and NGC 1818 being younger ($`\tau <10^8`$ yrs, Johnson et al 2001) than NGC 1831 and NGC 1868. The colour distributions for the older clusters is slightly flatter than those for the younger ones. Apparent gaps are seen at $`m_{555}m_{814}0.40`$ (in all cases) and at $`m_{555}m_{814}0.52`$ (except NGC 1831). A broader gap is also present at $`m_{555}m_{814}0.250.30`$ in NGC 1805 and NGC 1818 and possibly in NGC 1831 and NGC 1868.

Figure 11 shows the gap center plotted against gap width for all our gap candidates with $`>90\%`$ confidence ($`\chi ^2>2.8`$). Each panel corresponds to one of the 4 rich clusters discussed. The gaps at $`m_{555}m_{814}0.4`$ are significant in all clusters except NGC 1818. At $`m_{555}m_{814}0.52`$, only narrow and fairly unconspicuous gaps are confirmed. The bluer gap, at $`m_{555}m_{814}0.25`$, takes different widths and positions, being more significant in NGC 1805 and NGC 1831. Hence, the main feature, clearly present in 3 rich clusters, is the gap at $`0.35<m_{555}m_{814}<0.40`$. Notice that this colour coincides closely with that of the candidate MSTO, adding support to the statistical reality of this CMD feature. These gap limits correspond approximately to $`0.23<BV<0.27`$, which is close to the expected position of the Böhm-Vitense gap and somewhat displaced from the gap position favoured by Rachford & Caterna (2000).

The presence of colour gaps in most of our rich LMC clusters calls for a more common mechanism than merger events. Our gap detection results, therefore, favour the possibility that the Böhm-Vitense gap was detected for the first time in LMC clusters. On the other hand, our gap colour is inconsistent with the one found by de Bruijne et al (2000, 2001) in their very accurate Hyades data. In fact, the very narrow CMDs by these authors provide the strongest limits on the gap position and associated colour discontinuity, which seems to be of $`\mathrm{\Delta }(BV)0.05`$. This is significantly smaller than the colour difference between MS stars and the secondary MSTO found in our NGC 1868 data.

## 6 Conclusions

In this paper we have shown possible evidence for a second main sequence turn-off in a deep WFPC2/HST colour-magnitude diagram of NGC 1868. This feature is clearly visible in the cluster CMD, at $`m_{555}21`$ and $`m_{555}m_{814}0.4`$, especially after contaminating field stars are removed. The presence of these stars, as well as of a residual SGB/RGB brighter than $`m_{555}=21`$, is evidence for a secondary cluster population. In fact, previous ground-based data also show a hint of this feature: the distribution of stars seen redwards of the main-sequence in the NGC 1868 CMD by Corsi et al. (1994), despite the large scatter, has an excess of stars at the same position.

Assuming that the feature is really a MSTO associated with NGC 1868, isochrone fits yield an age of $`2.53.5Gyrs`$ and a metallicity of $`0.7<[Fe/H]<0.4`$ for this subpopulation. Using star counts in CMD regions where we expect only one cluster subpopulation to be present, we estimate the mass ratio of the older subpopulation relative to the younger to be in the range $`0.05`$ $`\stackrel{<}{}`$$`M_{old}/M_{young}`$ $`\stackrel{<}{}`$$`0.12`$. This range of values incorporates uncertainties in mass function slope, in age and metallicity of the secondary population, as well as uncertainties in the statistical removal of contaminating field LMC stars.

Even though the candidate second turn-off remains untouched after field stars are statistically subtracted from the CMD of NGC 1868, the possibility that this feature is associated with a field star population was investigated in detail and cannot be completely ruled out. CMDs of field stars from several different positions in the LMC show an excess of stars located in loci close to that of the NGC 1868 candidate turn-off. This excess is measured relative to the expected number of stars scattered to these loci due to photometric errors. Most of the features in the field CMDs, however, are broad and visually less conspicuous than the apparent secondary turn-off in NGC 1868, thus being more consistent with periods of enhanced field star formation lasting for longer than $`1Gyr`$. Another argument against a field origin for the candidate second turn-off is that its stars are concentrated towards the cluster centre.

Other possible explanations for the CMD feature have been explored. They include CMD spread due to unresolved binaries, which could mimic a MS turn-off if binarism is enhanced within a narrow range of primary star masses. It is interesting to notice that the mass range in question is close to that of Am stars, which are redder than normal A stars and for which a larger than usual binary fraction may exist. Estimates of binary fraction within globular clusters, however, yield smaller values than in the field. Besides, the binary fraction among Am stars is not yet well constrained. It is thus unlikely that the CMD feature discussed here is due to Am stars or unresolved binaries.

A final and exciting alternative would be that the Böhm-Vitense gap was for the first time detected in a stellar population outside the Galaxy. The fact that gaps were found in the colour distribution of two other LMC clusters besides NGC 1868, all of them at $`m_{555}m_{814}0.4`$ ($`(BV)0.25`$), calls for a more common mechanism than mergers. In fact, a merger of the type suggested by the presence of a second and much fainter MSTO as discussed in this paper should be rare (Vallenari et al 1998, Dieball & Grebel 2000). Most model predictions favour merging units that are nearly coeval and usually formed through encounters within the same giant molecular cloud, either before or after they are fully formed clusters (Fujimoto & Kumai 1997, Efremov & Elmegreen 1998, Leon et al 1999).

On the other hand, it is unclear to what extent the Böhm-Vitense gap may mimic a main sequence turn-off. The recent and precise CMD data on the Hyades by de Bruijne et al (2000, 2001) do not reveal strong turn-off like features associated with the Böhm-Vitense gap candidates found by those authors; the colour discontinuity in their CMD is of smaller amplitude than necessary to account for the NGC 1868 feature studied here. Besides, the gap associated with our NGC 1868 CMD feature is bluer than the gaps found by de Bruijne et al.

We should point out that the nature of the stars that make up the candidate MSTO in NGC 1868, which currently is uncertain and accountable for by at least two astrophysically interesting alternatives (merging and Böhm-Vitense gap), may be established by spectroscopic classification, a task likely to demand large, 8m class, telescopes.

## Acknowledgments

We thank Gerry Gilmore, Steinn Sigurdsson, Horacio Dottori and Eduardo Bica for useful discussions. This work was partially supported by CNPq and PRONEX/FINEP 76.97.1003.00.


format: text

text:
# Search for 𝜂_𝑏(1𝑆) in Inclusive Radiative Decays of the Υ(3𝑆)

## I Introduction

It has been more than 20 years since the $`\mathrm{{\rm Y}}`$(1S) was discovered , yet no singlet state of $`b\overline{b}`$ system has been seen, including the ground state of the $`b\overline{b}`$ system, $`\eta _b`$(1S).

Singlet S states ($`\eta _b`$) of bottomonium may be produced in the decays of triplet S states ($`\mathrm{{\rm Y}}`$) through magnetic dipole (M1) radiative transitions which occur between states that have opposite quark spin configurations and the same orbital angular momentum. The decay rate for magnetic dipole transitions between between $`\mathrm{{\rm Y}}`$ and $`\eta _b`$ is given by

$$\mathrm{\Gamma }[\mathrm{{\rm Y}}(nS)\eta _b(n^{}S)\gamma ]=\frac{4}{3}\alpha \frac{e_b^2}{m_b^2}I^2k^3$$
(1)
where $`\alpha `$ is the fine-structure constant, $`e_b`$ is the b-quark charge (-$`\frac{1}{3}`$), $`I`$ is the model dependent interaction matrix element, $`k`$ is the photon energy, and $`m_b`$ is the b-quark mass in GeV/$`c^2`$.

There are two classes of M1 transitions: *direct*, when the initial and the final radial quantum numbers are the same ($`n=n^{}`$), and *hindered*, when they are not the same ($`nn^{}`$). Small, non-zero values of $`I`$ for hindered transitions arise from either hyperfine interactions or finite size corrections, as well as other relativistic effects. Estimates of these effects are model dependent.

As Godfrey and Rosner  have pointed out, some hindered transitions could have observable rates, in spite of smaller matrix element $`I`$, because of the large phase space factor, $`k^3`$. In addition, photons emitted in direct transitions are expected to be very soft (i.e. $`k<100`$ MeV for $`\mathrm{{\rm Y}}(nS)\gamma \eta _b(nS)`$) and fall into the region where experimental backgrounds are high and energy resolution is poor. This makes detection of direct M1 transitions in the $`\mathrm{{\rm Y}}`$ system very difficult. From the standpoint of $`\gamma `$ detection and energy resolution, we have a better chance of detecting hindered transitions.

In this paper we present the result of a search for the hindered magnetic dipole (M1) transition $`\mathrm{{\rm Y}}`$(3S) $``$ $`\gamma \eta _b`$(1S) using the CLEO III detector. In Section II, we describe the detector, data sample and event selection used for this search. Section III is devoted to the electric dipole (E1) transition, $`\chi _b(2P_J)\gamma \mathrm{{\rm Y}}(1S)`$, whose emitted photon energy is $``$770 MeV. This is close to the predicted photon energy ($``$910 MeV ) in the $`\mathrm{{\rm Y}}(3S)\gamma \eta _b(1S)`$ transition (see Figure 1).

We use this nearby E1 transition to optimize our cuts for the signal photons. In Section IV, we describe how we fit to and extract the signal candidate from the data. Section V describes the photon detection efficiency and possible systematic errors in our calculation of branching ratios. Since no convincing signal is found, we present upper limits on the branching ratio. We summarize our results in section VI.

## II CLEO-III Data Sample, Detector and Event Selection

Recently the CLEO III detector has accumulated an integrated luminosity of 1.1 fb-1 using the Cornell Electron Storage Ring (CESR) running at the $`\mathrm{{\rm Y}}(3S)`$ resonance and 0.13 fb-1 in the continuum (below the $`\mathrm{{\rm Y}}(3S)`$ resonance). This is roughly a ten-fold increase in statistics over $`\mathrm{{\rm Y}}(3S)`$ data collected with the CLEO II detector. While several changes have been made to the detector in the upgrade from CLEO II to CELO III , the CsI calorimeter, upon which this analysis depends, has undergone only slight modification. The CsI calorimeter barrel section is unchanged with respect to CLEO II, and the endcaps have been rebuilt to accommodate the new CESR interaction quadrupole magnets. Material in front of the calorimeter was significantly reduced for the barrel ends and the endcap sections, improving photon efficiency and energy resolution.

Since $`\eta _b`$ is not expected to have a large branching ratio to any particular exclusive final state, we look inclusively for the photon from $`\mathrm{{\rm Y}}(3S)\gamma \eta _b(1S)`$. We first select generic hadronic events by requiring the following criteria; (i) number of charged tracks $`>`$ 2, (ii) total visible energy***Total energy of charged tracks and calorimeter energy of showers that do not match with charged tracks $`>0.2E_{CM}`$, (iii) number of charged tracks $`>`$ 4 or total CsI calorimeter energy $`>0.15E_{CM}`$ and either CsI calorimeter energy or twice the energy of the most energetic shower in calorimeter $`<0.75E_{CM}`$. We then look at the inclusive photon spectrum, searching for a peak in the $`900`$ MeV range.

According to our Monte Carlo in which generic $`\mathrm{{\rm Y}}`$(3S) decays were generated by LUND/JETSET, our hadronic event selection efficiency is 93.2%. The inefficiency here is mostly due to the leptonic decays of $`\mathrm{{\rm Y}}`$(3S) and small contributions due to the cascade decays of $`\mathrm{{\rm Y}}`$(3S) to $`\mathrm{{\rm Y}}`$(2S) or $`\mathrm{{\rm Y}}`$(1S) which subsequently decay into two leptons. We estimate the number of $`\mathrm{{\rm Y}}`$(3S) produced in CLEO III to be 4.7$`\times 10^6`$ by correcting the number of continuum subtracted hadronic events by this efficiency.

## III Cut optimization on the Electric Dipole (E1) Transition

To optimize our event selection, we use the nearby structure of the Electric Dipole (E1) transition, $`\chi _b(2P_J)\gamma \mathrm{{\rm Y}}(1S)`$ (see Figure 1). We optimize our cuts such that these E1 peaks in our data sample are best resolved. In this section, we first describe our selection cuts used to select isolated photons. Then we describe our efforts to suppress $`\pi ^0`$’s, which are the dominant background in this analysis.

### A Isolated Photon Selection Criteria

We select photons only in the barrel region ($`|\mathrm{cos}\theta |<0.804`$), $`\theta `$ is the polar angle measured relative to the beam pipe. which offers high efficiency and the best energy resolution. We reduce this sample to include only well-isolated photons by requiring that the ratio of the two sums of energies in the group of 9 and 25 crystals around the central crystal should peak close to 1. We set the cut to be the value determined by Monte Carlo such that the cut retains 99% of the generated photons. We further reject hadronic showers and showers due to electrons by requiring that they do not match charged tracks.

### B $`\pi ^0`$ Suppression

In any analysis of inclusive photon spectra, the background arises primarily from $`\pi ^0`$’s. In order to suppress this background, we pair our photon candidates with other neutral showers in either the endcap ($`|\mathrm{cos}\theta |>0.849`$) or barrel regionWe exclude the transition region between the endcap and barrel regions and reject the candidate of the pair that satisfies the following $`\pi ^0`$ criteria. We first cut on $`|\sigma _{\pi ^0}|=|(m_{\gamma \gamma }m_{\pi ^0})/\mathrm{\Delta }m_{\gamma \gamma }|<2.5`$ where $`m_{\gamma \gamma }`$ is an invariant mass of the two photons and $`\mathrm{\Delta }m_{\gamma \gamma }`$ is its estimated resolution. When $`|\sigma _{\pi ^0}|<2.5`$ we require the cosine of the opening angle between the two photons to be greater than 0.7.

## IV Fitting the data

We fit the inclusive photon spectrum with three components; (i) a 3rd order exponential polynomial for background, (ii) a peak due to $`\chi _b(2P_J)\gamma \mathrm{{\rm Y}}(1S)`$ represented by three overlapping Gaussians due to different members of the $`\chi _b(2P_J)`$ triplet, and (iii) a signal peak due to $`\mathrm{{\rm Y}}(3S)\gamma \eta _b(1S)`$ represented by a Crystal Ball Line Shape, which is a Gaussian with an asymmetric low energy tail.

We use logarithmic binning in energy which preserves nearly constant resolution across a wide range of energies. Figure 2 shows the fit. For illustrative purposes we show in Figure 3 a close-up of the signal region (background subtracted) with this fit superimposed. We performed a series of fits assuming $`E_\gamma `$ = 880 to 1000 MeV, obtaining a maximum signal yield of $`698\pm 463`$ which is only $`1.5\sigma `$ significant. Since we do not find evidence for the signal, we set upper limits on $`[\mathrm{{\rm Y}}(3S)\gamma \eta _b(1S)]`$ (see Section V C).

## V Detection Efficiency and Branching Ratio

In this section we describe how the photon detection efficiency is estimated. We also discuss our estimates of systematic errors in the calculation of the branching ratio. At the end of this section, we present upper limits on the branching ratio.

### A Detection Efficiency

In order to convert the fitted amplitude to the branching ratio of $`\mathrm{{\rm Y}}(3S)\gamma \eta _b(1S)`$, we generated 18000 $`\mathrm{{\rm Y}}(3S)`$ decays using the LUND/JETSET with radiative tails in which $`\mathrm{{\rm Y}}(3S)`$ always decays into $`\eta _b(1S)`$ and $`\gamma `$ isotropically.§§§In our signal MC, the mass of $`\eta _b(1S)`$ was set to be 9.4GeV/$`c^2`$ and it decays into 2 gluon jets according to the LUND/JETSET model. The fit to the Monte Carlo photon spectrum is shown in Fig. 4. From the fit, the signal efficiency is $`ϵ_{flat}=(50.8\pm 0.7)\%`$. This efficiency is further corrected to account for the true angular distribution of M1 photons, $`1+\mathrm{cos}^2\theta `$ . This function is integrated over the barrel region of the calorimeter, a corrected efficiency gives $`ϵ=0.91ϵ_{flat}`$.

### B Systematic Study

To allow for the possible mismodeling of the efficiency of various cuts, we calculate $`(\chi _b(2P_J)\gamma \mathrm{{\rm Y}}(1S))`$ using the same method as above, and then using the same method without the $`\pi ^0`$ suppression and $`\gamma `$ isolation cuts. Variations in the branching fractions calculated in these ways is taken as the systematic error.

In order to estimate the systematic error on the signal efficiency due to the Monte Carlo modeling of the decay, we generate signal Monte Carlo with a different $`\eta _b`$ decay model . We then compare the efficiencies derived from these two models.

The number of $`\mathrm{{\rm Y}}(3S)`$ decays in our sample is determined with 2% systematic error due to the uncertanities in the hadronic selection efficiency

We summarize our study of systematic error in Table I. The total systematic error is obtained by summing all entries in quadrature.

### C Upper Limit on Branching Ratio

We obtain a 90% confidence level upper limit on the fitted amplitude of signal candidates by the following method. We only consider the *physical region* of the signal amplitude assuming that the shape of the likelihood is Gaussian. We then determine the 90% confidence level upper limit to be the point below which 90% of the *positive side* of the likelihood distribution is included (e.g. see ). Once we find the upper limit of the amplitude, we then convert it to a branching ratio by dividing by the signal efficiency and the number of $`\mathrm{{\rm Y}}(3S)`$. Finally we accommodate our systematic errors by $`_{UL}=_{original}\times 1.056`$.

Figures 5 shows our upper limits on $`[\mathrm{{\rm Y}}(3S)\gamma \eta _b(1S)]`$, as a function of the emitted photon energy, $`E_\gamma `$ (equivalently the hyperfine splitting = $`M_{\mathrm{{\rm Y}}(1S)}M_{\eta _b(1S)}`$), in which we searched for the signal between 880MeV and 1000MeV by repeating the above calculations of upper limit branching ratios for each fixed signal photon energy. Some model predictions taken from Ref.  are also overlaid in the plot.

## VI Conclusions

Using $`4.7`$ million $`\mathrm{{\rm Y}}(3S)`$ decays collected by the CLEO III detector, we have searched for the ground state of the $`b\overline{b}`$ system, $`\eta _b(1S)`$, through the hindered Magnetic Dipole (M1) transition, $`\mathrm{{\rm Y}}(3S)\gamma \eta _b(1S)`$. No significant signal was found in the range of 880MeV$`E_\gamma `$ 1000MeV. We set 90% confidence level upper limits on $`[\mathrm{{\rm Y}}(3S)\gamma \eta _b(1S)]`$ in this photon energy range which rule out some previously published phenomenological predictions of this rate.

We gratefully acknowledge the effort of the CESR staff in providing us with excellent luminosity and running conditions. M. Selen thanks the PFF program of the NSF and the Research Corporation, and A.H. Mahmood thanks the Texas Advanced Research Program. This work was supported by the National Science Foundation and the U.S. Department of Energy.


text:
# First observation of a transition between “parallel tracks” in the kHz QPO frequency vs. intensity diagram

## 1. Introduction

Observations with the Rossi X-ray Timing Explorer (RXTE) have revealed kilohertz quasi-periodic oscillations (kHz QPOs) in some 20 X-ray binaries. It is generally thought that these kHz QPOs reflect the motion of matter in orbit at some preferred radius in the accretion disk around the neutron star (see van der Klis 2000 for a review of the phenomenology of these QPOs, and for a description of the models so far proposed to explain them).

Calculations by Miller, Lamb, & Psaltis (1998) show that the inner radius of the disk is set by angular momentum losses to the radiation field: When mass flow through the disk increases, the inner radius of the disk decreases, and therefore the QPO frequency (the Keplerian frequency at this radius) increases. But because in accretion-powered systems luminosity is proportional to the mass accreted onto the compact object, from the above it follows that there should be a one-to-one relation between QPO frequency and bolometric flux.

To the extent that X-ray flux is a good measure of the bolometric flux (see, e.g., Ford et al. 2000), observations with RXTE seem to contradict the above expectations. Figure 1a shows a plot of QPO frequency vs. X-ray intensity for the transient source 4U 1608–52 (Méndez et al. 1999). Each segment there represents an uninterrupted observation lasting $`\mathrm{¡}\mathrm{}`$1 hour, whereas different tracks denote observations separated by intervals longer than a day. From this Figure it is apparent that frequency and X-ray count rates are positively correlated during relatively short periods, but they are uncorrelated over longer time intervals.

## 2. Results and Discussion

Despite 5 years of observations with RXTE, so far no single transition between two tracks in a QPO frequency vs. X-ray intensity diagram had been observed. Here I report the first direct observation of one such transition, in an RXTE observation of 4U 1636–53 made in 1998 (see Figure 1b). The observation starts with the source at the upper end of “Track 1” (“BEGIN”), and ends at the upper end of “Track 2” (“END”). In between, 4U 1636–53 goes three times from one track to the other, as indicated by the arrows with numbers 1, 2, and 3. The transitions are very fast ($`\mathrm{¡}\mathrm{}`$300 s), consistent with upper limits inferred from the time intervals between two consecutive tracks in this and other sources, in diagrams similar to the one shown in Figure 1a (e.g., Méndez 2000).

During the transitions in 4U 1636–53, X-ray intensity changes by $`\mathrm{¡}\mathrm{}`$3 %. From Figure 1a we see that in 4U 1608–52 X-ray intensity differs by factors of a few between tracks (in 4U 1636–53 jumps of $``$30 % are observed over longer time intervals than shown in Figure 1b), it is therefore not clear whether this larger differences in X-ray intensity are the cumulative effect of several small jumps, or if they have a completely different origin.

I will present these results in more detail elsewhere (Méndez & van der Klis, 2002, in preparation), and there I will discuss more extensively their implications upon models of the accretion flow in X-ray binaries.

## References

Ford, E., et al. 2000, ApJ, 537, 368

Méndez, M., et al. 1999, ApJ, 511, L49

Méndez, M. 2000, 19th. Texas Symp, 15/16

Miller, C., Lamb, F. K., & Psaltis, D. 1998, ApJ, 508, 791

van der Klis, M. 2000, ARA&A, 38, 717


text:
# Griffiths-Harris rigidity of compact Hermitian symmetric spaces

## 1. Introduction

Let $`X^{n+a}`$ be a variety and let $`xX`$ be a smooth point. The projective second fundamental form of $`X`$ at $`x`$ (see ) is a basic differential invariant that measures how $`X`$ is moving away from its embedded tangent projective space at $`x`$ to first order. It determines a system of quadrics $`|II_{X,x}|S^2T_x^{}X`$. Now let $`X`$ be a general point, to what extent does $`|II_{X,x}|`$ determine $`X`$?

Let $`X`$ be such that $`|II_{X,x}|`$ is an isolated point in the moduli space of $`a`$-dimensional linear subspaces of the space of quadratic forms on $`^n`$ up to linear equivalance. We say $`X`$ is infinitesimally rigid at order two or is Griffiths-Harris rigid if whenever $`Y^N`$ is a complex manifold, $`yY`$ is a general point and $`|II_{Y,y}|=|II_{X,x}|`$, then $`\overline{Y}=X`$.

In , Griffiths and Harris posed the question as to whether the Segre variety $`Seg(^2\times ^2)^8`$ was infinitesimally rigid to order two and in  I answered the question affirmatively and showed that all rank two compact Hermitian symmetric spaces (in their minimal homogeneous embeddings) except for the quadric hypersurface, the Grassmanian $`G(^2,^5)^9`$ and the spinor variety $`D_5/P_5=𝕊_{10}^{15}`$ were infinitesimally rigid at order two. The quadric is not rigid to order two and Fubini showed  it is rigid to order three when $`n>1`$ and it is rigid to order five when $`n=1`$. In this paper I resolve the two remaining cases, and explain shorter and less computational proofs for the other cases presented in . I also reprove the rigidity of the three Severi varieties that are rigid to order two to illustrate the method. The new proofs use two tools, a higher order Bertini theorem, and elementary representation theory.

In  we showed that all rational homogeneous varieties other than the rank two compact Hermitian symmetric spaces fail to be rigid to order two, so the result of this paper is the best possible in this sense. One can compare this type of rigidity to that studied by Hwang and Mok, see, . Some differences are: in their study they require global hypotheses where here the hypotheses are at the level of germs (this is because the systems of quadrics under study admit no local deformations); in their study the objects of interest are not a priori given an embedding (although since they assume the Picard number is one, one gets something close to an embedding); and in their study the object of interest is the cone of minimal degree rational curves through a general point, which, a priori, has nothing to do with the cone of asymptotic directions I use here (in the systems under consideration, the base locus of $`II`$ determines $`II`$).

###### Theorem 1.1.

Let $`X^n^{n+a}`$ be a complex submanifold. Let $`xX`$ be a general point. If $`|II_{X,x}||II_{Z,z}|`$ where $`Z`$ is a compact rank two Hermitian symmetric space in its natural embedding, other than a quadric hypersurface, then $`\overline{X}=Z`$.

Some open questions and relations with the Fulton-Hansen connectedness theorem are discussed in . Another application of the techniques used here is given in .

## 2. Moving frames

For more details throughout this section, see any of .

Once and for all fix index ranges $`1\alpha ,\beta ,\gamma n`$, $`n+1\mu ,\nu n+a`$.

Let $`X^n^{n+a}=V`$ be a complex submanifold and let $`xX`$ be a general point. Let $`\pi :^1\mathrm{}X`$ denote the bundle of bases of $`V`$ (frames) preserving the flag

$$\widehat{x}\widehat{T}_xXV.$$

Here $`\widehat{T}_xX`$ denotes the affine tangent space (the cone over the embedded tangent projective space). Let $`(e_0,\mathrm{},e_{n+a})`$ be a basis of $`V`$ with dual basis $`(e^0,\mathrm{},e^{n+a})`$ adapted such that $`e_0\widehat{x}`$ and $`\{e_0,e_\alpha \}`$ span $`\widehat{T}_xX`$. I ignore twists and obvious quotients, writing $`e_\alpha `$ for $`(e_\alpha \text{ mod }e_0)\mathrm{}e^0T_xX`$ and $`e_\mu `$ for $`(e_\mu \text{ mod }\widehat{T}_xX)\mathrm{}e^0N_xX=T_xV/T_xX`$. Moreover, if $`x`$ and $`X`$ understood, I write $`T=T_xX`$ and $`N=N_xX`$.

The fiber of $`\pi :^1\mathrm{}X`$ over a point is isomorphic to the group

$$G_1=\{g=\left(\begin{array}{ccc}g_0^0& g_\beta ^0& g_\nu ^0\\ 0& g_\beta ^\alpha & g_\nu ^\alpha \\ 0& 0& g_\nu ^\mu \end{array}\right)gGL(V)\}.$$

While $`^1`$ is not in general a Lie group, since $`^1GL(V)`$, we may pullback the Maurer-Cartan from on $`GL(V)`$ to $`^1`$. Write the pullback of the Maurer-Cartan form to $`^1`$ as

$$\omega =\left(\begin{array}{ccc}\omega _0^0& \omega _\beta ^0& \omega _\nu ^0\\ \omega _0^\alpha & \omega _\beta ^\alpha & \omega _\nu ^\alpha \\ \omega _0^\mu & \omega _\beta ^\mu & \omega _\nu ^\mu \end{array}\right).$$

The adaptation implies that $`\omega _0^\mu =0`$ and the Maurer-Cartan equation $`d\omega =\omega \omega `$ together with the Cartan Lemma implies that for all $`\mu ,\alpha `$, $`\omega _\alpha ^\mu =q_{\alpha \beta }^\mu \omega _0^\beta `$ for some functions $`q_{\alpha \beta }^\mu =q_{\beta \alpha }^\mu :^1\mathrm{}`$. These functions determine the projective second fundamental form $`II=F_2=\omega _\alpha ^\mu \mathrm{}e_\mu =q_{\alpha \beta }^\mu \omega _0^\alpha \omega _0^\beta \mathrm{}e_\mu \mathrm{\Gamma }(X,S^2T^{}X\mathrm{}NX)`$.

While $`II`$ descends to be a section of $`(S^2T^{}X\mathrm{}NX)`$, higher order derivatives provide relative differential invariants $`F_k\mathrm{\Gamma }(^1,\pi ^{}(S^kT^{}\mathrm{}N))`$. For example,

$`F_3`$ $`=r_{\alpha \beta \gamma }^\mu \omega _0^\alpha \omega _0^\beta \omega _0^\gamma \mathrm{}e_\mu `$
$`F_4`$ $`=r_{\alpha \beta \gamma \delta }^\mu \omega _0^\delta \omega _0^\alpha \omega _0^\beta \omega _0^\gamma \mathrm{}e_\mu `$
where the functions $`r_{\alpha \beta \gamma }^\mu ,r_{\alpha \beta \gamma \delta }^\mu `$ are given by

(1)
$$r_{\alpha \beta \gamma }^\mu \omega _0^\gamma =dq_{\alpha \beta }^\mu q_{\alpha \beta }^\mu \omega _0^0q_{\alpha \beta }^\nu \omega _\nu ^\mu +q_{\alpha \delta }^\mu \omega _\beta ^\delta +q_{\beta \delta }^\mu \omega _\alpha ^\delta $$

(2)
$$r_{\alpha \beta \gamma \delta }^\mu \omega _0^\delta =dr_{\alpha \beta \gamma }^\mu 2r_{\alpha \beta \gamma }^\mu \omega _0^0r_{\alpha \beta \gamma }^\nu \omega _\nu ^\mu +𝔖_{\alpha \beta \gamma }(r_{\alpha \beta ϵ}^\mu \omega _\gamma ^ϵ+3q_{\alpha \beta }^\mu \omega _\gamma ^0q_{\alpha ϵ}^\mu q_{\beta \gamma }^\nu \omega _\nu ^ϵ).$$

If one chooses local affine coordinates and writes $`X`$ as a graph

$$x^\mu =q_{\alpha \beta }^\mu x^\alpha x^\beta +r_{\alpha \beta \gamma }^\mu x^\alpha x^\beta x^\gamma +r_{\alpha \beta \gamma \delta }^\mu x^\alpha x^\beta x^\gamma x^\delta +\mathrm{}$$

then there exists a local section of $`^1`$ such that

$`F_2`$ $`=q_{\alpha \beta }^\mu dx^\alpha dx^\beta \mathrm{}{\displaystyle \frac{}{x^\mu }}`$
$`F_3`$ $`=r_{\alpha \beta \gamma }^\mu dx^\alpha dx^\beta dx^\gamma \mathrm{}{\displaystyle \frac{}{x^\mu }}`$
$`F_4`$ $`=r_{\alpha \beta \gamma \delta }^\mu dx^\alpha dx^\beta dx^\gamma dx^\delta \mathrm{}{\displaystyle \frac{}{x^\mu }}`$
etc…

Since an analytic variety is uniquely determined by its Taylor series at a point, to show $`Z`$ is rigid to order two, it is sufficient to show that over varieties $`X`$ with $`|II_{X,x}|=|II_{Z,z}|`$ there exists a subbundle of $`^1`$ such that the $`F_k`$’s of $`X`$ coincide with those of $`Z`$. Moreover, the minuscule varieties, that is, the compact Hermitian symmetric spaces in their natural projective embeddings, have the property that on a reduced frame bundle all the differential invariants except for their fundamental forms are zero and in our case the only nonzero fundamental form is $`II`$.

The method in  was first to use the equations above to calculate relations among the coefficients of $`F_3`$. Enough relations were found that, combined with the coefficients that were normalizable to zero, I obtained that $`F_3`$ was zero, and the same technique was used for higher order invariants.

In this paper I decompose $`S^3T^{}\mathrm{}N`$ into irreducible $`R`$-modules, where $`RGL(T)\times GL(N)`$ is the subgroup preserving $`IIS^2T^{}\mathrm{}N`$. I also systematize the vanishing of coefficients of the $`F_k`$ somewhat using higher order Bertini theorems which I now describe. It would be nice to have a way to apply the higher order Bertini theorems directly to the irreducible modules instead of using the coefficients of $`F_3`$.

## 3. Vanishing tools

### 3.1. Higher order Bertini

Let $`T`$ be a vector space. The classical Bertini theorem implies that for a linear subspace $`AS^2T^{}`$, if $`qA`$ is generic, then $`vSing(q):=\{vTq(v,w)=0wT\}`$ implies $`vBase(A):=\{vTQ(v,v)=0QA\}`$.

###### Theorem 3.1 (Higher order Bertini).

Let $`X^nV`$ be a complex manifold and let $`xX`$ be a general point.

1. Let $`q|II_{X,x}|`$ be a generic quadric. Then $`q_{sing}\text{Base}\{F_2,\mathrm{},F_k\}`$ for all $`k`$. I.e., $`q_{sing}`$ is tangent to a linear space on the completion of $`X`$.

2. Let $`q|II_{X,x}|`$ be a any quadric, let $`Lq_{sing}\text{Base}|II_{X,x}|`$ be a linear subspace. Then for all $`v,wL`$, $`F_3^q(v,w,)=0`$.

3. With $`L`$ as in 2., if $`L^{}(\text{Base}\{|II_{X,x}|,F_3\})L`$ is a linear subspace then $`F_4^q(u,v,w,)=0`$ for all $`u,v,wL^{}`$ and so on for higher orders. Here $`F_k^q`$ denotes the polynomial in $`F_k`$ corresponding to the conormal direction of $`q`$. This is well defined by the lower order vanishing.

4. With $`L^{}`$ as in 3., if $`L^{\prime \prime }L^{}(F_3^q)_{sing}`$ is a linear space, then for all $`u,vL^{\prime \prime }`$, $`F_4^q(u,v,,)=0`$.

Analogous results hold for higher orders.

###### Proof.

Note that 1. is classical, but we provide a proof for completeness. Assume $`v=\underset{¯}{e}_1`$ and $`q=q^\mu `$. Our hypotheses imply $`q_{1\beta }^\mu =0`$ for all $`\beta `$. Formula (1) reduces to

$$r_{11\beta }^\mu \omega _0^\beta =q_{11}^\nu \omega _\nu ^\mu .$$

If $`q`$ is generic we are still working on $`^1`$ and so the $`\omega _\nu ^\mu `$ are independent of the semi-basic forms, thus the coefficients on both sides of the equality are zero, proving both the classical Bertini theorem and 1 in the case $`k=3`$.

If $`q`$ is not generic, in order to have $`q=q^\mu ,v=e_1`$ we have reduced to a subbundle $`^{}^1`$ and we no longer have the $`\omega _\nu ^\mu `$ independent. However hypothesis 2 states that $`q_{11}^\nu =0`$ for all $`\nu `$ and the required vanishing still holds.

For 3., note that $`r_{111\delta }^\mu \omega _0^\delta =r_{111}^\nu \omega _\nu ^\mu +er_{11ϵ}^\mu \omega _0^ϵ1+q_{1ϵ}^\mu q_{11}^\nu `$ and the right hand side is zero under our hypotheses. Part 4 is proven simliarly.

The extension to linear spaces holds by polarizing the forms. The analogous equation at each order proves the next higher order. ∎

###### Example 3.2.

Let $`X=G(2,m)`$ and let $`V=\mathrm{\Lambda }^2^m`$ have basis $`e_{st}`$ with $`1s<tm`$. At $`x=[e_{12}]`$ we have the adapted flag

$$\{e_{12}\}\{e_{1j},e_{2j}\}V$$

where $`3i<jm`$, and $`SL_2\times SL_{m2}`$ acts transitively on $`N_x\{e_{ij}\}`$. So here $`\alpha =\{(1j),(2,j)\}`$, $`\mu =\{(ij)\}`$. In these frames $`II=(\omega _0^{(1i)}\omega _0^{(2j)}\omega _0^{(1j)}\omega _0^{(2i)})\mathrm{}e_{ij}`$.

If $`m=5`$, then $`q^{45}`$ is a generic quadric with $`e_{13}q_{sing}^{45}`$. Thus we have

$`r_{(13)(13)(13)}^\mu `$ $`=0\mu `$
$`r_{(13)(13)\beta }^{45}`$ $`=0\beta .`$
If $`m>5`$ then $`q^{45}`$ is no longer generic, but since $`e_{(13)}\text{Base}|II_{X,x}|`$ we still may conclude

$$r_{(13)(13)\beta }^{45}=0\beta .$$

### 3.2. Normalizations

$`F_3`$ is translated in the fiber of $`^1`$ by the action of $`T\mathrm{}N^{}`$ and $`T^{}`$ (the $`g_\mu ^\alpha `$ and the $`g_\alpha ^0`$). We may decompose $`T\mathrm{}N^{}`$ and $`T^{}`$ into irreducible $`R`$ modules and determine which of these act nontrivially. In the case the variety is modeled on a rank two minuscule variety, we will have that all of $`T\mathrm{}N^{}`$ acts effectively, but the $`T^{}`$ action duplicates a factor in $`T\mathrm{}N^{}`$. This is because in the homogeneous model, the forms $`\omega _\beta ^0`$ are independent and the forms $`\omega _\mu ^\alpha `$ are linear combinations of the $`\omega _\beta ^0`$. We will let $`^n`$ denote the bundle where the action of $`T\mathrm{}N^{}`$ has been used to kill the corresponding components of $`F_3`$. Similarly, on $`^n`$, $`F_4`$ is translated by the action of $`N`$ and we will let $`^N`$ denote the subbundle of $`^n`$ where the component of $`N`$ in $`F_4`$ has been normalized to zero.

### 3.3. Remarks on decompositions of the $`F_k`$ and vanishing

Let $`IIS^2T^{}\mathrm{}N`$ arise from a trivial representation of a reductive group $`RGL(T)\times GL(N)`$. Let $`X^n^{n+a}`$ be a complex submanifold, let $`xX`$ be a general point and suppose $`II_{X,x}=II`$.

1. Since the oribit of a highest weight vector in any module spans the module, the component of $`F_k`$ in an irreducible module $`V`$ is zero if its highest weight vector is zero.

2. An irreducible module in $`S^3T^{}\mathrm{}N`$ can occur in $`F_3`$ only if it also occurs in $`(T\mathrm{}T^{})^{𝔯^c}\mathrm{}T^{}+(N\mathrm{}N^{})^{𝔯^c}\mathrm{}T^{}`$. Here $`𝔯`$, the Lie algebra of $`R`$, occurs as a submodule of $`T\mathrm{}T^{}`$ and $`N\mathrm{}N^{}`$ and $`(T\mathrm{}T^{})^{𝔯^c}`$ denotes the complement of $`𝔯`$ in $`T\mathrm{}T^{}`$. This is because the tangential and normal connection forms, $`\omega _\beta ^\alpha ,\omega _\nu ^\mu `$ may be decomposed into $`\rho _T(𝔯)`$-valued (resp. $`\rho _N(𝔯)`$-valued) forms and semi-basic forms. (Here $`\rho _T,\rho _N`$ denote the representations of $`𝔯`$ on $`T`$ and $`N`$.) The coefficients for the semi-basic components are linear combinations of the $`r_{\alpha \beta \gamma }^\mu `$, all of the same weight. On the other hand the coefficients give rise to $`R`$-modules respectively in $`(T\mathrm{}T^{})^{𝔯^c}\mathrm{}T^{}`$ and $`(N\mathrm{}N^{})^{𝔯^c}\mathrm{}T^{}`$.

3. Similarly, if $`F_3=0`$, and the normalizations of $`F_3`$ are exactly by $`T\mathrm{}N^{}`$, then an irreducible module in $`S^4T^{}\mathrm{}N`$ can occur in $`F_4`$ only if it also occurs in $`(T\mathrm{}N^{})^{T^c}\mathrm{}T^{}`$, where $`(T\mathrm{}N^{})^{T^c}`$ is the complement of $`T^{}`$ in $`T\mathrm{}N^{}`$. Thus in our normalizations, the forms $`\omega _\beta ^0`$ will remain independent and independent of the semi-basic forms. On the other hand the forms $`\omega _\nu ^ϵ`$ will become dependent on the semi-basic forms and the $`\omega _\beta ^0`$. Again, the components that will depend on the semi-basic forms will have coefficients consisting of linear combinations of monomials in $`F_4`$ of the same weight.

4. If $`F_3,F_4=0`$ (after normalizations), then an irreducible module in $`S^5T^{}\mathrm{}N`$ can occur in $`F_4`$ only if it also occurs in $`N`$.

5. If $`F_3,F_4,F_5=0`$ (after normalizations), then all higher $`F_k`$ are zero as well, see .

## 4. Case of $`G(2,5)`$ and $`𝕊_{10}`$

### 4.1. Model for $`G(2,5)`$

Write $`T=A^{}\mathrm{}B`$. We index bases of $`T`$ and $`N`$ as above. $`R=𝔰𝔩(A)+𝔰𝔩(B)+=𝔰𝔩_2+𝔰𝔩_3+`$. We write $`A_j`$ for the represention of $`𝔰𝔩(A)`$ with highest weight $`j`$ and $`B_{ij}`$ for the representation of $`𝔰𝔩(B)`$ of highest weight $`i\omega _1+j\omega _2`$. Here and throughout we use the notations and ordering of the weights of . The relevant modules are summarized in the table below.

### 4.2. Model for $`𝕊_{10}`$

Write $`^{16}=Clifford(^5)\mathrm{\Lambda }^{even}^5`$ with $`\widehat{x}\mathrm{\Lambda }^0W`$, $`T\mathrm{\Lambda }^2W,N\mathrm{\Lambda }^4WW^{}`$. We let $`e_{st}=e_se_t`$, $`1s<t5`$ index a basis of $`T`$ and $`e^s`$ index a basis of $`N`$. Note that, as with $`G(2,5)`$, $`R`$ acts transitively on $`N`$ so all quadrics in $`|II|`$ are generic.

Let $`V_{ijkl}`$ denote the $`𝔰𝔩_5`$ module with highest weight $`i\omega _1+j\omega _2+k\omega _3+l\omega _4`$. $`|II|`$ is given by the Pfaffians of the $`4\times 4`$ minors centered about the diagonal with $`e^j`$ corresponding to the Pfaffian obtained by removing the $`j`$-th row and column. The relevant modules are summarized in the following table.

$`T`$ $`=V_{0100}`$
$`=A_1\mathrm{}B_{10}`$
$`T^{}`$ $`=V_{0010}`$
$`=A_1\mathrm{}B_{01}`$
$`N`$ $`=V_{0001}`$
$`=A_0\mathrm{}B_{10}`$
$`S^2T^{}`$ $`=T^2\mathrm{}N^{}`$
$`S^3T^{}`$ $`=T^3\mathrm{}N^{}T^{}`$
$`S^3T^{}\mathrm{}N`$ $`=(T^3N\mathrm{}TT^2)\mathrm{}((N^{}T^{})N\mathrm{}N^{}T\mathrm{}T^{})`$
$`T\mathrm{}N^{}`$ $`=TN^{}\mathrm{}T^{}`$
$`(T\mathrm{}N^{})^{T^c}\mathrm{}T^{}`$ $`=N^{}TT^{}\mathrm{}N^2N\mathrm{}TN\mathrm{}N^{}`$
$`S^4T^{}`$ $`=T^4\mathrm{}N^{}T^2\mathrm{}N^2`$
$`S^4T^{}\mathrm{}N`$ $`=(T^5\mathrm{}TT^3N\mathrm{}N^{}T^3)\mathrm{}(N^{}T^3\mathrm{}N^{}TT^{}N\mathrm{}T^2N\mathrm{}N^2T^{}\mathrm{}TT^{})`$
$`\mathrm{}(N^2N\mathrm{}N^{})`$
The notation is such that if $`V_\lambda ,W_\mu `$ are the irreducible representations with highest weights $`\lambda ,\mu `$, then $`V^k`$, $`VW`$ are respectively the representations with highest weights $`k\lambda `$ and $`\lambda +\mu `$. $`VWV\mathrm{}W`$ is called the Cartan component.

To obtain the vanishing of $`F_3`$ we need to eliminate five modules. We first eliminate two by reducing to $`^n`$ as described above, so the last two factors are zero. Let $`^n^1`$ denote our new frame bundle.

On our new bundle there remains three modules to eliminate.

The first module in $`S^3T^{}`$ is decomposably generated by $`r_{(13)(13)(13)}`$ in the $`G(2,5)`$ case and $`r_{(12)(12)(12)}`$ in the $`𝕊_{10}`$ case. We already saw the $`G(2,5)`$ case is covered by Bertini, and the $`𝕊_{10}`$ case is as well because $`e_{(12)}q_{sing}^1`$, and all quadrics in the system are generic. Thus the first two modules in $`S^3T^{}\mathrm{}N`$ don’t appear in $`F_3`$.

At this point just $`(N^{}T^{})N`$ remains. In the $`G(2,5)`$ case $`(N^{}T^{})`$ has highest weight a linear combination of $`r_{(13)(13)(24)}`$ and $`r_{(13)(14)(23)}`$. In the $`𝕊_{10}`$ case it has highest weight a linear combination of $`r_{(12)(12)(34)}`$,$`r_{(12)(13)(24)}`$ and $`r_{(12)(14)(23)}`$ and thus the Cartan components respectively have highest weights linear combinations of $`r_{(13)(13)(24)}^{45}`$, $`r_{(13)(14)(23)}^{45}`$ and $`r_{(12)(12)(34)}^5`$, $`r_{(12)(13)(24)}^5`$ and $`r_{(12)(14)(23)}^5`$, all of which are zero by Bertini. Thus $`F_3`$ is zero.

We normalize away the $`N`$ factor in $`S^4T^{}\mathrm{}N`$ and study the remaining modules. Comparing $`S^4T^{}\mathrm{}N`$ and $`(T\mathrm{}N^{})^{T^c}\mathrm{}T^{}`$ modulo $`N^{}`$, their intersection is empty and thus $`F_4=0`$ on $`^N`$.

One can check that $`S^5T^{}\mathrm{}N`$ does not contain a copy of $`N`$, so we are done.∎

###### Remark 4.1.

If one compares modules, $`N^{}T^{}N`$ is not eliminated from $`F_3`$. On the other hand, if one just uses Bertini to study $`F_4`$, everthing coming from the first factor in $`S^4T^{}`$ and all Cartan products of $`N`$ with factors in $`S^4T^{}`$ are easily seen to be zero, but a few of the other modules are more complicated to eliminate.

## 5. Case of $`𝔸^2`$

Let $`𝔸_{}`$ respectively denote $`,,𝕆`$ and let $`𝔸=𝔸_{}\mathrm{}_{}`$.

I use the following model: $`T=𝔸\mathrm{}𝔸`$, where $`𝔸=`$, the complexified quaternions, for $`G(2,6)`$, and $`𝔸=𝕆`$, the complexified octonions for $`𝕆^2`$. I use $`(a,b)`$ as $`𝔸`$-valued coordinates. Then $`|II|=\{a\overline{a},b\overline{b},a\overline{b}\}`$ where $`a\overline{b}`$ represents $`\mathrm{dim}𝔸`$ quadrics. Let $`p=3,7`$. Write $`a=a_0+a_1ϵ_1+\mathrm{}+a_pϵ_p`$. We will need to work with null vectors so let $`e_1=1+iϵ_1`$, $`\overline{e}_1=1iϵ_1`$, $`e_2=1+iϵ_2`$ denote elements of the first copy of $`𝔸`$ (with coordinate $`a`$). We let $`e_a`$ denote the normal vector such that $`q^a=a\overline{a}`$ and similarly for $`e_b`$. Let $`e_0`$ denote the normal vector such that $`q^0=Re(a\overline{b})`$ and $`e_{ϵ_j}`$ such that $`q^{ϵ_j}`$ is the $`ϵ_j`$ coefficient of $`a\overline{b}`$.

Let $`V_{ijklm}`$ denote the $`𝔡_5`$-module with highest weight $`i\omega _1+j\omega _j+k\omega _3+l\omega _4+m\omega _5`$, and the $`𝔰𝔩(A)+𝔰𝔩(B)`$ modules are indexed in the obvious way. For the Segre case write $`T=U_{10}\mathrm{}W_{10}`$ and $`N=U_{01}\mathrm{}W_{01}`$. The remaining relevant modules are as follows:

$`T`$ $`=V_{00001}`$
$`=A_1\mathrm{}B_{100}`$
$`T^{}`$ $`=V_{00010}`$
$`=A_1\mathrm{}B_{001}`$
$`N`$ $`=N^{}=V_{10000}`$
$`=A_0\mathrm{}B_{010}`$
$`S^2T^{}`$ $`=T^2\mathrm{}N`$
$`S^3T^{}`$ $`=T^3\mathrm{}NT^{}`$
$`S^3T^{}\mathrm{}N`$ $`=(T^3N\mathrm{}T^2T)\mathrm{}(N^2T^{}\mathrm{}𝔤T^{}\mathrm{}NT\mathrm{}T^{})`$
$`T\mathrm{}N^{}`$ $`=TN^{}\mathrm{}T^{}`$
$`(N\mathrm{}N^{})^{𝔯^c}\mathrm{}T^{}`$ $`=N^2T^{}\mathrm{}NT`$
$`(T\mathrm{}T^{})^{𝔯^c}\mathrm{}T^{}`$ $`=T^2T\mathrm{}T_2T\mathrm{}𝔤T^{}\mathrm{}NT\mathrm{}T^{}`$
See  for an explanation of $`T_2`$.

The decompositions above show that there are $`6`$ components of $`F_3`$ on $`^1`$ and four when we restrict to $`^n`$.

We may choose our model such that $`e_1`$ is a highest weight vector (since it is in $`\text{Base}|II_{X,x}|`$). We may also have $`e^b`$ be a highest weight vector for $`N^{}`$.

Bertini easily kills the first component of $`F_3`$ as it has highest weight vector $`r_{111}^b`$. In fact, as in the cases above, the second Cartan component is also killed by Bertini. To see this note that the two irreducible components of $`S^3T^{}`$ respectively have highest weight vectors $`r_{111}`$ and a linear combination of $`r_{11\overline{1}}`$ and $`r_{12\overline{2}}`$. The two Cartan components in $`S^3T^{}\mathrm{}N`$ thus have highest weight vectors $`r_{111}^b`$ and a linear combination of $`r_{11\overline{1}}^b`$ and $`r_{12\overline{2}}^b`$. To see the second is zero, note that any linear combination of $`e_1,e_2`$ is in $`\text{Base}|II_{X,x}|`$ and $`\text{Sing}q^b`$.

It remains to eliminate the second and fourth modules in $`S^3T^{}\mathrm{}N`$, but neither of these occurs in $`(T\mathrm{}T^{})^{𝔯^c}\mathrm{}T^{}\mathrm{}(N\mathrm{}N^{})^{𝔯^c}\mathrm{}T^{}`$ and we are done with $`F_3`$.

The higher order invariants are safely left to the reader. ∎

To compare with the $`G(2,6)`$ case in the standard model, we have $`e_1=e_{(13)}`$,$`e_{\overline{1}}=e_{(24)}`$, $`q^a=q^{34}`$,$`q^b=q^{56}`$ etc….