format: text

text:
# Replica symmetric evaluation of the information transfer in a two-layer network in presence of continuous and discrete stimuli

## I Introduction

Recent analyses of extracellular recordings performed in two motor areas of behaving monkeys have tried to clarify how information about movements is trasmitted and received from higher to lower stages of processing, and to identify distinct roles of the two areas in the planning and execution of movements . Although this study failed to produce clearcut results, it remains interesting to try and understand, from a more theoretical point of view, how information about multi-dimensional correlates of neural activity may be transmitted from the input to the output of a simple network. In fact, a theoretical study is still lacking, which explores how the coding of stimuli with continuous as well as discrete dimensions is transferred across a network.

Information theory  has been widely used in the theory of communication, in presence of both binary  and linear  or weakly non-linear  channels. Moreover it has been recently proposed as an effective tool to explore the coding properties of neurons (see for example ), via both direct estimates from real data (for a review see ) and pure theoretical modelling .

The mutual information provides a quantitative and flexible measure of the efficiency of single cells or of population of cells in coding external stimuli and events relevant to behaviour: high values of the mutual information are obtained when the correlates can be discriminated with a small uncertainty on the basis of the neural responses; moreover the same formalism can be adapted to explore different types of code, from simple time-averaged rates, to more sophisticated descriptions, where the exact temporal sequence of action potentials is considered to be relevant.

In a previous report  the mutual information between the time averaged rates of a finite population of $`N`$ units and a set of correlates, which have both a discrete and a continuous angular dimension, has been evaluated analytically in the limit of large noise. This parameterization of the correlates can be applied to movements performed in a given direction and classified according to different ”types”; yet it is equally applicable to other correlates, like visual stimuli characterized by an orientation and a discrete feature (colour, shape, etc..), or in general to any correlate which can be identified by an angle and a ”type”. In this study, we extend the analysis performed for one population, to consider two interconnected areas, and we evaluate the mutual information between the firing rates of a finite population of $`M`$ output neurons and a set of continuous+discrete stimuli, given that the rate distribution in input is known. In input, a threshold nonlinearity has been shown to lower the information about the stimuli in a simple manner, which can be expressed as a renormalization of the noise . How does the information in the output depend on the same nonlinearity? How does it depend on the noise in the output units? Is the information transmission from input to output sensitive to the structure of the correlate, whether discrete or continuous?

We address these issues by calculating the mutual information, using the replica trick and under the assumption of replica symmetry (see for example ).

Saddle point equations are solved numerically. We analyze how the information trasmission depends on the parameters of the model, i.e. the level of output and input noise, on the ratio between the two population sizes, as well as on the tuning curve with respect to the continuous correlate, and on number of discrete correlates.

The input-output transfer function is a crucial element in the model. Many earlier theoretical and simulation studies  have mainly focused on binary and the sigmoidal functions; yet more recent investigations  have shown that the current-to-frequency transduction typical of real neurons is well captured, away from saturation, by a threshold-linear function. Such a function combines the threshold of real neurons, the linear behaviour typical of pyramidal neurons above threshold, and the accessibility to a full analytical treatment , as demonstrated here, too. For the sake of analytical feasibility, however, we take the input units to be purely gaussian. Therefore it should be kept in mind, in considering the final results, that the threshold nonlinearity is only applied to the output units.

## II The model

In analogy to the model studied in  we consider a set of $`N`$ input units which fire to an external continuous+discrete stimulus, parameterized by an angle $`\vartheta `$ and a discrete variable $`s`$, with a gaussian distribution:

$$p(\{\eta _j\}|\vartheta ,s)=\underset{j=1}{\overset{N}{}}\frac{1}{\sqrt{2\pi \sigma ^2}}exp\left[\left(\eta _j\stackrel{~}{\eta }_j(\vartheta ,s)\right)^2/2\sigma ^2\right];$$
(1)
$`\eta _j`$ is the firing rate in one trial of the $`j^{th}`$ input neuron, while the mean of the distribution, $`\stackrel{~}{\eta }_j(\vartheta ,s)`$ is written:

$$\stackrel{~}{\eta }_j(\vartheta ,s)=\epsilon _s^j\overline{\eta }_j(\vartheta )+(1\epsilon _s^j)\eta ^f;$$
(2)
$$\overline{\eta }_j(\vartheta \vartheta _j^0)=\eta ^0\mathrm{cos}^{2m}\left(\frac{\vartheta \vartheta _j^0}{2}\right);$$
(3)
where $`\epsilon _s^j`$ is a quenched random variable distributed between $`0`$ and $`1`$, $`\vartheta _i^0`$ is the preferred direction for neuron $`i`$. According to eq.(2) neurons fire at an average firing rate which modulates with $`\vartheta `$ with amplitude $`\epsilon _s`$, or takes a fixed value $`\eta ^f`$, independently of $`\vartheta `$, with amplitude $`1\epsilon _s`$.

We assume that quenched disorder is uncorrelated and identically distributed across units and across the $`K`$ discrete correlates, and that for each neuron all preferred directions are equally likely:

$$\varrho (\{\epsilon _s^i\})=\underset{i,s}{}\varrho (\epsilon _s^i)=\left[\varrho (\epsilon )\right]^{NK}$$
(4)
$$\varrho (\{\vartheta _i^0\})=\left[\varrho (\vartheta ^0)\right]^N=\frac{1}{(2\pi )^N}.$$

In  it has been shown that a cosinusoidal shaped function as in eq.(2) is able to capture the main features of directional tuning of real neurons in motor cortex. Moreover it has been shown that the presence of negative firing rates in the distribution (1), which is not biologically plausible, does not alter information values, with respect to a more realistic choice for the firing distribution, in that it leads to the same curves except for a renormalization of the noise.

Output neurons are connected to input neurons via uncorrelated gaussian connection weights $`J_{ij}`$. Each output neuron performs a linear summation of the inputs; the outcome is distorted by a gaussian distributed fast noise $`\delta _i`$ and then thresholded, as in the following:

$$\xi _i=[\xi _i^0+\underset{j}{}c_{ij}J_{ij}\eta _j+\delta _i]^+;i=1..M,j=1..N$$
(5)
In eq.(5)$`\xi _i^0`$ is a threshold term, $`c_{ij}`$ is a (0,1) binary variable, with mean $`c`$, which expresses the sparsity or dilution of the connectivity matrix, and

$$(J_{ij})^2=\sigma _J^2;J_{ij}=0;$$
(6)
$$(\delta _i)^2=\sigma _\delta ^2;\delta _i=0;$$
(7)
$`p(c_{ij}=1)`$ $`=`$ $`c;`$
$`p(c_{ij}=0)`$ $`=`$ $`1c;`$ (8)
$$[x]^+=x\mathrm{\Theta }(x).$$
(9)
## III Analytical estimation of the mutual information

We aim at estimating the mutual information between the output patterns of activity and the continuous+discrete stimuli:

$$I(\{\xi _i\},\vartheta s)=\underset{s=1}{\overset{K}{}}𝑑\vartheta \underset{i}{}d\xi _ip(\vartheta ,s)p(\{\xi _i\}|\vartheta ,s)\mathrm{log}_2\frac{p(\{\xi _i\}|\vartheta ,s)}{p(\{\xi _i\})}_{\epsilon ,\vartheta ^0,c,J,\delta };$$
(10)
$$p(\{\xi _i\}|\vartheta ,s)=\underset{j}{}d\eta _jp(\{\xi _i\}|\{\eta _j\})p(\{\eta _j\}|\vartheta ,s);$$
(11)
where the distribution $`p(\{\xi _i\}|\{\eta _j\})`$ is determined by the threshold linear relationship (5), $`p(\{\eta _j\}|\vartheta ,s)`$ is given in eq.(1) and $`.._{\epsilon ,\vartheta ^0,c,J,\delta }`$ is a short notation for the average across the quenched variables $`\{\epsilon _s^i\}`$,$`\{\vartheta _i^0\}`$,$`\{J_{ij}\}`$,$`\{c_{ij}\}`$ and on the fast noise $`\{\delta _i\}`$.

Contrary to the other quenched variables, $`\{\epsilon _s^i\}`$,$`\{\vartheta _i^0\}`$,$`\{J_{ij}\}`$,$`\{c_{ij}\}`$, the variable $`\delta _i`$ in eq.(5) is annealed: integration of relationships (5) across a zero mean gaussian distribution of $`\delta _i`$ with variance $`\sigma _\delta ^2`$ yields a gaussian distribution in $`\xi _i`$ with variance $`\sigma _\delta ^2`$. We assume that the stimuli are equally likely: $`p(\vartheta ,s)=1/2\pi K`$.

Eq.(10) can be written as:

$$I(\{\xi _i\},\vartheta s)=H(\{\xi _i\}|\vartheta ,s)_{\vartheta ,s}H(\{\xi _i\})$$
(12)
with:

$$H(\{\xi _i\}|\vartheta ,s)_{\vartheta ,s}=\underset{s}{}𝑑\vartheta \underset{i}{}d\xi _ip(\vartheta ,s)p(\{\xi _i\}|\vartheta ,s)\mathrm{log}_2p(\{\xi _i\}|\vartheta ,s)_{\epsilon ,\vartheta ^0,c,J,\delta };$$
(13)
$`H(\{\xi _i\})={\displaystyle \underset{s}{}}{\displaystyle }d\vartheta {\displaystyle }{\displaystyle \underset{i}{}}d\xi _ip(\vartheta ,s)p(\{\xi _i\}|\vartheta ,s)`$
$`\mathrm{log}_2\left[{\displaystyle \underset{s^{}}{}}{\displaystyle }d\vartheta ^{}p(s^{},\vartheta ^{})p(\{\xi _i\}|\vartheta ^{},s^{})\right]_{\epsilon ,\vartheta ^0,c,J,\delta }.`$ (14)
The analytical evaluation of $`H(\{\xi _i\}|\vartheta ,s)_{\vartheta ,s}`$ can be performed inserting eq.(11) in the expression (13), and using the replica trick  to get rid of the sums under logarithm; since these sums already multiply the logarithm, all replica indexes run from $`1`$ up to $`n+1`$:

$`H(\{\xi _i\}|\vartheta ,s)_{\vartheta ,s}=`$
$`\underset{n0}{lim}{\displaystyle \frac{1}{n\mathrm{ln}2}}\left({\displaystyle \underset{s}{}}{\displaystyle 𝑑\vartheta p(\vartheta ,s)\underset{j,\alpha }{}d\eta _j^\alpha \underset{j,\alpha }{}p(\eta _j^\alpha |\vartheta ,s)\underset{i}{}d\xi _i\underset{j,\alpha }{}p(\xi _i|\{\eta _j^\alpha \})}_{\epsilon ,\vartheta ^0,c,J,\delta }1\right).`$ (15)
To take into account the threshold-linear relation (5) we consider the following equalities:

$`{\displaystyle 𝑑\xi _i\underset{\alpha }{}p(\xi _i|\{\eta _j^\alpha \})}={\displaystyle \underset{\alpha }{}}p(\xi _i=0|\{\eta _i^\alpha \})+{\displaystyle _0^{\mathrm{}}}𝑑\xi _i{\displaystyle \underset{\alpha }{}}p(\xi _i|\{\eta _i^\alpha \})=`$
$`{\displaystyle _{\mathrm{}}^0}{\displaystyle \underset{\alpha }{}}d\xi _i^\alpha \delta (\xi _i^\alpha \xi _i^0{\displaystyle \underset{j}{}}c_{ij}J_{ij}\eta _j^\alpha \delta _i^\alpha )+{\displaystyle _0^{\mathrm{}}}𝑑\xi _i{\displaystyle \underset{\alpha }{}}\delta (\xi _i\xi _i^0{\displaystyle \underset{j}{}}c_{ij}J_{ij}\eta _j^\alpha \delta _i^\alpha ).`$ (16)
Inserting eq.(16) in eq.(15) one obtains:

$`H(\{\xi _i\}|\vartheta ,s)_{\vartheta ,s}=\underset{n0}{lim}{\displaystyle \frac{1}{n\mathrm{ln}2}}({\displaystyle \underset{s}{}}{\displaystyle }d\vartheta p(\vartheta ,s){\displaystyle }{\displaystyle \underset{j,\alpha }{}}d\eta _j^\alpha {\displaystyle \underset{j,\alpha }{}}p(\eta _j^\alpha |\vartheta ,s)_{\epsilon ,\vartheta ^0}`$
$`{\displaystyle \underset{i}{}}[{\displaystyle _{\mathrm{}}^0}{\displaystyle \underset{\alpha }{}}d\xi _i^\alpha {\displaystyle \underset{\alpha }{}}\delta (\xi _i^\alpha \xi _i^0{\displaystyle \underset{j}{}}c_{ij}J_{ij}\eta _j^\alpha \delta _i^\alpha )_{c,J,\delta }`$
$`+{\displaystyle _0^{\mathrm{}}}d\xi _i{\displaystyle \underset{\alpha }{}}\delta (\xi _i\xi _i^0{\displaystyle \underset{j}{}}c_{ij}J_{ij}\eta _j^\alpha \delta _i^\alpha )_{c,J,\delta }]1).`$ (17)
The average across the quenched disorder $`c`$,$`J`$,$`\delta `$ in eq.(17) can be performed in a very similar way as shown in : using the integral representation for each $`\delta `$ function, gaussian integration across $`J`$,$`\delta `$ is standard; the average on $`\{c_{ij}\}`$ can be performed assuming large the number $`N`$ of input neurons, that is for very small $`c`$.

The final outcome for $`H(\{\xi _i\}|\vartheta ,s)_{\vartheta ,s}`$ reads:

$`H(\{\xi _i\}|\vartheta ,s)_{\vartheta ,s}=\underset{n0}{lim}{\displaystyle \frac{1}{n\mathrm{ln}2}}({\displaystyle \underset{s}{}}{\displaystyle }d\vartheta p(\vartheta ,s){\displaystyle }{\displaystyle \underset{j,\alpha }{}}d\eta _j^\alpha {\displaystyle \underset{j,\alpha }{}}p(\eta _j^\alpha |\vartheta ,s)_{\epsilon ,\vartheta ^0}`$
$`[{\displaystyle _{\mathrm{}}^0}{\displaystyle \underset{\alpha }{}}d\xi ^\alpha {\displaystyle }{\displaystyle \underset{\alpha }{}}{\displaystyle \frac{dx^\alpha }{2\pi }}e^{(\sigma _\delta ^2/2)_\alpha (x^\alpha )^2}e^{(C\sigma _J^2/2N)_{\alpha ,\beta }x^\alpha x^\beta _j\eta _j^\alpha \eta _j^\beta }e^{i_\alpha (\xi ^\alpha \xi _0)x^\alpha }`$
$`+{\displaystyle _0^{\mathrm{}}}d\xi {\displaystyle }{\displaystyle \underset{\alpha }{}}{\displaystyle \frac{dx^\alpha }{2\pi }}e^{(\sigma _\delta ^2/2)_\alpha (x^\alpha )^2}e^{(C\sigma _J^2/2N)_{\alpha ,\beta }x^\alpha x^\beta _j\eta _j^\alpha \eta _j^\beta }e^{i(\xi \xi _0)_\alpha x^\alpha }]^M1),`$ (18)
where we have put $`cC/N`$.

Integration on $`\{x^\alpha \}`$ is straightforward. Integration on $`\{\eta _i^\alpha \}`$ can be performed introducing $`(n+1)^2`$ auxiliary variables $`z_{\alpha \beta }=\frac{1}{N}_j\eta _j^\alpha \eta _j^\beta `$ via $`\delta `$ functions expressed in their integral representation. Considering the expression (1) for the input distribution and with some rearrangement of the terms the final result can be expressed as:

$`H(\{\xi _i\}|\vartheta ,s)_{\vartheta ,s}=`$ (19)
$`\underset{n0}{lim}{\displaystyle \frac{1}{n\mathrm{ln}2}}({\displaystyle }{\displaystyle \underset{\alpha ,\beta }{}}{\displaystyle \frac{dz_{\alpha \beta }}{2\pi /N}}{\displaystyle }{\displaystyle \underset{\alpha ,\beta }{}}d\stackrel{~}{z}_{\alpha \beta }e^{iN_{\alpha \beta }z_{\alpha \beta }\stackrel{~}{z}_{\alpha \beta }}e^{\frac{N}{2}Tr\mathrm{ln}\mathrm{\Sigma }}{\displaystyle \underset{s}{}}{\displaystyle }d\vartheta p(\vartheta ,s)e^{_{\alpha ,\beta }(\delta _{\alpha \beta }\mathrm{\Sigma }_{\alpha \beta }^1)\stackrel{~}{\eta }(\vartheta ,s)^2/2\sigma ^2}_{\epsilon ,\vartheta ^0}^N`$
$`e^{\frac{M}{2}Tr\mathrm{ln}G}[{\displaystyle _{\mathrm{}}^0}{\displaystyle \underset{\alpha }{}}{\displaystyle \frac{d\xi ^\alpha }{\sqrt{2\pi }}}e^{_{\alpha ,\beta }(\xi ^\alpha \xi _0)(G_{\alpha \beta }^1/2)(\xi ^\beta \xi _0)}+{\displaystyle _0^{\mathrm{}}}{\displaystyle \frac{d\xi }{(2\pi )^{\frac{n+1}{2}}}}e^{_{\alpha ,\beta }(G_{\alpha \beta }^1/2)(\xi ^{}\xi _0)^2}]^M1),`$
where:

$`\mathrm{\Sigma }_{\alpha \beta }=\delta _{\alpha \beta }+2\sigma ^2i\stackrel{~}{z}_{\alpha \beta };`$
$`G_{\alpha \beta }=\sigma _\delta ^2\delta _{\alpha \beta }+C\sigma _J^2z_{\alpha \beta }.`$ (20)
The evaluation of $`H(\{\xi _i\})`$, eq.(14), can be carried out in a very similar way, introducing replicas in the continuous+discrete stimulus space. The final result reads:

$`H(\{\xi _i\})=\underset{n0}{lim}{\displaystyle \frac{1}{n\mathrm{ln}2}}({\displaystyle }{\displaystyle \underset{\alpha ,\beta }{}}{\displaystyle \frac{dz_{\alpha \beta }}{2\pi /N}}{\displaystyle }{\displaystyle \underset{\alpha ,\beta }{}}d\stackrel{~}{z}_{\alpha \beta }e^{iN_{\alpha \beta }z_{\alpha \beta }\stackrel{~}{z}_{\alpha \beta }}e^{\frac{1}{2}Tr\mathrm{ln}\mathrm{\Sigma }}`$
$`{\displaystyle \underset{s_1..s_{n+1}}{}}{\displaystyle }d\vartheta _1..d\vartheta _{n+1}\left[p(\vartheta ,s)\right]^{n+1}e^{_{\alpha ,\beta }(\delta _{\alpha \beta }\mathrm{\Sigma }_{\alpha \beta }^1)\stackrel{~}{\eta }(\vartheta _\alpha ,s_\alpha )\stackrel{~}{\eta }(\vartheta _\beta ,s_\beta )/2\sigma ^2}_{\epsilon ,\vartheta ^0}^N`$ (21)
$`e^{\frac{M}{2}Tr\mathrm{ln}G}[{\displaystyle _{\mathrm{}}^0}{\displaystyle \underset{\alpha }{}}{\displaystyle \frac{d\xi ^\alpha }{\sqrt{2\pi }}}e^{_{\alpha ,\beta }(\xi ^\alpha \xi _0)(G_{\alpha \beta }^1/2)(\xi ^\beta \xi _0)}+{\displaystyle _0^{\mathrm{}}}{\displaystyle \frac{d\xi }{(2\pi )^{\frac{n+1}{2}}}}e^{_{\alpha ,\beta }(G_{\alpha \beta }^1/2)(\xi ^{}\xi _0)^2}]^M1).`$
## IV Replica symmetric solution

The integrals in eq.(19),(21) cannot be solved without resorting to an approximation. In analogy to what is used in , we use a saddle-point approximation (which in general would be valid in the limit $`M,N\mathrm{}`$) and we assume replica symmetry  in the parameters $`\{z_{\alpha \beta }\}`$, $`\{\stackrel{~}{z}_{\alpha \beta }\}`$. This allows to explicitely invert and diagonalize the matrices $`G`$,$`\mathrm{\Sigma }`$:

$`z_{\alpha \alpha }`$ $`=`$ $`z_0(n);z_{\alpha \beta }=z_1(n);`$
$`i\stackrel{~}{z}_{\alpha \alpha }`$ $`=`$ $`\stackrel{~}{z}_0(n);i\stackrel{~}{z}_{\alpha \beta }=\stackrel{~}{z}_1(n);`$ (22)
In  it has been shown that the replica symmetric solution for the information transmitted by a threshold-linear net is stable in most of the phase diagram. A detailed study of the stability of the RS solution in the specific case of mixed continuous and discrete stimuli will be presented elsewhere . The saddle point approximation seems to have more subtle implications in the present situation, as it will be discussed in the next section and in the final discussion.

In replica symmetry the mutual information can be expressed as follows:

$`I(\{\xi _i\},\vartheta s)`$ $`=`$ $`\underset{n0}{lim}{\displaystyle \frac{1}{n\mathrm{ln}2}}\{e^{N\left[(n+1)z_0^A\stackrel{~}{z}_0^An(n+1)z_1^A\stackrel{~}{z}_1^A\frac{r}{2}\left(Tr\mathrm{ln}G(z_0^A,z_1^A)+F(z_0^A,z_1^A)\right)\frac{1}{2}Tr\mathrm{ln}\mathrm{\Sigma }(\stackrel{~}{z}_0^A,\stackrel{~}{z}_1^A)H^A(\stackrel{~}{z}_0^A,\stackrel{~}{z}_1^A)\right]}`$ (23)
$`e^{N\left[(n+1)z_0^B\stackrel{~}{z}_0^Bn(n+1)z_1^B\stackrel{~}{z}_1^B\frac{r}{2}\left(Tr\mathrm{ln}G(z_0^B,z_1^B)+F(z_0^B,z_1^B)\right)\frac{1}{2}Tr\mathrm{ln}\mathrm{\Sigma }(\stackrel{~}{z}_0^B,\stackrel{~}{z}_1^B)H^B(\stackrel{~}{z}_0^B,\stackrel{~}{z}_1^B)\right]}\},`$
with

$$F(z_0,z_1)=2\mathrm{ln}\left[_{\mathrm{}}^0\underset{\alpha }{}\frac{d\xi ^\alpha }{\sqrt{2\pi }}e^{_{\alpha ,\beta }(\xi ^\alpha \xi _0)(G_{\alpha \beta }^1/2)(\xi ^\beta \xi _0)}+_0^{\mathrm{}}\frac{d\xi }{(2\pi )^{\frac{n+1}{2}}}e^{_{\alpha ,\beta }(G_{\alpha \beta }^1/2)(\xi ^{}\xi _0)^2}\right];$$
(24)
$$H^A(\stackrel{~}{z}_0,\stackrel{~}{z}_1)=\frac{1}{N}\mathrm{ln}\left[\underset{s}{}𝑑\vartheta p(\vartheta ,s)e^{_{\alpha ,\beta }\left(\delta _{\alpha \beta }\mathrm{\Sigma }_{\alpha \beta }^1\right)\stackrel{~}{\eta }(\vartheta ,s)^2/2\sigma ^2}_{\epsilon ,\vartheta ^0}^N\right];$$
(25)
$$H^B(\stackrel{~}{z}_0,\stackrel{~}{z}_1)=\frac{1}{N}\mathrm{ln}[\underset{s_1..s_{n+1}}{}d\vartheta _1..d\vartheta _{n+1}[p(\vartheta ,s)]^{n+1}e^{_{\alpha ,\beta }\left(\delta _{\alpha \beta }\mathrm{\Sigma }_{\alpha \beta }^1\right)\stackrel{~}{\eta }(\vartheta _\alpha ,s_\alpha )\stackrel{~}{\eta }(\vartheta _\beta ,s_\beta )/2\sigma ^2}_{\epsilon ,\vartheta ^0}^N].$$
(26)
We have set $`r=\frac{M}{N}`$ and $`z_0^{A,B}`$,$`\stackrel{~}{z}_0^{A,B}`$,$`z_1^{A,B}`$,$`\stackrel{~}{z}_1^{A,B}`$ are the solutions of the saddle point equations:

$`z_0^{A,B}`$ $`=`$ $`{\displaystyle \frac{}{\stackrel{~}{z}_0}}\left[{\displaystyle \frac{1}{2}}Tr\mathrm{ln}\mathrm{\Sigma }(\stackrel{~}{z}_0,\stackrel{~}{z}_1)+H^{A,B}(\stackrel{~}{z}_0,\stackrel{~}{z}_1)\right];`$
$`z_1^{A,B}`$ $`=`$ $`{\displaystyle \frac{1}{n}}{\displaystyle \frac{}{\stackrel{~}{z}_1}}\left[{\displaystyle \frac{1}{2}}Tr\mathrm{ln}\mathrm{\Sigma }(\stackrel{~}{z}_0,\stackrel{~}{z}_1)+H^{A,B}(\stackrel{~}{z}_0,\stackrel{~}{z}_1)\right];`$
$`\stackrel{~}{z}_0^{A,B}`$ $`=`$ $`{\displaystyle \frac{}{z_0}}{\displaystyle \frac{r}{2}}\left[Tr\mathrm{ln}G(z_0,z_1)+F(z_0,z_1)\right];`$
$`\stackrel{~}{z}_1^{A,B}`$ $`=`$ $`{\displaystyle \frac{1}{n}}{\displaystyle \frac{}{z_1}}{\displaystyle \frac{r}{2}}\left[Tr\mathrm{ln}G(z_0,z_1)+F(z_0,z_1)\right].`$ (27)
All the equations must be evaluated in the limit $`n0`$. It is easy to check that all terms in the exponent in eq.(23) are order $`n`$. In fact, since when $`n0`$ only one replica remains, one has:

$`\underset{n0}{lim}Tr\mathrm{ln}G(z_0,z_1)+F(z_0,z_1)`$ $`=`$ $`0`$
$`Tr\mathrm{ln}G(z_0,z_1)+F(z_0,z_1)`$ $``$ $`n{\displaystyle \frac{}{n}}[Tr\mathrm{ln}G(z_0,z_1)+F(z_0,z_1)]_{|_{n=0}}.`$ (28)
Therefore, from the saddle point equations, $`\stackrel{~}{z}_0^{A,B}`$ are order $`n`$ and $`Tr\mathrm{ln}\mathrm{\Sigma }`$ is also order $`n`$:

$$Tr\mathrm{ln}\mathrm{\Sigma }n\frac{}{n}Tr\mathrm{ln}\mathrm{\Sigma }_{|_{n=0}}.$$
(29)
Since $`\stackrel{~}{z}_0^{A,B}=n\stackrel{~}{\stackrel{~}{z}}_0^{A,B}`$, it is easy to check by explicit evaluation that, when $`n0`$, all the $`n+1`$ diagonal terms among the matrix elements $`\{\delta _{\alpha \beta }\mathrm{\Sigma }_{\alpha \beta }^1\}`$ are order $`n`$ and all the $`n(n+1)`$ out-of-diagonal terms are order $`1`$. Then all terms in the exponent of eqs.(25),(26) are order $`n`$, and we can expand the exponentials, which allows us to perform the quenched averages across $`\{\epsilon ,\vartheta ^0\}`$. Considering the expression of $`\stackrel{~}{\eta }(\vartheta ,s)`$, eq.(2), one obtains:

$`H^A(\stackrel{~}{z}_0^A,\stackrel{~}{z}_1^A)`$ $``$ $`n(\stackrel{~}{\stackrel{~}{z}}_0^A\stackrel{~}{z}_1^A)\mathrm{\Lambda }_\eta ^1;`$
$`H^B(\stackrel{~}{z}_0^B,\stackrel{~}{z}_1^B)`$ $``$ $`n\left((\stackrel{~}{\stackrel{~}{z}}_0^B\stackrel{~}{z}_1^B)\mathrm{\Lambda }_\eta ^1+{\displaystyle \frac{\stackrel{~}{z}_1^B}{1+2\sigma ^2\stackrel{~}{z}_1^B}}\left[\mathrm{\Lambda }_\eta ^1\mathrm{\Lambda }_\eta ^2\right]\right);`$ (30)
$`\mathrm{\Lambda }_\eta ^1`$ $`=`$ $`{\displaystyle \underset{s}{}}{\displaystyle 𝑑\vartheta p(s,\vartheta )[\stackrel{~}{\eta }(\vartheta ,s)]^2_{\epsilon ,\vartheta ^0}}`$ (31)
$`=`$ $`(\eta ^0)^2\left[(A_2+\alpha ^22\alpha A_1)\epsilon ^2_\epsilon +\alpha ^2+2\alpha (A_1\alpha )\epsilon _\epsilon \right];`$
$`\mathrm{\Lambda }_\eta ^2`$ $`=`$ $`{\displaystyle \underset{s_1,s_2}{}}{\displaystyle 𝑑\vartheta _1𝑑\vartheta _2[p(s,\vartheta )]^2\stackrel{~}{\eta }(\vartheta _1,s_1)\stackrel{~}{\eta }(\vartheta _2,s_2)_{\epsilon ,\vartheta ^0}}`$ (32)
$`=`$ $`(\eta ^0)^2\left[(A_1\alpha )^2\left({\displaystyle \frac{K1}{K}}\epsilon _\epsilon ^2+{\displaystyle \frac{1}{K}}\epsilon ^2_\epsilon \right)+\alpha ^2+2\alpha (A_1\alpha )\epsilon _\epsilon \right];`$
$$A_1=\frac{1}{2^{2m}}\left(\begin{array}{c}2m\\ m\end{array}\right);A_2=\frac{1}{2^{4m}}\left(\begin{array}{c}4m\\ 2m\end{array}\right);\alpha =\frac{\eta ^f}{\eta ^0}.$$
(33)
A similar expansion in $`n`$ for $`Tr\mathrm{ln}\mathrm{\Sigma }(\stackrel{~}{z}_0,\stackrel{~}{z}_1)`$ and for $`Tr\mathrm{ln}G(z_0,z_1)+F(z_0,z_1)`$ allows to derive explicitely the saddle point equations:

$`z_0^{A,B}`$ $`=`$ $`\sigma ^2+\mathrm{\Lambda }_\eta ^1;`$
$`z_1^A`$ $`=`$ $`{\displaystyle \frac{2\sigma ^4\stackrel{~}{z}_1^A}{2\sigma ^2\stackrel{~}{z}_1^A+1}}+\mathrm{\Lambda }_\eta ^1;`$
$`z_1^B`$ $`=`$ $`{\displaystyle \frac{2\sigma ^4\stackrel{~}{z}_1^B}{2\sigma ^2\stackrel{~}{z}_1^B+1}}+\left[1{\displaystyle \frac{1}{\left(1+2\sigma ^2\stackrel{~}{z}_1^B\right)^2}}\right]\mathrm{\Lambda }_\eta ^1+{\displaystyle \frac{1}{\left(1+2\sigma ^2\stackrel{~}{z}_1^B\right)^2}}\mathrm{\Lambda }_\eta ^2;`$
$`\stackrel{~}{z}_1^{A,B}`$ $`=`$ $`C\sigma _J^2{\displaystyle \frac{r}{2}}\{\sigma \left({\displaystyle \frac{\xi ^0}{\sqrt{p+q}}}\right){\displaystyle \frac{\xi ^0}{\left(p+q\right)^{\frac{3}{2}}}}{\displaystyle \frac{1}{p}}\mathrm{erf}\left({\displaystyle \frac{\xi ^0}{\sqrt{p+q}}}\right)`$
$`+{\displaystyle _{\mathrm{}}^{\mathrm{}}}Dt[1+\mathrm{ln}\left(\mathrm{erf}({\displaystyle \frac{\xi ^0t\sqrt{q}}{\sqrt{p}}})\right)]\sigma \left({\displaystyle \frac{\xi ^0t\sqrt{q}}{\sqrt{p}}}\right){\displaystyle \frac{1}{p^{\frac{3}{2}}}}[\xi ^0t{\displaystyle \frac{q+p}{\sqrt{q}}}]\};`$
where:

$$\mathrm{erf}(x)=_{\mathrm{}}^xDx^{}=_{\mathrm{}}^x𝑑x^{}\sigma (x^{});\sigma (x)=\frac{1}{\sqrt{2\pi }}e^{\frac{x^2}{2}};$$

$$p=\sigma _\delta ^2+C\sigma _J^2(z_0z_1);q=C\sigma _J^2z_1.$$

From the expression of $`z_0^{A,B}`$ in eq.(LABEL:saddlepoint), it is easy to verify that the dependence on $`\stackrel{~}{\stackrel{~}{z}}_0^{A,B}`$ in eq.(30), which might affect the information in eq.(23), cancels out with the products $`z_0^A\stackrel{~}{\stackrel{~}{z}}_0^A`$,$`z_0^B\stackrel{~}{\stackrel{~}{z}}_0^B`$ which should contribute to the information in the limit $`n0`$ (see eq.(23)). Therefore, since $`z_0^{A,B}`$ is known and $`z_1^{A,B}`$ depends only on $`\stackrel{~}{z}_1^{A,B}`$, the mutual information can be expressed as a function of $`z_1^{A,B}`$,$`\stackrel{~}{z}_1^{A,B}`$, which in turn are to be determined self-consistently by the saddle point equations.

The average information per input cell can be written, finally:

$$\frac{I}{N}(\{\xi _i\},\vartheta s)=\frac{1}{\mathrm{ln}2}\left\{\stackrel{~}{z}_1^Bz_1^B\stackrel{~}{z}_1^Az_1^A+r\left[\mathrm{\Gamma }_1(z_0^B,z_1^B)\mathrm{\Gamma }_1(z_0^A,z_1^A)\right]+\mathrm{\Gamma }_2^B(\stackrel{~}{z}_1^B)\mathrm{\Gamma }_2^A(\stackrel{~}{z}_1^A)\right\},$$
(35)
with

$`\mathrm{\Gamma }_1(z_0,z_1)`$ $`=`$ $`\sigma \left({\displaystyle \frac{\xi ^0}{\sqrt{p+q}}}\right){\displaystyle \frac{\xi ^0p}{2\left(p+q\right)^{\frac{3}{2}}}}+{\displaystyle \frac{1}{2}}\mathrm{ln}p\mathrm{erf}\left({\displaystyle \frac{\xi ^0}{\sqrt{p+q}}}\right)`$ (36)
$``$ $`{\displaystyle _{\mathrm{}}^{\mathrm{}}}Dt\mathrm{erf}\left({\displaystyle \frac{\xi ^0t\sqrt{q}}{\sqrt{p}}}\right)\mathrm{ln}\left[\mathrm{erf}\left({\displaystyle \frac{\xi ^0t\sqrt{q}}{\sqrt{p}}}\right)\right];`$
$$\mathrm{\Gamma }_2^A(\stackrel{~}{z}_1^A)=+\frac{1}{2}\mathrm{ln}(1+2\sigma ^2\stackrel{~}{z}_1^A)\stackrel{~}{z}_1^A(\sigma ^2+\mathrm{\Lambda }_\eta ^1);$$
(37)
$$\mathrm{\Gamma }_2^B(\stackrel{~}{z}_1^B)=\frac{1}{2}\mathrm{ln}(1+2\sigma ^2\stackrel{~}{z}_1^B)\stackrel{~}{z}_1^B(\sigma ^2+\mathrm{\Lambda }_\eta ^1)+\frac{\stackrel{~}{z}_1^B}{1+2\sigma ^2\stackrel{~}{z}_1^B}\left[\mathrm{\Lambda }_\eta ^1\mathrm{\Lambda }_\eta ^2\right].$$
(38)
The expression for the mutual information only contains terms linear in either $`N`$ or $`M`$. Since the last of the saddle-point equations, (LABEL:saddlepoint), contains $`r`$, if one fixes $`N`$ and increases $`M`$ the information grows non-linearly, because the position of the saddle point varies. It turns out that, as shown below, the growth is only very weakly sublinear, at least when $`MN`$. Analogously, fixing $`M`$ and varying $`N`$ we would find a non-linearity due to the $`r`$-dependence of the saddle point. If $`r`$ is fixed and $`N`$ and $`M`$ grow together, the information rises purely linearly.

What our analytical treatment misses out, however, is the nonlinearity required to appear as the mutual information approaches its ceiling, the entropy of the stimulus set. The approach to this saturating value was described at the input stage , where also the initial linear rise (in $`N`$) was obtained in the large noise limit . Our saddle point method is in same sense similar to taking a large (input) noise limit, $`\sigma \mathrm{}`$, to its leading (order $`N/\sigma ^2`$) term. It is possible that the saddle point method could be extended, to account also for successive terms in a large noise expansion. This would probably require integrating out the fluctuations around the saddle point, but by carefully analysing the relation of different replicas to different values of the quenched variables. We leave this possible extension to future work. The present calculation, therefore, although employing a saddle point method which is usually applicable for large $`N`$ and $`M`$, should be considered effectively as yielding the initial linear rise in the mutual information, the one observed with $`M`$ small.

## V Numerical results

Eq.(LABEL:saddlepoint) for $`\stackrel{~}{z}_1^{A,B}`$ has been solved numerically using a Matlab Code. Convergence to self-consistency has been found already after $`50`$ iterations with an error lower than $`10^{10}`$.

Fig.1 shows the mutual information as a function of the output population size, for an input population size equal to $`100`$ cells. This is contrasted with the information in the input units, about exactly the same set of correlates, calculated as in , by keeping only the leading (linear) term in $`N`$. In fact, in  the mutual information carried by a finite population of neurons firing according to eq.(1) had been evaluated analytically, in the limit of large noise, by means of an expansion in $`N(\eta ^0)^2/4\sigma ^2`$. To linear order in $`N`$ the analytical expression for the information carried by $`N`$ input neurons reads:

$$I_{input}(\{\eta _j\},\vartheta s)=\frac{1}{\mathrm{ln}2}\frac{N}{2\sigma ^2}\left(\mathrm{\Lambda }_\eta ^1\mathrm{\Lambda }_\eta ^2\right);$$
(39)
where $`\mathrm{\Lambda }_\eta ^1`$, $`\mathrm{\Lambda }_\eta ^2`$ are defined, again, as in eqs.(31), (32). In analogy to what had been done in  we have set $`C\sigma _J^2=1`$. As evident from the graph, also the output information is essentially linear up to a value of $`r0.5`$, and quasi-linear even for $`r=1`$. It should be remined, again, that our saddle point method only takes into account the term linear in $`N`$ in the information input units carry about the stimulus. It is not possible, therefore, for eq.(35) to reproduce the saturation in the mutual information as it approaches the entropy of the stimulus set (which is finite, if one considers only discrete stimuli). The nearly linear behaviour in $`M`$ thus reflects the linear behaviour in $`N`$ induced, in the intermediate quantity (the information available at the input stage), by our saddle point approximate evaluation.

As it is clear from the comparison in Fig.1, when the two populations of units are affected by the same noise the input information is considerably higher than the output one. This is expected, since output and input noise sum up while influencing the firing of output neurons, but also because the input distribution is taken to be a pure gaussian, while the output rates are affected by a threshold. If the input-output tranformation were linear and the output noise much smaller than the input one, one would expect that output and input units would carry the same amount of information.

Briefly, in a linear network with zero output noise one has:

$$p(\{\xi _i\}|\{\eta _j\})=\underset{i}{}\delta (\xi _i\underset{j}{}c_{ij}J_{ij}\eta _j);$$
(40)
Considering eqs.(11),(1), an effective expression for the distribution $`p(\{\xi _i\}|\vartheta ,s)`$ can be obtained by direct integration of the $`\delta `$ functions $`\delta (\xi _i_jc_{ij}J_{ij}\eta _j)`$ via their integral representation, on $`\{\eta _j\}`$:

$$p(\{\xi _i\}|\vartheta ,s)=\frac{1}{\sqrt{(2\pi )^Mdet\mathrm{\Sigma }}}e^{_{i,j}(\xi _i\stackrel{~}{\xi }_i(\vartheta ,s))(\mathrm{\Sigma }_{ij}^1)/2(\xi _j\stackrel{~}{\xi }_j(\vartheta ,s))};$$
(41)
$$\stackrel{~}{\xi }_i(\vartheta ,s)=\underset{j}{}c_{ij}J_{ij}\stackrel{~}{\eta }_j(\vartheta ,s);$$
(42)
$$\mathrm{\Sigma }_{ij}=\sigma ^2\underset{k}{}c_{ik}J_{ik}c_{kj}^TJ_{kj}^T;$$
(43)
This distribution is then used to evaluate both the equivocation, eq.(13), and the entropy of the responses, eq.(14). We do not report the calculation, that is straightforward and analogous to the one reported in . The final result, which is valid for a finite population size $`M`$, and up to the linear approximation in $`M(\eta ^0)^2/4\sigma ^2`$, is analogous to eq.(39):

$$I_{lin}(\{\xi _i\},\vartheta s)=\frac{1}{\mathrm{ln}2}\frac{M}{2\sigma ^2}\left(\mathrm{\Lambda }_\eta ^1\mathrm{\Lambda }_\eta ^2\right);$$
(44)
Thus, we expect that taking the limits $`\xi ^0\mathrm{}`$ and $`r0`$ simultaneously in eq.(35), we should get to the same result: the output information should equal the input one when $`\sigma ^2`$ grows large.

From eq.(35) it is easy to show that:

$$\underset{r0}{lim}\underset{\xi ^0\mathrm{}}{lim}I(\{\xi _i\},\vartheta s)=\frac{1}{\mathrm{ln}2}\frac{M}{2}\mathrm{ln}\left[1+\frac{2C\sigma _J^2\left(\mathrm{\Lambda }_\eta ^1\mathrm{\Lambda }_\eta ^2\right)}{\sigma _\delta ^2+2C\sigma _J^2\sigma ^2}\right];$$
(45)
When $`\sigma ^2\sigma _\delta ^2,\mathrm{\Lambda }_\eta ^1,\mathrm{\Lambda }_\eta ^2`$ one obtains exactly the linear limit, eq.(44). We have verified this analytical limit by studying numerically the approach to the asymptotic value of the mutual information. Fig.2 shows the dependence of output information on the output noise $`\sigma _\delta ^2`$, for 4 different choices of the (reciprocal of the) threshold, $`\xi ^0`$. A large value, $`\xi ^0=10`$, implies linear output units. As expected, the output information, which always grows for decreasing values of the output noise, for $`\xi ^0=10`$ approaches asymptotically the input information. For increasing values of the output noise, the information vanishes with a typical sigmoid curve, with its point of inflection when the output matches the input noise.

We have then examined how the information in output (compared to the input) depends on the number $`K`$ of discrete correlates and on the width of the tuning function (3), parametrized by $`m`$, with respect to the continuous correlate. Fig.3 shows a comparison between input and output information for a sample of 10 cells, as a function of $`K`$. Both curves quickly reach an asymptotic value, obtained by setting $`K\mathrm{}`$ in eq.(32) for $`\mathrm{\Lambda }_\eta ^2`$. The relative information loss in output is roughly constant with $`K`$. A comparison is shown with the case where correlates are purely discrete, which is obtained by setting $`m=0`$ in eq.(3). The curves exhibit a similar behaviour, even if the rise with $`K`$ is steeper, and the asymptotic values are higher. This may be surprising, but it is in fact a consequence of the specific model we have considered, eq.(2), where a unit has the same tuning curve to each of the discrete correlates, only varying its amplitude with respect to a value constant in the angle. As $`K\mathrm{}`$, most of the mutual information is about the discrete correlates, and the tuning to the continuous dimension, present for $`m=1`$, effectively adds noise to the discrimination among discrete cases, noise which is not present for $`m=0`$.

With respect to the continuous dimension, the selectivity of the input units can be increased by varying the power $`m`$ of the cosine from 0 (no selectivity) through 1 (very distributed encoding, as for the discrete correlates) to higher values (progressively narrower tuning functions). Fig.4 reports the resulting behaviour of the information in input and in output, for the case $`K=1`$ (only a continuous correlate) and $`K=4`$ (continuous+discrete correlates). Increasing selectivity implies a ”sparser”  representation of the angle, the continuous variable, and hence less information, on average. However if the correlate is purely continuous there is an initial increase, before reaching the optimal sparseness. It should be kept in mind, again, that the asymptotic equality of the $`K=1`$ and $`K=4`$ cases is a consequence of the specific model, eq.(2), which assigns the same preferred angle to each discrete correlate. The resolution with which the continuous dimension can be discriminated does not, within this model, improve with larger $`K`$, while the added contribution, of being able to discriminate among discrete correlates, decreases in relative importance as the tuning becomes sharper.

Figures 3 and 4 show that, as long as the output noise is non zero and the threshold is finite, information is lost going from input to output, but the information loss does not appear to depend on the structure and on the dimensionality of the correlate.

Note that, while the purely continuous case has been easily obtained by setting $`K=1`$ in the expression of $`\mathrm{\Lambda }_\eta ^2`$, eq.(32), for the purely discrete case it is enough to set $`m=0`$.

## VI Simulation results: information estimates via decoding

One way to test the range of validity of our analytical approximation si via numerical estimates of the information. In our specific case where we deal with populations of continuous neurons and with $`4`$ different sources of quenched disorder, direct numerical integration is prohibitive. A more feasible method is to generate simulated data both in input and in output and then to estimate the information from the data, resorting to some algorithm.

Many previous studies (revised in ) have shown that information estimates from data are extremely sensitive to sampling and the distortion is the more serious the larger is the space of the possible configurations which has to be sampled. In our case where neurons are characterized by their time averaged continuous firing rates, a simple binning of the responses into discrete distributions results in a considerable distortion, which becomes even more serious since a sparse sampling is required to perform the average across the four quenched distributions.

Several procedures have been proposed to correct the bias affecting information estimates; some of them (see ) are based on regularizations like binning or smoothing; other ones rely on more theoretical approaches aiming at providing an analytical expression for the correction .

When the responses vary in a continuum and the population size is very large it has been suggested that the best estimate is obtained via decoding: the method consists in generating a predicted stimulus $`s^{}(𝒓_k)`$ from each simulated response vector $`𝒓_k(s)`$ in each trial $`k`$ by matching $`𝒓_k(s)`$ to the average response vector $`𝒓_{av}(s^{})`$, for all stimuli. The predicted stimulus will be the one corresponding to the best match. Summing on all the trials one can derive a probability table $`p(s,s^{})`$; then, the mutual information between the true and the pseudo-stimuli is computed instead of the original one, between stimuli and responses:

$$I(s,s^{})=\underset{s=1}{\overset{p}{}}\underset{s^{}=1}{\overset{p}{}}P(s,s^{})\mathrm{log}_2\frac{P(s,s^{})}{P(s)P(s^{})}.$$
(46)
From a theoretical point of view the optimal transformation to derive the probability $`p(s^{}|𝒓)`$ is defined by Bayes rule (Bayesian decoding):

$$p(𝒓|s)p(s)=p(𝒓)p(s|𝒓);$$
(47)
In our specific case the input distribution is defined in eq.(1), so that relationship (47) can be explicitely inverted; on the contrary the output distribution is not explicitly known and one has to fit some function to the responses to be able to invert the relationship (47). Further details about these procedures can be found in .

Fig.5 shows the results of simulations for a sample of $`20`$ output cells receiving from $`1000`$ input cells. A MATLAB code has been used to generate data in an amount of $`2000`$ responses per neuron per stimulus, where we have considered the simpler case of $`4`$ purely discrete stimuli ($`m=0`$ in eq.3). The decoding in output has been optimized choosing a high value for the threshold $`\xi ^0`$, so that the single cell output distribution, as derived from the simulated data, could be roughly fitted by a gaussian. For each population size we have chosen many random subsamples of units out of the whole population, both in input and in output, and we have then averaged the information value across subsamples. Quenched averages have been performed resorting to a sparse sampling.

The plot shows that the curves obtained via simulations (dashed lines) match the analytical prediction (solid lines) for a very small number of cells, but they deviate when the number of cells increases; since both the input and the output noise are large and the information values are much lower than the upper bound of $`2`$ bits, this mismatch cannot be due to a deviation from linearity close to the ceiling regime. It is more probably an effect due to the distortion caused by the decoding, which is known to increase with the population size. The discrepancy between the true and the decoded information has also been recently investigated and quantified analytically .

The analytical approximation seems to have a wider applicability in output, where even for a population size larger than $`4`$-$`5`$ cells the analytical curve falls within the error. This is mainly an effect due to the larger error characterizing the output information, which in turn is due to the stronger fluctuations in the information values across the different subsamples of cells. We have checked that the analytical results are always confirmed for a population size of $`1`$ or $`2`$ cells varying the number of discrete correlates and the value of the input and output noise.

We certainly cannot conclude from this that our analytical approximation is not valid for population sizes larger than $`2`$-$`3`$ cells. On the contrary we are currently devising new algorithms to reduce the bias in the information estimates and preliminary results seem to suggest a much wider range of validity of the analytical approximation. A more detailed study comparing decoding to other algorithms will be published elsewhere .

## VII Discussion

We have attempted to clarify how information about multi-dimensional stimuli, with both a continuous and a discrete dimension, is transmitted from a population of units with a known coding scheme, down to the next stage of processing.

Previous studies had focused on the mutual information between input and output units in a two-layer threshold-linear network either with learning  or with simple random connection weights .

More recent investigations have tried to quantify the efficiency of a population of units in coding a set of discrete  or continuous  correlates. The analysis in  has been then generalized to the more realistic case of multi-dimensional continuous+discrete correlates .

This work correlates with both research streams, in an effort to define a unique conceptual framework for population coding. The main difference with the second group of studies is obviously the presence of the network linking input to output units. The main difference with the first two papers, instead, is the analysis of a distinct mutual information quantity: not between input and output units, but between correlates (”stimuli”) and output units. In  it had been argued, for a number $`K`$ of purely discrete correlates, that the information about the stimuli reduces to the information about the ”reference” neural activity when $`K\mathrm{}`$. The reference activity is simply the mean response to a given stimulus when the information is measured from the variable, noisy responses around that means; or it can be taken to be the stored pattern of activity, when the retrieval of such patterns is considered, as in . True, the information about the stimuli saturates at the entropy of the stimulus set, but for $`K\mathrm{}`$ this entropy diverges, only the linear term in $`N`$ is relevant , and the two quantities, information about the stimuli and information about the reference activity, coincide.

Our present saddle point calculation is only able to capture, effectively, the mutual information which is linear in the number of input units, as mentioned above. It fails to describe the approach to the saturating value, the entropy of the set of correlates, be this finite or infinite. Therefore, ours is close to a calculation of the information about a reference activity - in our case, the activity of the input units. The remaining difference is that we can take into account, albeit solely in the linear term, the dependence on $`K`$ (through the equation for $`\mathrm{\Lambda }_\eta ^2`$, eq.(32)), without having to take the further limit $`K\mathrm{}`$.

Due to the presence of a threshold and of a non zero output noise the information in output is lower than that in input, and we have shown analytically that in the limit of a noiseless, linear input-output transfer function the ouptput information tends asymptotically to the input one. We have not, however, introduced a threshold in the input units, which would be necessary for a fair comparison. In an independent line of research, recent work  has also quantified the contribution to the mutual information, in a different model, of cubic and higher order non-linearities in the transfer function, by means of a diagrammatic expansion in a noise parameter. In  it has been shown that the effect of a threshold in the input units on the input information results merely in a renormalization of the noise. The resulting effect on the output information remains to be explored, possibly with similar methods.

Considering mixed continuous and discrete dimensions in our stimulus set, we had been wondering whether the information loss in output depended on the presence or absence of discrete or continuous dimensions in the stimulus structure. We have shown that for a fixed, finite level of noise this loss does not depend significantly on the structure of the stimulus, but solely on the relative magnitude of input and output noise, and on the position of the output threshold.

Our analytical efforts have been also motivated by the difficulty to perform a simulation study for this specific scheme of continuous rate coding in presence of several distinct sources of quenched disorder. Nonetheless we have performed some simulations using a decoding procedure to estimate the information from simulated data both in input and in output. Our results confirm previous findings in that the distortion due to decoding grows with the population size, so that the simulations confirm the anlytical prediction only for a very small number of cells. We are currently devising new computational methods to improve the match between the analytical and the simulation results.

Further developments of this analysis include the evaluation of the output information in presence of learning, in line with , and with correlations in the firing of input units.

A recent work has shown that the interplay between short and long range connectivities in the Hopfield model leads to a deformation of the phase diagram with the appearence of novel phases . It would be interesting to introduce short and long range connections in our model, and to examine how the coding efficiency of output neurons depends on the interaction between short and long range connections. This will be the object of future investigations.

### Acknowledgements

We have enjoyed extensive discussions with Inés Samengo and Elka Korutcheva. Valeria Del Prete thanks Isaac Perez-Castillo for very useful comments. Partial support from Human Frontier Science Programme grant RG 0110/1998-B.


text:
# Electromagnetic transitions between giant resonances within a continuum-RPA approach

## 1 Introduction

Since the very beginning of nuclear physics, electromagnetic transitions between nuclear states have been known to be a sensitive tool to investigate structure of the nuclear wave functions. In recent years significant theoretical efforts have been devoted to describe intensities of the radiative transitions between charge-exchange giant resonances (GR) in nuclei -. The interest in the problem is related to the possibility to obtain corresponding experimental data from the (3He,$`t\gamma `$)-reaction cross sections. A confrontation of the data with the calculation results would be especially interesting in view of the theoretical speculations on the possible enhancement of the transition intensities -.

It should be mentioned that conclusions on the enhancement have been based upon the consideration of the relevant sum rules. However, the main contribution to the sum rules seems to come from the nuclear states of 2 particle - 2 hole type (double GR). For instance, the dipole sum rule for the Gamow-Teller resonance (GTR) should be mainly exhausted by the giant dipole resonance (GDR) built on the top of the GTR, i.e. a double GR, but not by the spin-dipole resonance (SDR), a single GR. In this connection the problem of a correct description for the mixing of single and double GR arises, because it might change significantly the transition intensities. The relevance of this problem is indicated by the observation that the calculations of the radiative transition intensities within TDA - and RPA  when taking only particle-hole structure of GRs into consideration have strongly underestimated the corresponding sum rules. It is noteworthy, that in recent works  the importance of taking into account the GDR admixture has been demonstrated to describe correctly the decay amplitude of the first $`1^{}`$ (two-phonon type) state to the ground state.

In the present work a general expression for the amplitude of the radiative transitions between different GRs is obtained for closed-shell nuclei within the continuum-RPA using a diagrammatic representation for the three-point Green’s function. The expression allows us to take the entire basis of particle-hole states into consideration as well as to incorporate the important effects of mixing of single and double GR.

We apply the general approach developed in this paper to calculate the intensity of E1 transition between the SDR and GT states. The calculation results for 90Nb and 208Bi nuclei differ markedly from the previous ones -, mainly due to the effects of mixing of the single and double GR. The corresponding experimental data for the latter nucleus is to appear soon .

## 2 The amplitude of electromagnetic transitions between giant resonances

We start with the well-known formula for the partial width corresponding to a radiative transition between initial $`|1`$ and final $`|2`$ nuclear states with the excitation energies $`E_1`$ and $`E_2`$, respectively,:

$$\mathrm{\Gamma }_\gamma ^{(12)}=C_\gamma \underset{m_2\mu }{}\left|2,m_2|\widehat{𝒬}_{k\mu }|1,m_1\right|^2=C_\gamma \frac{\left|2\widehat{𝒬}_k1\right|^2}{2J_2+1}.$$
(1)
Here, $`\widehat{𝒬}_{k\mu }=_aQ_{k\mu }(x_a)`$ is a single-particle multipole operator, $`2\widehat{𝒬}_k1`$ is the reduced matrix element, $`x`$ is the nucleon coordinate (including spin and isospin variables), $`m`$ is the angular momentum projection, $`C_\gamma `$ is an appropriate dimensional coefficient. In the case of E1 transitions, considered in the present paper to apply the general approach (see Sect. 3), $`Q_{1\mu }=D_\mu =\frac{1}{2}erY_{1\mu }(\stackrel{}{n})\tau ^{(3)}`$ is the isovector electric dipole operator and $`C_\gamma =\frac{16\pi }{9}\left(\frac{E_\gamma }{\mathrm{}c}\right)^3`$ with $`E_\gamma =E_1E_2`$ being the gamma-quantum energy.

Let $`|1`$ and $`|2`$ be states of the particle-hole (p-h) type so that their structure can be described within the RPA. Expression for the transition amplitude $`2\widehat{𝒬}_k1`$ can be obtained in Green function technique used in the theory of finite Fermi-systems  to describe the Fermi-system response to a single-particle probing operator. Let us consider the diagrams for the 3-point Green function (3-point vertex function) describing the transition amplitude of the system under the action of the external field $`\widehat{𝒬}`$ (Fig. 1).

The p-h structure of the state 1 (2) enters these graphs by means of the transition fields $`v_{1(2)}(x)=F(xx^{})\rho _{1(2)}(x^{})𝑑x^{}`$ with $`\rho _{1(2)}(x)`$ and $`F(xx^{})`$ being the transition densities and the particle-hole interaction (depicted by the full diamonds), respectively. The transition fields are proportional to the residues in the poles of the effective operator : $`\stackrel{~}{V}_{1(2)}(\omega ){\displaystyle \frac{M_{1(2)}v_{1(2)}}{\omega E_{1(2)}}}`$, with $`\stackrel{~}{V}`$ satisfying the equation :

$$\stackrel{~}{V}(x,\omega )=V(x)+F(xx^{})A(x^{}x^{\prime \prime },\omega )\stackrel{~}{V}(x^{\prime \prime },\omega )𝑑x^{}𝑑x^{\prime \prime },$$
(2)
Here, $`A(xx^{},\omega )`$ is the free particle-hole propagator, carrying the quantum numbers of the appropriate p-h state, the probing single-particle operator $`V_{1(2)}(x)`$ has the matrix element $`M_{1(2)}=1(2)|\widehat{𝒱}_{1(2)}|0`$ ($`\widehat{𝒱}=_aV(x_a)`$).

The matrix element corresponding to the graphs is:

$`2\widehat{𝒬}_k1=()^{J_1k}{\displaystyle \underset{345}{}}\{\begin{array}{ccc}J_2\hfill & k\hfill & J_1\hfill \\ j_3\hfill & j_4\hfill & j_5\hfill \end{array}\}[()^{J_1+J_2+k}4v_2^+55Q_k33v_14`$ (5)
$`\times \left({\displaystyle \frac{n_4(1n_5)(1n_3)}{(\epsilon _4+E_2\epsilon _5)(\epsilon _4+E_1\epsilon _3)}}{\displaystyle \frac{(1n_4)n_5n_3}{(\epsilon _4+E_2\epsilon _5)(\epsilon _4+E_1\epsilon _3)}}\right)`$
$`4v_133Q_k55v_2^+4({\displaystyle \frac{(1n_4)n_5n_3}{(\epsilon _4E_2\epsilon _5)(\epsilon _4E_1\epsilon _3)}}{\displaystyle \frac{n_4(1n_5)(1n_3)}{(\epsilon _4E_2\epsilon _5)(\epsilon _4E_1\epsilon _3)}})],`$
with $`n_\lambda `$ and $`\epsilon _\lambda `$ being occupation numbers and energies of the single-particle states, respectively ($`\lambda `$ is the set of the single-particle quantum numbers, $`(\lambda )=\{lj\}`$). The definition of the reduced single-particle matrix elements $`\lambda Q_k\mu `$ by the Wigner-Eckart theorem is taken in accordance with Ref. .

The Eq. (5) can be transformed to express the transition amplitude in terms of the RPA $`X`$ and $`Y`$ amplitudes (see, e.g., Refs. ):

$`2\widehat{𝒬}_k1=\sqrt{(2J_1+1)(2J_2+1)}{\displaystyle \underset{345}{}}()^{J_1+1+j_4+j_3}\{\begin{array}{ccc}J_2\hfill & k\hfill & J_1\hfill \\ j_3\hfill & j_4\hfill & j_5\hfill \end{array}\}`$ (8)
$`\times [()^{J_1+J_2+k}5Q_k3(X_{34}^{(1)}X_{54}^{(2)}+()^{J_1+J_2+j_3+j_5}Y_{43}^{(1)}Y_{45}^{(2)})`$
$`3Q_k5(X_{43}^{(1)}X_{45}^{(2)}+()^{J_1+J_2+j_3+j_5}Y_{34}^{(1)}Y_{54}^{(2)})]`$ (9)
with the use of the following representations for the $`X`$ and $`Y`$:

$`X_{21}^{(1)}=()^{j_2j_1+J_1+1}{\displaystyle \frac{2v_11}{\sqrt{2J_1+1}}}{\displaystyle \frac{n_1(1n_2)}{(\epsilon _1+E_1\epsilon _2)}}`$ (10)
$`Y_{21}^{(1)}=()^{j_2j_1+1}{\displaystyle \frac{1v_12}{\sqrt{2J_1+1}}}{\displaystyle \frac{n_1(1n_2)}{(\epsilon _1E_1\epsilon _2)}}`$ (11)
One sees in Fig. 1 that there is an asymmetry between the vertex 1,2 (”dressed”) on the one hand and vertex 3 (”undressed”) on the other. This is because we have not incorporated effects of particle-hole interaction in channel 3, or, in other words, we neglected the effect of the virtual excitation of the corresponding giant multipole resonance. This effect has been well-known to play a crucial role in describing the neutron radiative capture since the model of direct-semidirect capture was suggested  (see also Ref.  for more recent developments). This additional contribution which takes into account virtual excitation of the giant resonance is shown in Fig. 2.

The wavy line in the channel 3 depictures the total p-h propagator $`\stackrel{~}{A}`$ describing off-shell excitation of all p-h states (including the corresponding GR) with appropriate spin and parity. Using the fact that $`\stackrel{~}{A}Q=A\stackrel{~}{Q}`$, one can see that the sum of the graphs of Figs.1-2 is equivalent to the graphs of Fig.1 only if one replaces the external field $`Q_k(x)`$ by the corresponding effective field $`\stackrel{~}{Q}_k(x,\omega )`$ satisfying Eq.(2).

The Eq.(5) with the substitution $`Q_k(x)\stackrel{~}{Q}_k(x,\omega )`$ can be transformed further with the use of the spectral decomposition for the Green’s function of the single-particle radial Schrödinger equation $`g_{(\lambda )}(rr^{},\epsilon )={\displaystyle \underset{\lambda }{\stackrel{}{}}}{\displaystyle \frac{\chi _\lambda (r)\chi _\lambda (r^{})}{\epsilon \epsilon _\lambda }}`$ to include explicitly the single-particle continuum:

$`2\widehat{𝒬}_k1=()^{J_1k}[{\displaystyle \underset{4}{}}n_4(()^{J_1+J_2+k}G_4(E_1,E_2)+G_4^T(E_1,E_2))`$
$`{\displaystyle \underset{3,5}{}}n_3n_5(()^{J_1+J_2+k}G_{3,5}(E_1,E_2)+G_{3,5}^T(E_1,E_2))],`$ (12)
$`G_4(E_1,E_2)={\displaystyle \underset{(3)(5)}{}}t_{(3)(4)(5)}(4|v_2^+(g_{(5)}(\epsilon _4+E_2){\displaystyle \underset{5}{\stackrel{}{}}}n_5{\displaystyle \frac{|5)(5|}{\epsilon _4+E_2\epsilon _5}})`$
$`\times \stackrel{~}{Q}_k(g_{(3)}(\epsilon _4+E_1){\displaystyle \underset{3}{\stackrel{}{}}}n_3{\displaystyle \frac{|3)(3|}{\epsilon _4+E_1\epsilon _3}})v_1|4),`$
$`G_{3,5}(E_1,E_2)={\displaystyle \underset{(4)}{}}t_{(3)(4)(5)}{\displaystyle \frac{(5|\stackrel{~}{Q}_k|3)}{\epsilon _5\epsilon _3+E_1E_2}}(3|v_1(g_{(4)}(\epsilon _3E_1)g_{(4)}(\epsilon _5E_2))v_2^+|5),`$
$`t_{(3)(4)(5)}=\{\begin{array}{ccc}J_2\hfill & k\hfill & J_1\hfill \\ j_3\hfill & j_4\hfill & j_5\hfill \end{array}\}(4)v_2^+(5)(5)Q_k(3)(3)v_1(4),`$ (15)
where we have decomposed the reduced matrix elements $`\lambda v\mu (\lambda )v(\mu )(\lambda |v|\mu )`$ into its spin-angular $`(\lambda )v(\mu )`$ and radial $`(\lambda |v|\mu )=\chi _\lambda (r)v(r)\chi _\mu (r)𝑑r`$ parts. The radial integrals (for brevity omitted) can be easily recovered, for instance,

$`(4|v_2^+g_{(5)}(\epsilon _4+E_2)\stackrel{~}{Q}_kg_{(3)}(\epsilon _4+E_1)v_1|4)=`$
$`{\displaystyle \chi _4(r)v_2^+(r)g_{(5)}(rr^{},\epsilon _4+E_2)\stackrel{~}{Q}_k(r^{})g_{(3)}(r^{}r^{\prime \prime },\epsilon _4+E_1)v_1(r^{\prime \prime })\chi _4(r^{\prime \prime })𝑑r𝑑r^{}𝑑r^{\prime \prime }}.`$
The superscript $`T`$ means transposition and $`^{}`$ runs over single-particle states with the same $`l`$ and $`j`$.

The Eq. (2) gives the general expression for the amplitude of the electromagnetic transitions within the CRPA. Apart from the single-particle continuum, it contains also contribution from the double phonon configurations. To see this, let us decompose the amplitude into ”direct” and ”semi-direct” parts according to the partition $`\stackrel{~}{Q}_k=Q_k+(\stackrel{~}{Q}_kQ_k)`$. The first, direct, part describes the amplitude in the single-phonon space and it has been only taken into account in all previous theoretical considerations -. As for the semi-direct part, the following approximate representation holds:

$$\stackrel{~}{Q}_k(x,\omega )Q_k(x)\underset{i}{}\frac{M_iq_i(x)}{\omega \omega _i}.$$
(16)
with $`q_i(x)`$ being the transition fields for the doorway states forming the GDR. Thus, one can see, that the semi-direct part originates from admixtures of double-phonon configurations, like $`|2+_i\alpha _i|1|i`$. The contribution to the transition amplitude is determined by both the matrix elements $`M_i=i|\widehat{𝒬}_k|0`$ (states $`|1`$ or $`|2`$ serve as spectators in these electromagnetic transitions) and the admixture amplitudes $`\alpha _i`$.

It can be seen from Eqs. (2,16) that the mixing is described within first order perturbation theory with respect to the particle-hole interaction. Nevertheless, the approximation is satisfied in most cases due to the small values of the mixing amplitudes, that can be also seen from the calculated transition amplitudes being much less than the amplitude between the GDR and the ground state.

## 3 E1 transitions between the spin-dipole and Gamow-Teller resonances.

As an application of the approach described in the previous section, we calculate intensities and branching ratios for E1 decay of the charge-exchange spin-dipole resonance to some Gamow-Teller states (GTR and low-lying satellites) identified in Ref. . To calculate the GT and SD strength distribution, we use the CRPA approach of Refs. . The phenomenological isoscalar part of the nuclear mean field and the zero-range Landau-Migdal particle-hole interaction are the ingredients of the approach along with some selfconsistency conditions. The approach has allowed one to reproduce very well the experimental GTR energies in 90Nb and 208Bi nuclei . For each spin $`J_{SDR}^\pi =(0^{},1^{})`$ of the SD components we choose just one state with the maximal strength to be the SDR, lying at (26.6, 24.1) MeV in 208Bi and at (22.4, 20.1) MeV in 90Nb, and for $`J_{SDR}^\pi =2^{}`$ we choose three states with the maximal excitation energies, lying at (20.0, 21.3, 23.1) MeV in 208Bi and at (13.5, 14.4, 17.4) MeV in 90Nb.

The details of the calculation of the effective fields for the SD and GT cases can be found in Refs. , and for the isovector dipole one - in Ref. . In the latter case we take the velocity dependent particle-hole forces along with the isovector Landau-Migdal interaction that allows us to reproduce in the calculations both the experimental GDR energy and the observed excess of the energy-weighted sum rule over the TRK one. Because some transition energies are rather close to the GDR energy, the influence of the GDR spreading becomes important. We use the substitution $`\omega \omega +\mathrm{i}\mathrm{\Gamma }(\omega )/2`$ in Eq. (2) to simulate the effect of GDR spreading (the mean energy-dependent doorway spreading width $`\mathrm{\Gamma }(\omega )`$ has been chosen as in Ref. ).

We calculate the reduced matrix elements $`GTR\widehat{𝒟}_kSDR,J`$ according to Eq. (2) and Eq. (5) (with and without taking into account the virtual GDR excitation, respectively), and then the corresponding radiative widths $`\mathrm{\Gamma }_\gamma `$ and $`\mathrm{\Gamma }_\gamma ^0`$ according to Eq. (1). The calculated widths for the E1 decay of the above-mentioned SD states to the GT states at the excitation energy $`E_{GT}`$ are listed in Table 1. The corresponding branching ratios are calculated as the ratio $`\mathrm{\Gamma }_\gamma /\mathrm{\Gamma }^{}`$ with $`\mathrm{\Gamma }^{}`$ being the mean doorway-state spreading width chosen to reproduce the experimental total SDR width in the strength-function calculations ($`\mathrm{\Gamma }^{}=4.7`$ MeV for 208Bi , $`\mathrm{\Gamma }^{}=4.`$ MeV for 90Nb).

As can be seen from the calculation results listed in the Table 1, taking into account of the virtual GDR excitation affects the results drastically. In the cases where the transition energy is well below the GDR energy, a destructive interference between the direct and semidirect parts of the amplitude always occurs, in agreement with both the qualitative description in terms of the effective charge  and previous quantitative consideration of Ref. . In the cases when the transition energy is close to the GDR energy or exceeds it, the opposite situation takes place and one can see a significant enhancement of the corresponding amplitudes. This circumstance along with the steep $`E_\gamma ^3`$ dependence of the E1 radiative width leads to the fact that the widths for SDR decay to some low-lying GT states can be comparable to those for SDR $``$ GTR E1 transitions. Some of the calculated branching ratios are of order of $`10^3`$, that seem to be accessible in experiments. In fact, the statistical background hinders detection of SDR $``$ GTR decay, whereas it is small enough to allow detection of SDR decay to low-energy GT states which has been already observed in (3He,$`t\gamma `$) experiments on 208Pb . For instance, strong gamma transitions have been observed to the unresolved GT states with E=3.8, 4.5, 4.9 MeV . The energy of these gamma transitions (about 14 MeV) is very close to the energy of the GDR, and the predicted branching ratio ($`10^3`$) can explain the observed transitions. However, we have to mention that the present calculations overestimate the mean energy of the SDR by about 2 MeV . Bearing in mind strong energy-dependent renormalization of the transition amplitude found in this paper, this can lead to a noticeable difference between the calculated and observed branching ratios. We have not also taken the finite widths of the GTR and the SDR into consideration.

Special interest for testing the model lies in the E1 transition from the GTR to a 2.9 MeV 2- SD state in 208Bi, which was observed recently at KVI . The latter state has presumably a ($`\pi 1h_{9/2})(\nu 1i_{13/2})^1`$ particle-hole structure and has a predicted excitation energy of 2.4 MeV. For this E1 transition we have found a large branching ratio $`b_\gamma =4.310^3`$ due to the fact that the transition energy almost exactly coincides with the GDR energy, therefore we obtain maximal enhancement.

Finally, it is also worth to mention the sensitivity of the radiative widths to the spin of the SDR components due to their different energy position that could be in principle used to get experimentally information on their energy splitting.

## 4 Conclusion

A general continuum-RPA approach has been developed in the present paper to describe the intensities of electromagnetic transitions between different giant resonances. We have used a diagrammatic representation for the three-point Green’s function to derive the expression for the transition amplitude. The expression has allowed us to take the entire basis of particle-hole states into consideration as well as to incorporate important effects of mixing of single and double giant resonances. The radiative widths for E1 transition between the SDR and GT states (including GTR) have been calculated for 90Nb and 208Bi nuclei. Some of the widths seem to be large enough to be accessible in experiments.

## Acknowledgments

Authors are grateful to Prof. M.N. Harakeh and Dr. A. Krasznahorkay for useful discussions and suggestions as well as to the Referee for putting forward a proposal improving the figures’ content. This work was supported in part by the Nederlandse Organisatie voor Wetenschapelijk Onderzoek. V.A.R. would like also to thank the Graduiertenkolleg ”Hadronen im Vakuum, in Kernen und Sternen” (GRK683) for supporting his stay in Tübingen and Prof. A. Fäßler for hospitality.