text:
# Long-Distance Quantum Communication with Entangled Photons using Satellites

## I Introduction

Quantum entanglement is at the heart of quantum physics Schrodinger35a  and at the same time the basis of most quantum communication protocols such as quantum cryptography Ekert91 ; Jennewein00 ; Tittel00 ; Naik00 , quantum dense coding Mattle96a , quantum teleportation Bennett93a  or methods to exploit the computational advantages of quantum communication complexity Brassard01 ; Buhrman99 ; Brukner02 . Each of those schemes allows efficient communication and computation beyond the capabilities of classical communication, which makes it attractive as a new emerging quantum information technology. This might lead to the build-up of a global quantum communication network, where the distribution and manipulation of quantum entanglement on a global scale is a central task. However, while the realization of such schemes is routine work in the laboratory, non-trivial problems emerge in long-distance applications. At present, the only suitable system for long-distance quantum communication are photons. Other systems such as atoms or ions are studied thoroughly, however their applicability for quantum communication schemes is presently not feasible within the near future, leaving photons as the only choice for long-distance quantum communication. One of the problems of photon-based schemes is the loss of photons in the quantum channel. This limits the bridgeable distance for single photons to some estimated 100 km in present silica fibers Waks02 ; Gisin02 . In principle, this drawback can eventually be overcome by subdividing the larger distance to be bridged into smaller sections over which entanglement can be teleported. The subsequent application of so-called “entanglement swapping” Zukowski93a  may result in transporting of entanglement over long distances. Additionally, to diminish decoherence effects possibly induced by the quantum channel, quantum purification might be applied to eventually implement a full quantum repeater Briegel98a . In fact, the experimental building blocks for a full-scale quantum repeater based on linear optics have been successfully demonstrated over the last years by the realization of teleportation and entanglement swapping Bouwmeester97a ; Pan98a ; Jennewein01  and, only recently, by the implementation of a quantum purification protocol Pan03b . Two related, recent results, both of relevance for long-distance applications, are the demonstration of quantum state teleportation over a distance of several tens of meters Marcikic03  and the first realization of freely propagating teleported qubits Pan03 , which eventually will allow the subsequent use of teleported states. From the present point of view it seems obvious that a full implementation of a quantum repeater is within reach.

Despite those achievements of quantum communication experiments, the distances over which entanglement can be distributed in a single section, i.e. without a quantum repeater in-between, are by far not of a global scale. Experiments based on present fiber technology have demonstrated that entangled photon pairs can be separated by distances ranging from several hundreds of meters up to 10 km Tapster94 ; Tittel98 ; Weihs98 , but no significant improvements are to be expected. On the other hand, optical free-space links could provide a unique solution to this problem since they allow in principle for much larger propagation distances of photons due to the low absorption of the atmosphere in certain wavelengths ranges. Also, the almost non-birefringent character of the atmosphere Buttler98c  guarantees the preservation of polarization entanglement to a high degree. Free-space optical links have been studied and successfully implemented already for several years for their application in quantum cryptography based on faint classical laser pulses Buttler98c ; Hughes02 ; Kurtsiefer02 . A next crucial step is the distribution of quantum entanglement via such free space links.

However, terrestrial free-space links suffer from obstruction of objects in the line of sight, from possible severe attenuation due to weather conditions and aerosols Horvath02  and, eventually, from the Earth’s curvature. They are thus limited to rather short distances. To fully exploit the advantages of free-space links, it will be necessary to use space and satellite technology. By transmitting and/or receiving either photons or entangled photon pairs to and/or from a satellite, entanglement can be distributed over truly large distances and thus would allow quantum communication applications on a global scale. Such a scenario looks unrealistic at first sight, but in this paper we will show that the demonstration of quantum communication protocols using satellites is already feasible today. To do so, we will describe possible space scenarios based on entanglement. We then analyze prerequisites to distribute entanglement via satellites, describe experimental scenarios for first proof-of-principle experiments and finally give an outlook on the perspectives of satellite-aided quantum communication.

## II Scenarios of Space Experiments

When considering space scenarios that allow the distribution of entangled photon pairs we can distinguish the cases in which a satellite is used to carry either a transmitter of entangled photons, or a receiver, or a relay station to distribute photons to further locations. These scenarios will permit different applications.

### II.1 Earth-based transmitter terminal

The scenarios involving an Earth-based transmitter terminal allow to share quantum entanglement between ground and satellite, between two ground stations or between two satellites and thus to communicate between such terminals employing quantum communication protocols. In the most simple case, a straight uplink to one satellite-based receiver (see Fig. 1 a) can be used to perform secure quantum key distribution between the transmitter station and the receiver. Here, one of the photons of the entangled pair is being detected right at the transmitter site and thus the entangled photon source is used as a triggered source for single photons. If the satellite acts as a relay station (see Fig. 1 b), the same protocol can be established between two distant Earth-based communication parties. Shared entanglement between two parties can be achieved by pointing each of the photons of an entangled pair either towards an Earth-based station and a satellite or towards two separate satellites (see Figs. 1 c and d). Another set of satellite-based relays can be used to further distribute the entangled photons to two ground stations (see Fig. 1 e). Possible applications for shared entanglement between two parties are quantum key distribution or entanglement-enhanced communication protocols Bennett96c .

### II.2 Space-based transmitter terminal

In a second scenario, a transmitter with an entangled photon source is placed on a space-based platform. This allows not only longer link distances because of reduced influence of atmospheric turbulence (see Appendix B). It will also be the preferred configuration for global quantum communication, since only one downlink per photon of the entangled pair is necessary to share entanglement between two Earth-based receivers. Again, already a simple downlink allows to establish a single-photon link e.g. for quantum cryptography (see Fig. 2 a). In this configuration, a key exchange between two ground stations is also possible. To this end each of the two ground stations has to establish a quantum key with the satellite. Since the space terminal has access to both keys, it can transmit a logical combination of the keys, which can then be used by either ground station or both ground stations such that they arrive at the same key. This logical combination can easily be chosen such that it cannot reveal any information about the key. Note that the key does not have to be generated simultaneously at both receiver stations. In principle, a quantum key exchange can be performed between arbitrarily located ground stations. This is also possible for a ground-based transmitter terminal as shown in Fig. 1 a. However, in all such scenarios based on single photons the security requirement for the transmitter terminal is as high as it is for the ground station. Only the use of entangled states sent to two separate ground stations allows instantaneous key exchange between these two communicating earth-bound parties and also relaxes the security requirement for the transmitter module. Furthermore, more advanced quantum communication schemes will be feasible. The required shared entanglement can be established either by two direct downlinks (Fig. 2 b) or by using additional satellite relay stations (Fig. 2 c). Quantum entanglement can also be distributed between a ground station and a satellite (Fig. 2 d) or between two satellites (Fig. 2 e). In an even more elaborate scheme a third party might be involved, which is capable of performing a Bell-state analysis on two independent photons (Fig. 2 f). This allows quantum state teleportation and even entanglement swapping and could thus resemble a large-scale quantum repeater for a truly global quantum communication network. Applied to quantum cryptography, this third party might be used to control the communication between the two other parties in a ”Third-Man” cryptography protocol. For example in a polarization-based experiment, depending on whether he performs a simple polarization analysis of the independent photons or a Bell-state measurement, the ”Third-Man” can communicate secretly with either of the two parties (or with both) or he can control whether the two can communicate secretly or not without knowing the content of the communication.

### II.3 Link requirements

The maximal acceptable link attenuation for a quantum communication system based on entangled photons is determined by the timing resolution and the dark count rates of the detectors used, as well as by the net production rate of the source. As the minimum signal-to-noise ratio we assume that necessary for the violation of a Bell-inequality (see Appendix A). With a typical detection efficiency $`\eta _{Det}=0.3`$, a photon production rate of $`P=510^5s^1`$, an estimated total background count rate of $`S=10^3s^1`$ and a coincidence timing window of $`\mathrm{\Delta }\tau =510^9`$ s, the link efficiency should obey

$$\eta _{link}6.66\mathrm{}10^7$$
(1)
when following the calculations presented in Appendix A. Roughly speaking, a total link efficiency of $`\eta _{link}=\eta _{link1}\eta _{link2}10^6`$ ($``$60 dB) is necessary.

The link attenuation is also important for determining the number of photon pairs that can be received in a certain time window. This could be crucial in scenarios where the links are only available for short times as is for example the case in uplinks to low orbiting LEO satellites.

## III Link Attenuation

The overall link efficiency of 10-6, corresponding to a maximum attenuation of 60 dB, imposes quite a strong restriction to the various space scenarios. In the following, we will investigate the link attenuation for optical free-space links involving space infrastructure. The attenuation factor calculated includes the effect of beam diffraction, attenuation and turbulence-induced beam spreading caused by the atmosphere, receive aperture diameter, losses within the telescopes acting as antennas, as well as antenna pointing loss. Effects not included in this factor are the detection efficiency of the photon-counter modules ($``$3.5 dB per detector) and reflective and absorptive losses at optical components (typically $``$3 dB per individual photon link). One may take these losses into account when calculating individual link budgets. Figure 3 summarizes the scenarios considered based on satellites in geostationary orbit (GEO) and in low earth orbit (LEO). Such satellites may serve as a platform for transmitters or receivers. We presently do not envision the use of passive relays, e.g. retro-reflectors or mirrors, because of the high link loss they would introduce and because of the difficulty to implement a point-ahead angle 111The point-ahead angle denotes the difference angle between transmit and receive direction of the telescope. Its occurrence is a consequence of the movement of the satellite together with the finite velocity of light of the signal propagation.

### III.1 Satellite–ground links

#### III.1.1 Ground–LEO or LEO–Ground links

For the case of a LEO-based transmitter or receiver (Link 1 in Fig. 3), link attenuation poses no problems. Even for quite small telescopes onboard the LEO satellite, the attenuation factor is well below $`60\text{dB}`$ for all cases. Figure 4 is a contour plot of the link attenuation as a function of transmitter and receiver aperture diameter ($`D_T`$,$`D_R`$) for the ground-to-LEO uplinks operated at a wavelength of $`\lambda =800\text{nm}`$. Two additional vertical scales give the link distance L for $`30\text{cm}`$ receive telescope aperture as well as for the receive telescope aperture for a link distance of L=$`500\text{km}`$ (For the equation and further parameters used to arrive at Fig. 4 \- and also at some subsequent figures - see Appendix B.). The lines of equal attenuation are separated by $`5\text{dB}`$. The corresponding plot for LEO-to-ground downlinks is shown in Figure 5. One notes that the attenuation is much larger for the uplink than for the downlink. This is caused by the pronounced influence of atmospheric turbulence for the uplink, where the turbulent layers are close to the transmitter. In contrast, for a downlink the effect of the turbulent layer close to the receiver is negligible to first order. Another consequence of turbulence is that increasing the transmitter aperture for the uplink beyond $`60\text{cm}`$ hardly decreases the link attenuation.

In the case of links connecting a LEO and a ground station the possible duration for communication is comparatively short (e.g. a few minutes) and the angular velocity with which the telescope at the ground station has to be moved to track the satellite along its orbit is high. For all ground-to-space links the possibility of communication is weather dependent. While for clear weather and sufficient altitude of the ground station the uplink attenuation caused by the atmosphere is mainly determined by turbulence-induced beam spread 222For a diffraction limited telescope of $`20\text{cm}`$ diameter and a link distance of $`500\text{km}`$, some $`20\text{dB}`$ attenuation have to be expected pfennigbauer02\_2 ., clouded skies will make any link impossible. This influence is augmented by the low elevation angles typical for LEO-to-ground links and the thus increased fraction of the propagation path within the atmosphere.

#### III.1.2 Ground–GEO or GEO–Ground links

The long distance in links between GEO and ground (Link 2 in Fig. 3) results in a relatively high attenuation. With a ground station aperture of $`D=100`$ cm and a GEO terminal aperture of $`D=30`$ cm one will meet the 60 dB requirement in a downlink, but not in an uplink (compare Table 1 of Section V).

### III.2 Satellite-satellite links

While from a technological point of view a satellite-to-satellite link is the most demanding configuration it offers highly attractive scientific possibilities. It allows to cover, in principle, arbitrarily large distances and might thus also be a possibility for further novel fundamental tests on quantum entanglement.

We calculated the attenuation factor as a function of transmitter and receiver aperture diameter for a LEO-LEO link (Link 3 in Fig. 3). Figure 6 displays the attenuation for a wavelength of 800 nm as a function of the satellite distance $`L`$ and telescope diameters $`D_T`$ and $`D_R`$, assumed to be equal for both terminals. We conclude, that the $`60\text{dB}`$-limit poses no problem for LEO-LEO links with reasonable link distance.

For GEO-GEO links (Link 4 in Fig. 3), the attenuation can be read off Fig. 7, where again equal telescope apertures have been assumed. For a distance of $`L=45000\text{km}`$, an attenuation of $`A=55`$ dB would result for $`D_T=D_R=30`$ cm.

## IV Technological Prerequisites for Entanglement in Space

For performing the quantum communication experiments as described above, the following minimum hardware is required: a transmitter terminal to generate and transmit the entangled particles and one or more receiver terminals suited for single-photon manipulation and detection. Below we briefly outline their main features.

### IV.1 Transmitter, receiver and relay modules

A transmitter module comprises a photon source for entangled photon pairs (including passive or active manipulation of single qubit-states), a module for timing synchronization with the receiver station and channel for classical communication. Present entangled photon sources rely on laser-pumped spontaneous parametric down conversion Bouwmeester00 . This technology is very likely capable of being miniaturized to a size suitable for satellite modules.

A receiver module comprises one ore more optical input channels, each of which allows independent manipulation of qubits such as the rotation of photon polarization or the modulation of an interferometric phase. Additionally, it has to be equipped with single-photon detectors at each input port, a receiver module for timing-synchronization, and a classical channel for communication with the transmitter. Depending on whether active (remote) control of optical elements for qubit manipulation is possible or not (via, e.g., a polarizer or a retarder), we distinguish between active and passive receiver modules. Passive manipulation only requires a static setup of linear optics components. Typically, beam splitters in the input ports would randomly distribute incoming photons to differently oriented retarders, polarizers or beam splitters, where a manipulation and successive detection of single photons takes place. This kind of passive receiver module has recently also been suggested by Rarity et al. in a space-suited system for single-photon quantum key distribution Rarity02\_1 . For active control of single qubit manipulation, an additional information concerning the arrival time (i.e. a timing synchronization) is required. More advanced quantum communication schemes, such as quantum dense coding or quantum teleportation, require at some stage of the protocol the projection of independent photons in a joint Bell-state, i.e. a Bell-state measurement. Up to now, efficient achievement of this projection is only possible by means of two-particle-interferometry Mattle96a  333Recently, another scheme based on the nonlinear couplings has been suggested to achieve a full discrimination of all four Bell states although with low efficiency Kim01 .This requires that the arrival time difference of two photons at the two input ports of a beam splitter has to be less than their coherence time (typically 0.5 ps) Zukowski95 .

A relay module would redirect and/or manipulate qubit states without actually detecting them. Possibilities for its implementation range from a simple retroreflector to a more sophisticated relay-satellite (e.g. for deep-space communications), where the entanglement of photons establishing the quantum channel between ground station and deep-space satellite could be purified. We emphasize, that a relay can not serve as an amplifier. This is a consequence of the quantum no-cloning theorem Wooters82 .

### IV.2 Existing optical space technology

Optical transceivers for space-to-ground links or intersatellite links are almost state of the art. The major design parameters for the transmission sub-system are laser wavelength, modulation format and data rate, and reception technique. Of equal importance is the sub-system required for beam pointing, link acquisition, and automatic mutual terminal tracking (PAT). Because of the very narrow widths of the communication beams involved, PAT asks for highly sophisticated concepts and for electro-mechanic and electro-optic hardware meeting exceptional technological standards. Major parameters entering the link capacity are telescope size, optical transmit power, link distance, and receiver sensitivity. Other aspects are mass, volume, and power consumption of the terminal. Examples for existing space laser communication links include ESA’s intersatellite link SILEX (Semiconductor Laser Intersatellite Link Experiment) and a satellite ground link, which was only recently realized between ARTEMIS and ESA’s optical ground station OGS at Tenerife.

Photon sources and detectors presently implemented in such classical space laser communication systems can, in general, not be directly employed in quantum communication systems. However, the experience available may serve as a starting point for the development of space qualified components needed for quantum space experiments. The available optical communication technology could of course be applied to provide the classical channel that is always necessary in parallel to the quantum channel. One would also make synergistic use of some of the optics employed for PAT and employ one and the same telescope as antenna for both the classical and the quantum channel, which is a novel way of quantum-classical-multiplexing.

### IV.3 Proof-of-principle Experiments

The establishment of entanglement in space and, subsequently, its use for fundamental quantum physics experiments and quantum communication applications necessitates certain experimental stages:

#### Stage I

Creation and detection of qubits (here: single photons) via an optical space link. From an application point of view this achievement would already allow to perform quantum key distribution based on single photons.

#### Stage II

Establishment of entanglement (i.e. non-classical correlations via shared entangled particles) between the communicating parties. This includes the ability to detect single qubits synchronously at the spatially separated locations of the communicating parties. This stage already allows the most fundamental experiment in quantum physics, the violation of Bell’s inequality. It also makes possible further experiments such as quantum key distribution based on entangled qubits.

#### Stage III

Bell-state analysis of independent qubits. For the case of photons, the most efficient scheme relies on two-photon interferometry at a beam splitter. Technically speaking, the arrival time of the photonic qubits at the receiver module has to be synchronized such that photon wave-packets overlap at the beam splitter within their coherence length. If this problem is solved, all advanced quantum communication and computation protocols such as quantum state teleportation or quantum dense coding can be implemented.

The selection of an experimental scenario, in which all of these stages can be performed, requires a trade-off between link attenuation and experimental flexibility. It has been shown above that the total link attenuation of the experimental setup must not exceed some $`60\text{dB}`$, assuming present-day quantum-optics technology. Presently, without the use of quantum memories, entanglement can only be shared when more than one link is available. For the symmetric case of two equally long quantum links (one transmitter and two receivers), this limits the maximal single-link attenuation (between one transmitter and one receiver) to approx. $`30\text{dB}`$. Although space-to-space links have the attractive advantage of not being influenced by the Earth’s atmosphere, we have to discard them at present due to the expected disproportionate technological and financial effort as compared to alternative schemes with at least one of the communication terminals is on ground. Since two links should be established, it is therefore most reasonable to place the transmitter module in space, while the receiver modules stay in easily accessible ground-based laboratories. Most envisioned quantum experiments require higher flexibility at the receiver due to active polarization control or data analysis. Also, the atmosphere causes a larger footprint in an uplink than in a downlink, due to the higher influence of turbulence. With such perspectives it becomes obvious that in a first proof-of-principle experiment one should place the transmitter module into space and the receiver module(s) on Earth. Because of their relative stationarity, terminals placed on GEO satellites do not require such a highly sophisticated pointing, acquisition and tracking (PAT) systems as those on a LEO satellite. They would also allow for long-duration experiments. On the other hand, the link attenuation and cost are significantly larger for GEO links compared to LEO links. Therefore, when trading GEO-based against LEO-based systems, we would rather accept the more complex PAT system and the limited connection time per orbit and suggest to use a LEO platform for the transmitter terminal for first proof-of-principle experiments.

## V Perspectives and limits of space-aided quantum communications

We determined the typical attenuation for the links indicated in Fig. 3. The values listed in Table 1 were calculated using equ. 7 for wavelengths of 800 nm and of 1550 nm, respectively, assuming telescope apertures characteristic for present day optical space technology. The first wavelength seems reasonable because the best single-photon detectors exist for 800 nm, while the second wavelength is mainly used in standard telecom systems. Applying this wavelength may thus increase compatibility and reduce development effort because commercial off-the-shelf components are available. However, the link attenuation is slightly increased for $`\lambda =`$1550 nm due to the higher absorption in the atmosphere and due to the higher beam divergence at larger wavelength. For each link the default parameters are specified in Appendix C.

Based on present-day technology and assuming reasonable link parameters, it seems feasible to achieve enough entangled photons per receiver pair to demonstrate a quantum communication protocol. For example, assuming a LEO-based transmitter terminal, a simultaneous link to two separate receiving ground stations and a (conservatively estimated) total link attenuation of approx. $`51\text{dB}`$444Note again, that in times without quantum memories, quantum entanglement can only be shared when more than one link is available. For the symmetric case, i.e. two equally long quantum links involving one transmitter and two receivers, this limits the maximal single link attenuation between one transmitter and one receiver to approx. 30 dB. Here we assume a loss of $`25.5\text{dB}`$ for each of the downlinks., one can expect a local count rate of approx. $`2600`$ per second in total at each of the receiver terminals. The number of shared entangled photon pairs is then expected to be approx. $`4`$ per second. For a link duration of 300 seconds this accumulates to a net reception of 1200 entangled qubits. One can expect erroneous detection events on the order of 7 per 100 seconds, which yields a bit error of approx. 2%. This would already allow a quantum key distribution protocol between the two receiver stations. It is thus clear, that a demonstration of basic quantum communication protocols based on quantum entanglement can already be achieved today.

All proposed setups are based on the utilization of entangled photon pairs as carrier of quantum information. Given state-of-the-art technologies present in today’s quantum optics labs we can specify some practical limitations for the preparation and detection of entangled qubits.

The rate of information transfer is limited by the maximal number of photons or entangled photon pairs that can be created and detected. Typical standard repetition rates for pulsed laser sources able to create (entangled) qubit states are in the order of $`10^610^7s^1`$, which will faint out to only a few thousands per second due to optical filtering, finite coupling efficiency and finite detection efficiency. Additionally one has to take into account transmission losses, which can limit the qubit transmission drastically. Also, state of the art detector systems have low dynamic ranges over a maximum of six orders of magnitude 555The highest dynamic range of about six orders of magnitude is achieved with Si avalanche photodiodes (APDs), whereas systems based on InGaAs up to now reach only three to four orders of magnitude. and a maximal detection rate of some MHz 666Some tens of kHz in the case of InGaAs.. Further development of source and detector technology will lead to additional improvements of the qubit rates.

Besides the possibility of establishing a truly global quantum communication network, space-based distribution of quantum entanglement provides us with additional advantages. On the one hand, quantum communication provides means to establish secure and efficient communication. Space communication links certainly match the category of links that should be secure and efficient for several reasons: at first, satellite remote control is a highly sensitive area with respect to security and, up to now, an unsolved technical problem. Secondly, earth-to-satellite communication not only requires considerable expense but is also only possible within limited time-intervals (e.g. in the case of LEO-satellites). Secondly, resources for communication are rare and/or expensive, since active communication segments in space are specified for low power consumption due to limited power resources; therefore, their potential of communication to other (space- or ground-segments) is strictly limited. Deep space communication is a specific example where communication time is limited and resources are expensive and efficient communication is mandatory. Quantum communication complexity could provide a unique means of information transport with very little consumption of communication resources.

On the other hand, the distribution of quantum entanglement will allow to expand the scale for testing the validity of quantum physics by several orders of magnitude. This is a major challenge for future fundamental experiments in quantum physics.

## Appendix A Link requirements

The accidental coincidence rate is given by

$$C_{acc}=S_1S_2\mathrm{\Delta }\tau ,$$
(2)
where $`S_1,S_2`$ are the dark count rates of the two detectors and $`\mathrm{\Delta }\tau `$ is the timing resolution for the electronic registration of a two-fold coincidence event. As the minimum signal to noise ratio we assume that required for violating a Bell-inequality, since this guarantees at the same time the security of certain quantum cryptography schemes Fuchs97 . For the case of polarization-entangled photons this necessitates a two-fold coincidence visibility of at least 71%, corresponding to a signal-to-noise ratio (SNR) of $`6:1`$ 777The visibility, in terms of the signal $`S`$ and the noise $`N`$, is defined by $`(SN)/(S+N)`$.. Below that ratio a local realistic modeling of the observed correlations is possible thus allowing unobserved eavesdropping 888Note, that phase-coded entanglement results in slightly higher requirements to show that no local realistic model describes the corresponding correlations Aerts99 .. Therefore, in order to discriminate the signal from the background coincidences, the minimal observed coincidence rate $`C_{min}`$ must be at least 6 times larger than $`C_{acc}`$.

The coincidence detection rate is determined by the total coincidence efficiency $`\eta _{link}`$, which is the product of the individual efficiencies for the two qubit links,

$$\eta _{link}=\eta _{Link1}\eta _{Link2}.$$
(3)
The detected signal coincidences $`C`$ are given by the product

$$C=P\eta _{link}\eta _{det1}\eta _{det2},$$
(4)
where $`P`$ is the pair production rate of the source and $`\eta _{det1},\eta _{det2}`$ are the detection probabilities. In order to achieve a violation of Bell’s inequality, the signal coincidences must exceed the limit $`C_{min}=SNRC_{acc}`$, which leads to the following limit for the total link efficiency

$$\eta _{link}SNR\frac{C_{acc}}{P\eta _{det1}\eta _{det2}}=SNR\frac{S_1S_2\mathrm{\Delta }\tau }{P\eta _{det1}\eta _{det2}}.$$
(5)
## Appendix B Modeling the link attenuation

We define the link attenuation factor $`A`$ as the ratio of the mean transmit and receive power, $`P_T`$ and $`P_R`$ reyes02\_2 , measured at the entrance and the exit of the transmit and the receive telescope, respectively. Thus losses due to single photon detection efficiency and optical elements such as filters, polarizers or retarders are not included in this number. Then the attenuation factor $`A`$ of a one-way free-space link is thus given by

$$A=\frac{L^2\lambda ^2}{D_T^2D_R^2}\frac{1}{T_T(1L_P)T_R},$$
(6)
where $`L`$ is the link distance, $`\lambda `$ the wavelength, and $`D_T`$ and $`D_R`$ the diameters of the transmit and receive telescope. With $`T_T`$ and $`T_R`$ we denote the transmission factors ($`1`$) of the telescopes, $`L_P`$ is the pointing loss due misalignment of transmitter and receiver. This basic relationship applies (i) if the receiver is in the transmitter’s far field, i.e. $`LD_T^2/\lambda `$, (ii) if the transmit telescope is diffraction limited, and (iii) if there is no influence of the atmosphere.

### Influence of atmosphere

Atmospheric effects on propagation at optical beams can be divided into three categories: absorption, scattering, and turbulence winick86\_1 ; andrews98\_1 . While absorption and scattering mainly depend on wavelength and visibility conditions, the net impact of atmospheric turbulence additionally depends on elevation angle and direction of transmission fried66\_1 . The main effect of atmospheric turbulence is an enlarged beam divergence, resulting in a reduced amount of signal power collected by the receive telescope. Further turbulence-induced effects are beam-wander, loss of coherence, scintillation and pulse distortion and broadening 999Turbulence induced pulse distortion and broadening might actually impose an upper limit to the spectral bandwidth in the pulsed downconversion schemes. Gilbert00 . The effect of turbulence is in general quite different for a space-to-ground link and a ground-to-space link. In a space-to-ground link the light propagates through vacuum for the most of the distance first before being disturbed by the atmosphere, whereas for a ground-to-space link the beam spreading effects of turbulence take place at the beginning of the propagation, causing a strongly enhanced divergence.

### Ground-to-space links

For ground-to-space links we therefore modify Equ. 6 to take into account an additional attenuation of the atmosphere and the influence of turbulence. The diffraction-limited divergence caused by the aperture diameter of the transmit telescope is increased when the beam passes turbulent atmosphere. The influence of the atmosphere can be taken into account by the so-called *Fried parameter*, $`r_0`$, which can be interpreted as an “effective aperture” fried66\_1 . We will assume that the divergence due to turbulence adds quadratically to the divergence of the telescope pfennigbauer02\_2 . The attenuation factor may then be approximated by

$$A=\frac{L^2(\theta _T^2+\theta _{\text{atm}}^2)}{D_R^2}\frac{1}{T_T(1L_P)T_R}10^{A_{\text{atm}}/10},$$
(7)
where $`A_{\text{atm}}`$ is the attenuation of the atmosphere, given in dB. The divergence angle resulting from the transmit telescope is assumed to be

$$\theta _T=\frac{\lambda }{D_T}$$
(8)
and the turbulence causes the additional divergence

$$\theta _{\text{atm}}=\frac{\lambda }{r_0}.$$
(9)
This calculation probably underestimates the turbulence effect 101010In comparison to the values presented in sodnik00\_1 , the divergence obtained with our model is lower by a factor of $`1.5`$. However, we do not know the exact turbulence conditions assumed in sodnik00\_1 . Also, the experimental results of the ARTEMIS–OGS downlink are slightly worse than our calculations would predict., but our model is considered to be suitable to calculate a lower-bound estimation for the attenuation factor.

## Appendix C Default parameters

For the altitude of the LEO satellite we assume 500 km, which thus represents the lower limit of the link distance. For an elevation angle of e.g. 15, the link distance is of some 1400 km 111111Note that for low elevation angles the influence of the atmosphere is increased, a fact not taken into account by the model used here..

Geostationary satellites have an altitude of 36 000 km. Again, the link distance may be larger, depending on the elevation of the satellite (for the ARTEMIS-OGS link the link distance is 41 229 km).

The baseline for the ground aperture is 1 m because this is the telescope diameter of ESA’s optical ground station at Tenerife (OGS). Telescopes with a diameter of 20 to 30 cm are small and light enough to be operated even onboard a small LEO satellite. Larger telescopes are feasible, especially for GEO satellites.

We assume the transmission factors $`T_P`$, $`T_R`$ of the involved telescopes to be $`0.8`$. The pointing loss is $`L_P=0.2`$ for all links except for the LEO-LEO link, where we assume $`L_P=0.3`$ to take into account the – possibly – high relative velocity of the satellites that might result in reduced tracking accuracy.

The assumption of an atmospheric attenuation of $`A_{atm}=1`$ dB applies for excellent sight conditions (no haze, fog, or clouds) and is valid only in certain wavelength regions.

A recently obtained estimate for the Fried parameter, valid for the optical ground station at Tenerife (OGS), is $`r_0=90`$ mm for a wavelength of 800 nm in case of weak turbulence pfennigbauer02\_2 .

For the calculations presented we have assumed a wavelength of 800 nm and 1550 nm. The following tables summarize the link characteristics.

### Table A I: Parameters for ground – LEO and LEO – ground links (default values are underlined)

| link distance | $`L`$ | 500$`\text{to}\mathrm{\hspace{0.33em}1400}\text{km}`$ |
| --- | --- | --- |
| ground aperture | $`D_T,D_R`$ | $`1\text{m}`$ |
| LEO aperture | $`D_T,D_R`$ | $`20\text{to}\mathrm{\hspace{0.33em}30}\text{cm}`$ |
| wavelength | $`\lambda `$ | $`800\text{nm}`$, $`1550\text{nm}`$ |
| telescope transmission factor | $`T_T=T_R`$ | $`0.8`$ |
| pointing loss | $`L_p`$ | $`0.2`$ |
| atmospheric attenuation | $`A_{\text{atm}}`$ | $`1\text{dB}`$ |
| Fried parameter | $`r_0`$ | $`9\text{cm}`$ |

### Table A II: Parameters for Ground – GEO and GEO – Ground links

| link distance | $`L`$ | $`36000\text{km}`$ |
| --- | --- | --- |
| ground aperture | $`D_T,D_R`$ | $`1\text{m}`$ |
| LEO aperture | $`D_T,D_R`$ | $`20\text{to}\mathrm{\hspace{0.33em}30}\text{cm}`$ |
| wavelength | $`\lambda `$ | $`800\text{nm}`$, $`1550\text{nm}`$ |
| telescope transmission factor | $`T_T=T_R`$ | $`0.8`$ |
| pointing loss | $`L_p`$ | $`0.2`$ |
| atmospheric attenuation | $`A_{\text{atm}}`$ | $`1\text{dB}`$ |
| Fried parameter | $`r_0`$ | $`9\text{cm}`$ |

### Table A III: Parameters for LEO – LEO links

| link distance | $`L`$ | $`2000\text{km}`$ |
| --- | --- | --- |
| aperture | $`D_T=D_R`$ | $`20\text{to}\mathrm{\hspace{0.33em}30}\text{cm}`$ |
| wavelength | $`\lambda `$ | $`800\text{nm}`$, $`1550\text{nm}`$ |
| telescope transmission factor | $`T_T=T_R`$ | $`0.8`$ |
| pointing loss | $`L_p`$ | $`0.3`$ |

### Table A IV: Parameters for LEO – GEO links

| link distance | $`L`$ | $`35500\text{km}`$ |
| --- | --- | --- |
| aperture | $`D_T=D_R`$ | $`20\text{to}\mathrm{\hspace{0.33em}30}\text{cm}`$ |
| wavelength | $`\lambda `$ | $`800\text{nm}`$, $`1550\text{nm}`$ |
| telescope transmission factor | $`T_T=T_R`$ | $`0.8`$ |
| pointing loss | $`L_p`$ | $`0.2`$ |

### Table A V: Parameters for GEO – GEO links

| link distance | $`L`$ | $`40000\text{km}`$ |
| --- | --- | --- |
| aperture | $`D_T=D_R`$ | $`20\text{to}\mathrm{\hspace{0.33em}30}\text{cm}`$ |
| wavelength | $`\lambda `$ | $`800\text{nm}`$, $`1550\text{nm}`$ |
| telescope transmission factor | $`T_T=T_R`$ | $`0.8`$ |
| pointing loss | $`L_p`$ | $`0.2`$ |

## Acknowledgment

This paper evolved from a project supported by the European Space Agency under ESTEC/Contract No. 16358/02/NL/SFe, ”Quantum Communications in Space”. We wish to thank Josep Maria Perdigues Armengol for monitoring and supporting this work. We further acknowledge support by the Austrian Science Foundation (FWF) and the Alexander von Humboldt Foundation.


format: text

text:
# Biologically inspired learning in a layered neural net

## I Introduction

In this article we will try to contribute to the study of biological neural networks, in particular with respect to learning. Since we are interested in the basic principles rather than subtle biological details, we will use simplified models, although we do not allow for any properties that are unrealistic from a biological point of view. In this way we hope to reveal the essentials, without blurring the analysis with (irrelevant, biological) details.

It is also not our aim to construct a network that is optimized for some particular task; our only purpose is to study a model that resembles an actual biological neural net and the way it might learn.

In section II we define the model that we will study: a simple feed–forward network with one hidden layer of which not all possible connections are present (i.e., dilution unequal zero). Each neuron of the net has three variables associated with it: a (fixed) threshold potential $`\theta `$, a (variable) membrane potential $`h`$ and a state $`x`$, which is assumed to take two values only, depending on the fact whether the neuron fires or is quiescent.

This article fits into a line of biologically motivated research. Chialvo and Bak bak  suggested learning by punishment only, via the release of some hormone. Heerema et al. leeuwen  derived rules for a biological neural net when it acts as a memory. Bosman et al. bosman  added this rule to the model of Chialvo and Bak, and, thereby, found a significant improvement of the network’s performance.

The dynamics of a neural net is determined by a rule that tells a neuron when to fire. A biological neuron fires if its membrane potential $`h`$ exceeds its threshold $`\theta `$. In the model of Chialvo and Bak this biological rule is replaced by the rule that in each layer a fixed number of neurons, having the highest membrane potentials, fire. They refer to this rule by the name of ‘extremal dynamics’. Extremal dynamics has the drawback that the number of active neurons is artificially fixed, restricting the number of possible output states. Alstrøm and Stassinopoulos alstrom  did not use extremal dynamics, but they ran into the difficulty that the network’s activity became too small or too large for the network to function satisfactorily. In order to keep the activity at a desired level, they adapted the neuron threshold potentials $`\theta `$ at each step of the learning process.

It is one of our purposes to find a way in which a biological net can keep its neuronal activity around an acceptable level. Instead of putting in, by hand, some controlling mechanism, we started from four biologically motivated assumptions (III.1), which lead to the conclusion that only the so–called Hebbian and Anti–Hebbian learning rules are the plausible ones. We show that the Hebbian learning rule fixes and strengthens the action of the network at the moment it is applied, while the Anti–Hebbian learning rule does the opposite, and, when applied repeatedly, will change the network’s action. We conclude that Hebbian learning should be associated with reward, and should be applied if the network realizes the desired output state in reaction to its input, while Anti–Hebbian learning should be associated with punishment, and should be applied when the output of the network is wrong, in order to enable the network to search for better output (III.2).

In section III.3 we propose a particularly simple Anti–Hebbian learning rule (essentially two constants) of which we expect, however, that it will keep the activity of the neural network around a desired value automatically, i.e., without the need of some controlling mechanism. In section V, we perform a number of numerical simulations, the details of which are given in section IV, in order to verify whether our rule is capable of keeping the activity around a desired level. Because the Hebbian learning rule is already studied in leeuwen , we will first focus, in section V, on the effect of the Anti–Hebbian learning rule. Our simulations show that our Anti–Hebbian learning rule indeed is successful in keeping the average activity around a desired level, and, moreover, is very efficient in generating different output states with adequate activities.

Having studied the effect of the Anti–Hebbian learning rule we simulate, in section VI, networks using the complete learning rule, including both the Hebbian and the Anti–Hebbian contributions. We show that the complete learning rule enables the network to learn a number of input–output relations with an acceptable efficiency. Furthermore, it is shown that for non–zero threshold potentials, the performance is only acceptable if the network is sufficiently diluted. So ‘cutting away’ synaptic connections enhances the performance of the network. The observation that – in our model – some degree of non–connectedness is a conditio sine qua non for a properly functioning biological net, is one of our main observations.

The article closes with conclusions (section VII) and an outlook (section VIII).

## II Modeling learning

In this section we describe our model and define quantities like activity and performance.

### II.1 Network model

We consider a feed–forward neural net with one hidden layer (figure 1). The net is taken to be diluted, i.e. not all neurons are connected to all others. There are connections from the input layer to the hidden layer and from the hidden layer to the output layer but there are no direct connections from the input layer to the output layer nor any lateral or feed–back connections.

Let us suppose that the neural net consists of $`N`$ binary neurons $`i`$, $`i=1,2,\mathrm{},N`$. We will use the symbols I, H and O to refer to the Input, Hidden and Output layers. In order to denote that a neuron $`i`$ of the net belongs to one of these layers we will write $`i\text{I}`$, $`i\text{H}`$ or $`i\text{O}`$, respectively. The state $`x_i`$ of neuron $`i`$ either is active ($`x_i=1`$) or is non–active ($`x_i=0`$). The potential $`h_i`$, the difference in potential between the inner and the outer part of a neuron, is supposed to depend linearly on the activities of the incoming synaptic connections:

$$h_i=\underset{jV_i}{}w_{ij}x_j$$
(1)
where $`V_i`$ is the collection of neurons $`j`$ that have an afferent synaptic connection to neuron $`i`$. This formula can be viewed as the defining expression for the weights $`w_{ij}`$. Since the $`x_i`$’s are chosen to be dimensionless, the weights have the dimension of a potential. Let $`\theta _i`$ be the potential that should be exceeded in order that neuron $`i`$ becomes active, i.e.,

$$x_i=1\text{if}h_i>\theta _i,x_i=0\text{if}h_i<\theta _i.$$
(2)
An alternative way to specify the state of neuron $`i`$ as given by equation (2), is to write

$$x_i=\mathrm{\Theta }_\text{H}(h_i\theta _i).$$
(3)
The function $`\mathrm{\Theta }_\text{H}`$ is the Heaviside step–function \[$`\mathrm{\Theta }_\text{H}(y)=0`$ if $`y<0`$ and $`\mathrm{\Theta }_\text{H}(y)=1`$ if $`y>0`$\]. The state of the neurons of the input layer determine the state of the hidden layer via (1) and (3). The state of the hidden layer, in turn, determines the state of the output layer. Thus, if the states $`x_i`$ for $`i\text{I}`$ are given at each time step $`t_n`$, we can determine the network state at every time $`t_n`$, once we have a rule that fixes the $`w_{ij}`$ at time $`t_n`$ ($`n=0,1,2,\mathrm{}`$).

The number of neurons in layer X (X = I, H or O) is denoted by $`N_\text{X}`$. The activity $`a_\text{X}`$ in layer X is defined by

$$a_\text{X}=\frac{1}{N_\text{X}}\underset{i\text{X}}{}x_i,\text{X}=\text{I},\text{H},\text{O}.$$
(4)
The weights of the synapses connecting either the input layer with the hidden layer or the hidden layer with the output layer both are indicated by the symbol $`w_{ij}`$.

In this feed–forward network there are maximally $`N_\text{I}N_\text{H}`$ possible connections from the input layer to the hidden layer. If only the fraction $`(1d_\text{H})`$, $`0d_\text{H}1`$, of these connections to the hidden layer is actually realized, we call $`d_\text{H}`$ the dilution (of the connections with H). Similarly, $`d_\text{O}`$ is the dilution of the connections from the hidden layer to the output layer.

### II.2 Learning input–output relations

Let us denote the state of layer X by $`𝒙_\text{X}=(x_{\text{X}1},x_{\text{X}2},\mathrm{},x_{\text{X}N_\text{X}})`$. We want the network to associate with a particular, prescribed input state $`𝝃_\text{I}^\mu `$, chosen from a collection of $`p`$ input states $`𝝃_\text{I}^\mu `$ ($`\mu =1,\mathrm{},p`$), the particular state $`𝝃_\text{O}^\mu `$, chosen from a collection of $`p`$ prescribed output states $`𝝃_\text{O}^\mu `$ ($`\mu =1,\mathrm{},p`$). The goal of the learning process is that the network will be able to generate, for *all* input patterns $`𝝃_\text{I}^\mu `$, the correct output pattern $`𝝃_\text{O}^\mu `$. This will be achieved by a learning procedure, in which the weights are adapted, stepwise, according to some rule.

If we present a pattern $`𝝃_\text{I}^\mu `$ to the network by setting $`𝒙_\text{I}`$ equal to $`𝝃_\text{I}^\mu `$, the network will respond by generating an output state $`𝒙_\text{O}`$ which, in general, will not be equal to the desired output state $`𝝃_\text{O}^\mu `$. We will associate a variable $`r`$ with each of the following two possibilities: $`r=1`$ if the output is right, i.e., if $`𝒙_\text{O}=𝝃_\text{O}^\mu `$ and $`r=0`$ if the output is wrong, i.e., if $`𝒙_\text{O}𝝃_\text{O}^\mu `$. In practice, we proceed as follows. We present the first input pattern $`𝝃_\text{I}^1`$ to the net, which results in an output state $`𝒙_\text{O}`$. We keep repeating this until $`𝒙_\text{O}`$ equals $`𝝃_\text{O}^1`$, the output pattern to be associated with the input pattern $`𝝃_\text{I}^1`$. The net then has ‘learned’ the first input–output relation $`(𝝃_\text{I}^1,𝝃_\text{O}^1)`$. Next, we present the second input pattern, and continue until the second input–output relation $`(𝝃_\text{I}^2,𝝃_\text{O}^2)`$ has been learned, and so on until the $`p`$–th input–output relation $`(𝝃_\text{I}^p,𝝃_\text{O}^p)`$ has been learned. While doing so, we continuously change the weights according to some $`r`$–dependent rule: the ‘learning rule’.

When an input–output relation has been learned, one or more input–output relations learned earlier may have been forgotten. Therefore, we start a second round, in which we try to learn all $`p`$ input–output relations $`(𝝃_\text{I}^\mu ,𝝃_\text{O}^\mu )`$ again. In order to prevent possible effects due to a specific learning order, the patterns are presented to the network in random order: each round their order is shuffled. After a number of rounds all input–output relations should be recalled at once, i.e., input $`𝝃_\text{I}^\mu `$ should result immediately in output $`𝝃_\text{O}^\mu `$ for all input–output relations.

Do learning rules exist which accomplish all this? The answer is positive bak ; bosman . A particular useful one is the one proposed in section III.

### II.3 Performance

In order to judge the performance of our network, we need some measure.

The a priori probability $`P(N_\text{O}^{(a)})`$ that the state of the output layer $`𝒙_\text{O}`$ equals $`(1,1,\mathrm{},1,0,0,\mathrm{},0)`$, i.e., the state with the first $`N_\text{O}^{(a)}`$ elements equal to one and the remaining last $`N_\text{O}N_\text{O}^{(a)}`$ elements equal to zero, is given by

$$P(N_\text{O}^{(a)})=p_\text{O}^{N_\text{O}^{(a)}}(1p_\text{O})^{N_\text{O}N_\text{O}^{(a)}}$$
(5)
where $`p_\text{O}`$ is the probability that an arbitrary neuron of the output layer is active ($`0p_\text{O}1`$). Note that the probability to find an output pattern with $`N_\text{O}^{(a)}`$ active neurons at arbitrary places is also given by (5). Let $`N_\text{O}^{(a)}(𝝃_\text{O}^\mu )`$ be the number of active neurons in pattern $`𝝃_\text{O}^\mu `$. Then, the average a priori number of trials needed to arrive at the desired output pattern equals $`1/P(N_\text{O}^{(a)}(𝝃_\text{O}^\mu ))`$. If $`p`$ input–output relations $`(𝝃_\text{I}^\mu ,𝝃_\text{O}^\mu )`$ are to be realized, ($`\mu =1,\mathrm{},p`$), the average a priori number $`M_a`$ of trials needed equals

$$M_a=\underset{\mu =1}{\overset{p}{}}1/P(N_\text{O}^{(a)}(𝝃_\text{O}^\mu )).$$
(6)
Comparing the average a priori number of trials and the actual average number of trials $`M`$ needed to learn all $`p`$ input–output relations we get a measure of how well the network performs. This leads us to define the performance $`R`$ as the quotient

$$R:=M_a/M$$
(7)
with $`M_a`$ given by (6). Note that the performance tends to zero when the network is unable to learn all input–output relations ($`M\mathrm{}`$) and the performance will be 1 for an ‘ideal’ network ($`M=M_a`$).

In some models bak ; bosman , the number of neurons that fire is fixed in some way. The a priori probability $`P_{\text{fixed}}(N_\text{O}^{(a)}(𝝃_\text{O}^\mu ))`$ that the output state $`𝒙_\text{O}`$ equals the output pattern $`𝝃_\text{O}^\mu `$ is then given by

$$P_{\text{fixed}}(N_\text{O}^{(a)}(𝝃_\text{O}^\mu ))=\frac{N_\text{O}^{(a)}(𝝃_\text{O}^\mu )!(N_\text{O}N_\text{O}^{(a)}(𝝃_\text{O}^\mu ))!}{N_\text{O}!},$$
(8)
the inverse of the number of ways that a state with $`N_\text{O}^{(a)}`$ active neurons can be realized.

### II.4 Activity distribution

Let us consider an arbitrary collection of $`M`$ neurons, all of which have (independent) probabilities $`p`$ to fire ($`0p1`$). Then the probability $`Q(a)`$ that the activity of this collection of $`M`$ neurons equals $`a`$ is given by

$$Q(a)=\frac{M!}{(Ma)![M(1a)]!}p^{Ma}(1p)^{M(1a)},$$
(9)
where $`Ma`$ is the number of neurons that fire ($`Ma`$ is an integer). This formula differs from (5) by a binomial factor, corresponding to the fact that there are $`M!/\{(Ma)![M(1a)]!\}`$ possible states with the same activity $`a`$. We will use this formula in section V when we compare $`Q(a)`$ to the actual activity of the net.

### II.5 Stability coefficient

Let us define the ‘two time stability coefficient’

$$\gamma _i(t_n,t_m):=\left(2x_i(t_n)1\right)\left(h_i(t_m)\theta _i\right)$$
(10)
which is a generalization of the usual stability coefficient kinzel . With equation (3) it follows that, at two times $`t_n`$ and $`t_m`$ we have

$$\gamma _i(t_n,t_m)>0x_i(t_n)=x_i(t_m)$$
(11)
$$\gamma _i(t_n,t_m)<0x_i(t_n)x_i(t_m).$$
(12)
In particular, if $`n=m`$, we have

$$\gamma _i(t_n,t_n)>0.$$
(13)
The state of neuron $`i`$ does not change if $`\gamma _i(t_n,t_m)`$ is positive. Moreover, the larger $`\gamma _i(t_n,t_n)`$, the more stable the system is with respect to changes in the synaptic weights krauth ; gardner . The $`\gamma `$’s will play a role in section III.2.

## III Determining the learning rule

As noted in the introduction, many neural networks have a tendency to develop into a state where most neurons are either all active or non–active. Chialvo and Bak bak  and Alstrøm and Stassinopoulos alstrom  solved this problem in a biologically less plausible way: the authors do not consider the threshold potentials at all bak , or allow them to grow indefinitely alstrom . We approach the problem in a different way: we start from four biologically plausible restrictions on the learning rule. This leads to the conclusion that the learning rule can be seen as a superposition of two types of learning, Hebbian and Anti–Hebbian learning. Taking for the Anti–Hebbian part the most simple ansatz one can think of in dependence of the state $`x_i`$ of the postsynaptic neuron $`i`$, we arrive at a learning rule that, a priori, is biologically plausible. Moreover, it turns out that the learning rule found in this way is such that the neuronal activity is well–behaved.

### III.1 Biologically plausible learning rules

Let us suppose that the weights $`w_{ij}(t_n)`$ may be changed stepwise by some learning process

$$w_{ij}(t_{n+1})=w_{ij}(t_n)+\mathrm{\Delta }w_{ij}(t_n)$$
(14)
where $`j`$ is the presynaptic and $`i`$ the postsynaptic neuron. The changes, given by $`\mathrm{\Delta }w_{ij}(t_n)`$, are the quantities we are after.

We start from four assumptions, each of which needs not be strictly true, but each of which is very plausible biologically, at least in first approximation. The contents of these assumptions generalize the assumptions of Heerema and Van Leeuwen leeuwen  by including the effect of the feed–back signal $`r`$.

* The changes in $`w_{ij}`$, $`\mathrm{\Delta }w_{ij}`$, depend on the global variable $`r`$ and the local variables $`x_i`$, $`x_j`$, $`h_i`$, $`\theta _i`$ and $`w_{ij}`$, i.e.,

$$\mathrm{\Delta }w_{ij}=\mathrm{\Delta }w_{ij}(r,x_i,x_j,h_i,\theta _i,w_{ij}).$$
(15)
Biologically this means that only variables that can be ‘felt’ at the synapse between $`j`$ and $`i`$ can influence the weight change. Of course, this includes the global variable $`r`$, which could be realized in an actual biological system by some chemical substance released throughout the brain, and the strength of the weight $`w_{ij}`$ itself. Since the state $`x_j`$ of neuron $`j`$ determines whether or not neurotransmitters are released at the synapse, $`x_j`$ can influence the weight change. Because the synapse is located at the dendrites or cell body of neuron $`i`$, we suppose that variables local to neuron $`i`$, the state $`x_i`$ and the potentials $`h_i`$ and $`\theta _i`$, also can influence the strength of the weight change.
* The *sign* of $`\mathrm{\Delta }w_{ij}`$ depends on $`r`$, and the neuron states $`x_i`$ and $`x_j`$ *only*, i.e.,

$$\mathrm{\Delta }w_{ij}=\sigma (r,x_i,x_j)\epsilon _{ij}(r,x_i,x_j,h_i,\theta _i,w_{ij})$$
(16)
where $`\sigma `$ equals $`1`$ or $`+1`$ and where $`\epsilon _{ij}0`$.

Biologically this means that as long as the states of the presynaptic and postsynaptic neurons do not change *and* the global feed–back signal $`r`$ does not change, the sign of the weight change will not change, i.e., the variables $`h_i`$, $`\theta _i`$ and $`w_{ij}`$ can influence the *magnitude* of the weight change, but they cannot switch the learning from an increase to a decrease as long as the states and $`r`$ do not change.
* There is only a change in $`w_{ij}`$ if the presynaptic neuron $`x_j`$ is active, i.e.,

$$\epsilon _{ij}=\epsilon _{ij}(r,x_i,h_i,\theta _i,w_{ij})x_j.$$
(17)
Biologically this means that if the presynaptic neuron $`j`$ does not fire, there will be no change in the synaptic efficacy $`w_{ij}`$. This is supposed because we think that it is unlikely that the synaptic efficacy will change if nothing happens, i.e., no neurotransmitters are released into the synaptic gap. Substituting (17) into (16) we find

$$\mathrm{\Delta }w_{ij}=\sigma (r,x_i)\epsilon _{ij}(r,x_i,h_i,\theta _i,w_{ij})x_j,$$
(18)
where we replaced $`\sigma (r,x_i,x_j)`$ by $`\sigma (r,x_i)`$, which is allowed since $`\mathrm{\Delta }w_{ij}`$ is only non–zero if $`x_j`$ equals $`1`$.
* Both in case $`r=0`$ and $`r=1`$, there is not only enhancement or only diminishment of the weights. This implies that $`\sigma (r=0,x_i)`$ must take on both the values $`+1`$ and $`1`$. The same is true for $`\sigma (r=1,x_i)`$. Since $`x_i`$ is the only variable that is still available to influence $`\sigma `$, we must have $`\sigma (r,x_i=0)=\sigma (r,x_i=1)`$, or, equivalently

$$\sigma (r,x_i)=\sigma (r)(2x_i1)$$
(19)
where $`\sigma (r)`$ equals $`1`$ or $`+1`$.

Biologically, this means that we think that it is implausible that weights can either only increase or only decrease. If the weights would only increase, all the membrane potentials (1) would only increase, and all neurons would end up firing \[see eq. (3)\]. In case the weights would only decrease, all membrane potentials would only decrease, and all neurons would become non–active. Since $`r=0`$ can be true for long times as long as something is not learned, and $`r=1`$ can also last for long times as long as the behavior is as desired, the assumption must hold true both for $`r=0`$ and $`r=1`$.

Inserting (19) into (18) we finally have

$$\mathrm{\Delta }w_{ij}=\sigma (r)\epsilon _{ij}(r,x_i,h_i,\theta _i,w_{ij})(2x_i1)x_j,$$
(20)
where $`\sigma (r)=\pm 1`$ (where $`r`$ can take on the values $`0`$ and $`1`$) and $`\epsilon _{ij}0`$.

### III.2 Implementing reward and punishment

We will now discuss the effects of the weight changes (20) for $`\sigma (r)=+1`$ and $`\sigma (r)=1`$. To that end, consider neuron $`i`$, a fixed but arbitrary neuron of the network. Suppose that the neurons $`jV_i`$ do not change their states during the time step $`t_nt_{n+1}`$:

$$x_j(t_{n+1})=x_j(t_n),jV_i$$
(21)
Now, we multiply both sides of (14) by $`x_j(t_n)`$, sum over all indices $`jV_i`$, and subtract the threshold potential $`\theta _i`$ to obtain

$$h_i(t_{n+1})\theta _i=h_i(t_n)\theta _i+\underset{jV_i}{}\mathrm{\Delta }w_{ij}(t_n)x_j(t_n)$$
(22)
where we used (1) and (21). Next, we multiply by the factor $`(2x_i(t_n)1)`$. Using equation (10), we then get

$$\gamma _i(t_n,t_{n+1})=\gamma _i(t_n,t_n)+\underset{jV_i}{}\mathrm{\Delta }w_{ij}(t_n)(2x_i(t_n)1)x_j(t_n).$$
(23)
We now distinguish between $`\sigma (r)=+1`$ and $`\sigma (r)=1`$.

Substituting (20) into (23) with $`\sigma (r)=+1`$ we find, using $`(2x_i1)^2=1`$ and $`x_j^2=x_j`$, that

$$\gamma _i(t_n,t_{n+1})=\gamma _i(t_n,t_n)+\underset{jV_i}{}\epsilon _{ij}x_j(t_n).$$
(24)
Recalling that $`\gamma _i(t_n,t_n)`$ and $`\epsilon _{ij}`$ are positive, we see that $`\gamma _i(t_n,t_{n+1})`$ is positive. Hence, according to (11) the state of neuron $`i`$ has not changed:

$$x_i(t_{n+1})=x_i(t_n).$$

In our case of a feed–forward network, the neurons of the hidden layer get their input from neurons of the input layer only. Consequently, if the input layer does not change, going from time $`t_n`$ to time $`t_{n+1}`$, the state of a neuron $`i`$ of the hidden layer does not change. This holds true for any neuron $`i`$ of the hidden layer, implying that the input of the output layer will not change either. In other words, the output of the net does not change when the input remains the same, although the weights $`w_{ij}(t_n)`$ change to $`w_{ij}(t_{n+1})`$ according to the rule (14). Hence, the rule (20) with $`\sigma (r)=+1`$ conserves an input–output relation.

Moreover, since $`\gamma _i(t_n,t_{n+1})`$ is larger than $`\gamma _i(t_n,t_n)`$, as follows from (24), the new net is more stable. In other words, learning with $`\sigma (r)=+1`$ engraves an input–output relation into the memory of the net by properly adapting its weights.

Next, substituting (20) into (23), but now with $`\sigma (r)=1`$, we find

$$\gamma _i(t_n,t_{n+1})=\gamma _i(t_n,t_n)\underset{jV_i}{}\epsilon _{ij}x_j(t_n).$$
(25)
Since both $`\gamma _i(t_n,t_n)`$ and $`\epsilon _{ij}`$ are positive, $`\gamma _i(t_n,t_{n+1})`$ is smaller than $`\gamma _i(t_n,t_n)`$. Hence, the learning rule (20) with $`\sigma (r)=1`$ has the effect of decreasing the stability of the network. As long as the state of the network does not change (i.e. $`x_j(t_{n+1})=x_j(t_n)`$ for all $`j`$), all stability coefficients $`\gamma _i(t_n,t_{n+1})`$ decrease, and, at a certain moment $`t_m`$ ($`m>n`$), $`\gamma _i(t_m,t_{m+1})`$ will become negative, at least for one neuron $`i`$, implying, with (12), that

$$x_i(t_{m+1})x_i(t_m).$$

Consequently, repeated learning with $`\sigma (r)=1`$ will result in a change of the output related to the same input.

We now come to the main conclusion of this section. If the network output is the wrong one ($`r=0`$), we must adapt the weights such that other output results, i.e., we should use $`\sigma (0)=1`$. If the network output is the right one ($`r=1`$), we should use $`\sigma (1)=+1`$ to consolidate this situation. Hence, we may conclude

$$\sigma (r)=2r1.$$
(26)
With (26) we may write instead of (20)

$$\mathrm{\Delta }w_{ij}=r\mathrm{\Delta }w_{ij}^\text{H}+(1r)\mathrm{\Delta }w_{ij}^\text{A}$$
(27)
where

$`\mathrm{\Delta }w_{ij}^\text{H}`$ $`=`$ $`+\epsilon _{ij}^\text{H}(x_i,h_i,\theta _i,w_{ij})(2x_i1)x_j`$ (28)
$`\mathrm{\Delta }w_{ij}^\text{A}`$ $`=`$ $`\epsilon _{ij}^\text{A}(x_i,h_i,\theta _i,w_{ij})(2x_i1)x_j`$ (29)
with

$`\epsilon _{ij}^\text{H}`$ $`=`$ $`\epsilon _{ij}(r=1,x_i,h_i,\theta _i,w_{ij})`$ (30)
$`\epsilon _{ij}^\text{A}`$ $`=`$ $`\epsilon _{ij}(r=0,x_i,h_i,\theta _i,w_{ij}).`$ (31)
If both the presynaptic and postsynaptic neurons are active ($`x_j=x_i=1`$), the terms $`\mathrm{\Delta }w_{ij}^\text{H}`$ and $`\mathrm{\Delta }w_{ij}^\text{A}`$ are positive and negative, respectively. We will refer to them by the names of Hebbian and Anti–Hebbian learning. In conclusion, we see from (27) that reward ($`r=1`$) and punishment ($`r=0`$) may be associated in a unique way with Hebbian and Anti–Hebbian learning, respectively. A similar rule has been postulated by Barto and Anandan barto .

What remains is to find explicit expressions for the (positive) functions $`\epsilon _{ij}^\text{H}`$ and $`\epsilon _{ij}^\text{A}`$. This will be the subject of the next section.

### III.3 Determining explicit rules

For the Hebbian function $`\epsilon _{ij}^\text{H}`$ we choose the biologically plausible expression derived in leeuwen :

$$\epsilon _{ij}^\text{H}=\eta _i\left[\kappa (h_i\theta _i)(2x_i1)\right]$$
(32)
where $`\eta _i`$ and $`\kappa `$ are constants. Note that $`\kappa `$ must be large enough in order that $`\epsilon _{ij}^\text{H}`$ be positive.

We now come to the Anti–Hebbian function $`\epsilon _{ij}^\text{A}(x_i,h_i,\theta _i,w_{ij})`$. Experimentally nor theoretically there are clues regarding the precise form of this term. Therefore, we simply choose two (positive) constants, $`c_i^{(1)}`$ and $`c_i^{(2)}`$: $`\epsilon _{ij}^\text{A}(x_i=0)=c_i^{(1)}`$ and $`\epsilon _{ij}^\text{A}(x_i=1)=c_i^{(2)}`$. Since any two positive constants can be expressed as $`c_i^{(1)}=\rho _i\alpha _i`$ and $`c_i^{(2)}=\rho _i(1\alpha _i)`$ where $`\rho _i`$ and $`\alpha _i`$ are two other constants with $`0<\alpha _i<1`$ and $`\rho _i>0`$, we can write

$$\epsilon _{ij}^\text{A}(x_i=0)=\rho _i\alpha _i,\epsilon _{ij}^\text{A}(x_i=1)=\rho _i(1\alpha _i)$$
(33)
or, equivalently,

$$\epsilon _{ij}^\text{A}=\rho _i\left[\frac{1}{2}(\alpha _i\frac{1}{2})(2x_i1)\right].$$
(34)
Note that the expression (34) can be obtained from (32) by the substitutions $`\eta _i\rho _i`$, $`h_i\alpha _i`$, $`\kappa \frac{1}{2}`$ and $`\theta _i\frac{1}{2}`$. However, for (32) exists a derivation, whereas (34) is an educated guess only.

Upon substituting (32) and (34) into (28) and (29) we obtain

$`\mathrm{\Delta }w_{ij}^\text{H}`$ $`=`$ $`\eta _i\left[\kappa (2x_i1)(h_i\theta _i)\right]x_j`$ (35)
$`\mathrm{\Delta }w_{ij}^\text{A}`$ $`=`$ $`\rho _i(x_i\alpha _i)x_j.`$ (36)
The numerical study of section V will show that the effect of the Anti–Hebbian term (36) is that the average neuronal activity of the network takes a value controlled by the parameter $`\alpha `$.

## IV Simulating the network

This section is intended mainly for readers interested in technical details of the simulations.

### IV.1 Determining parameter values

Up to now we did not specify the parameters $`\eta _i`$, $`\kappa `$, $`\theta _i`$, $`\rho _i`$ and $`\alpha _i`$ occurring in the learning rule (27) combined with (35) and (36). In leeuwen  it is argued that the Hebbian learning rate $`\eta _i`$ should be proportional to the inverse of the average number of neurons $`jV_i`$ that fire. It is argued also that a reasonable approximation will suffice. Therefore, we may choose $`\eta _i`$ equal to $`\eta _\text{X}`$ for all $`i\text{X}`$, i.e., we may choose $`\eta _i`$ the same for all neurons $`i`$ of layer X ($`\text{X}=\text{H},\text{O}`$):

$$\eta _i=\eta _\text{H},i\text{H},\eta _i=\eta _\text{O},i\text{O}$$
(37)
with

$$\eta _\text{H}=\eta \frac{1}{\overline{a_\text{I}}N_\text{I}(1d_\text{H})},\eta _\text{O}=\eta \frac{1}{\overline{a_\text{H}}N_\text{H}(1d_\text{O})}$$
(38)
where the bar denotes a time average and where $`\eta `$ is some positive constant, which we will call the (global) learning rate. Note that $`\eta _\text{H}`$ is the learning rate associated with the connections from the input layer I to the hidden layer H. Similarly, $`\eta _\text{O}`$ is associated with connections from H to O.

Analogously we take for the Anti–Hebbian learning rate, or punishment rate, $`\rho `$:

$$\rho _i=\rho _\text{H},i\text{H},\rho _i=\rho _\text{O},i\text{O}$$
(39)
with

$$\rho _\text{H}=\rho \frac{1}{N_\text{I}\overline{a_\text{I}}(1d_\text{H})},\rho _\text{O}=\rho \frac{1}{N_\text{H}\overline{a_\text{H}}(1d_\text{O})}.$$
(40)
We will set the parameters $`\theta _i=\theta _\text{X}`$, $`i\text{X}`$ and $`\alpha _i=\alpha _\text{X}`$, $`i\text{X}`$, i.e., we take these parameters the same also for neurons belonging to one and the same layer.

The margin parameter $`\kappa `$ will be fixed at the value 1, in agreement with literature (see e.g. leeuwen ). This can be done, because, instead of varying $`\kappa `$, one can also vary both the learning and punishment rates $`\eta `$ and $`\rho `$, with the same effect.

### IV.2 Addition of noise

Since the equations describing the network dynamics are deterministic, and the number of possible states of the network is finite, the system may suffer from periodic behavior, which is not realistic biologically, since in an actual biological net there is always some disturbing effect. Therefore, when performing our simulations, we add some noise, in order to mimic reality.

To that end we employ the Gaussian distribution

$$F(x,\mu ,\sigma )=\frac{1}{\sigma \sqrt{2\pi }}e^{(x\mu )^2/2\sigma ^2}$$
(41)
with mean $`\mu `$ and standard deviation $`\sigma `$. In our simulations we will replace the deterministic value $`\mathrm{\Delta }w_{ij}`$ by the probabilistic value $`\mathrm{\Delta }w_{ij}^{(\text{noisy})}`$, the distribution of which is given by

$$P(\mathrm{\Delta }w_{ij}^{(\text{noisy})})=F(\mathrm{\Delta }w_{ij}^{(\text{noisy})},\mu ,\sigma )$$
(42)
with

$$\mu =\mathrm{\Delta }w_{ij},\sigma =\mathrm{\Delta }w_{ij}\delta $$
(43)
i.e., we replaced $`\mathrm{\Delta }w_{ij}`$ by a Gaussian distributed quantity $`\mathrm{\Delta }w_{ij}^{(\text{noisy})}`$ with mean value $`\mathrm{\Delta }w_{ij}`$ and standard deviation $`\mathrm{\Delta }w_{ij}\delta `$. Note that we have chosen the standard deviation of the noise proportional to $`\mathrm{\Delta }w_{ij}`$ with proportionality constant $`\delta `$.

### IV.3 Initializing the network

As long as a network does not memorize input–output relations, the Anti–Hebbian term of the learning rule will adapt the weights in such a way that all neurons will ‘hesitate’ between firing and not firing. When the Anti–Hebbian learning rule is applied for a long time, the network can be viewed as ‘fresh’: the network has not stored any information and can quickly change its behavior. We want to start our simulations with networks that have their weights distributed according to such a fresh distribution. This can be accomplished by starting with arbitrary weights and applying the Anti–Hebbian learning rule a large number of times. We will use this approach, and, in every step, offer an arbitrary input pattern with activity $`a_\text{I}`$ to the net.

Since it may take a very long time before the distribution of all weights has reached its equilibrium by the effect of the Anti–Hebbian learning rule, we will not start with entirely arbitrary weights, but, instead, start from a rough approximation. We choose to initialize with weights that are distributed according to the Gaussian distribution (41). For weights connecting neurons of the input layer to neurons of the hidden layer, the changes of the weights due to the Anti–Hebbian learning rule will be of the order of $`\rho _\text{H}`$. Therefore, we suppose that the width of the distribution of the weights will also be of the order of $`\rho _\text{H}`$, and we take

$$\sigma =\frac{1}{2}\rho _\text{H}$$
(44)
for the standard deviation of the initial Gaussian distribution. For the mean $`\mu `$ we will use a value $`w_\text{H}`$, which is a rough approximation of the average value of the strength of the weights connecting neurons of the input layer to neurons of the hidden layer. Since the Anti–Hebbian learning rule causes each neuron to change its state $`x_i`$ over and over again, the membrane potentials $`h_i`$ will fluctuate around the threshold potentials $`\theta _i`$. As an approximation we suppose that each membrane potential $`h_i`$ of a neuron of the hidden layer will be, on average, equal to its threshold potential:

$$h_i\theta _\text{H},i\text{H}.$$
(45)
We now approximate $`h_i`$, given by (1), according to

$$h_iw_\text{H}\underset{jV_i}{}x_j.$$
(46)
The sum in this formula can be approximated by

$$\underset{jV_i}{}x_j\overline{a_\text{I}}N_\text{I}(1d_\text{H}),$$
(47)
with $`\overline{a_\text{I}}`$ the average activity of the input layer, which, in our model, will be equal to the average activity of the input patterns. Furthermore, $`N_\text{I}(1d_\text{H})`$ is an approximation for the number of presynaptic neurons. Substituting (47) and (45) into (46), we obtain a rough approximation of the average value of the strengths of the weights $`w_\text{H}`$:

$$w_\text{H}\frac{\theta _\text{H}}{\overline{a_\text{I}}N_\text{I}(1d_\text{H})}$$
(48)
In the same way, we find for the weights connecting the neurons of the hidden layer to the neurons of the output layer

$$w_\text{O}\frac{\theta _\text{O}}{\overline{a_\text{H}}N_\text{H}(1d_\text{O})}$$
(49)
where $`\overline{a_\text{H}}`$ is the average activity of the hidden layer. As will turn out later, this average activity will be around the parameter $`\alpha _\text{H}`$ occurring in the Anti–Hebbian learning rule. We will use $`\alpha _\text{H}`$ as an estimation for $`a_\text{H}`$ in the above formula. Analogous with (44), $`\frac{1}{2}\rho _\text{O}`$ will be used for the standard deviation $`\sigma `$ for the weights connecting neurons of the hidden layer to neurons of the output layer.

## V Effect of the Anti–Hebbian component

It proves useful to study the Hebbian and Anti–Hebbian learning rules separately. Since the Hebbian rule has been studied earlier, we here consider Anti–Hebbian learning; in the next section the complete rule will be discussed.

We will show that the activities, averaged in time, of the hidden and output layers are given by $`\alpha _\text{H}`$ and $`\alpha _\text{O}`$ respectively. Moreover, we will show that the distribution of the activities around the value $`\alpha _\text{H}`$ (or $`\alpha _\text{O}`$) is such that, effectively, each neuron $`i`$ in layer H (or O) has a probability $`\alpha _\text{H}`$ (or $`\alpha _\text{O}`$) to fire, independent of the activities of the other neurons. In other words, the proposed learning rule (36) focuses the average activity in a natural way around the values $`\alpha _\text{H}`$ and $`\alpha _\text{O}`$.

In case the threshold potentials $`\theta _\text{H}`$ and $`\theta _\text{O}`$ vanish, we will find that the state of a neuron in the output layer behaves independently of the states of the other neurons of the output layer. In case the threshold potentials do not vanish, we will observe strong correlations between the states of the output neurons, which vanish, however, when the dilution is taken to be sufficiently high.

### V.1 Vanishing threshold potentials

We start by studying the case that $`\theta _\text{H}=\theta _\text{O}=0`$ and $`d_\text{H}=d_\text{O}=0`$. In order to get a first impression of the network behavior, we will plot the activity as a function of time, as well as, in a histogram, the distribution of its values ($`0a1`$). We will offer the network $`p=1000`$ input patterns. Each input pattern will be repeatedly offered to the net until the output pattern to be associated with the input pattern is found. The input and associated output patterns are chosen at random but with a certain specified activity. As soon as the correct output is produced, the next pattern is presented until all output patterns associated with the input patterns have been found.

In figure 2 (top) we can observe that the activity $`a_\text{H}`$ of the hidden layer fluctuates around the value $`\alpha _\text{H}=0.05`$, the value which we have chosen in the learning rule for changes of weights of connections between the input and hidden layers. This is what we had hoped to achieve when we postulated (36). The larger fluctuations occur when the net is confronted with a new input pattern, to which it must learn to react by a new, prescribed output pattern. We see that it takes only a short period of time before the net has found back its balance. Similar observations can be made with respect to the output layer: in figure 2 (bottom) the activity is seen to fluctuate around the value $`\alpha _\text{O}=0.3`$, chosen in the learning rule (36). It can also be seen that the distributions closely resemble the distribution $`Q(a)`$ \[eq. (9)\]. This means that the Anti–Hebbian learning rule effectively causes each neuron to behave and fire, with a probability $`a_\text{X}`$, as if it is independent of the states of the other neurons.

### V.2 Non–vanishing threshold potentials

In this section we study a network that is the same as in the preceding section, but with non–vanishing threshold potentials: we will take $`\theta _\text{H}=\theta _\text{O}=1`$. When we now repeat the simulations, we find one important difference: the neurons of the output layer turn out to exhibit strong correlations: almost always, either all of them are active or all of them are non–active. This can be seen in figure 3 (left). This is alarming and most undesirable, since it would imply that the network has, effectively, only two different output states. Note, that although the probability that $`a_\text{O}`$ equals $`\alpha _\text{O}`$ now is very small, its average $`\overline{a_\text{O}}`$ is still close to $`\alpha _\text{O}`$.

It can be expected that if the average number of neurons of some layer $`\text{X}_B`$ that are postsynaptic to a neuron of a preceding layer $`\text{X}_A`$ becomes lower, the correlated behavior of the neurons of $`\text{X}_B`$, which, apparently, was induced by the neurons of $`\text{X}_A`$, will decrease. In order to verify this, we now study a network with non–vanishing dilution. In figure 3 (right), we plotted the distribution of the activity $`a_\text{O}`$ of the output layer for five different values of the dilution $`d_\text{O}`$. Comparing the left and right pictures of figure 3 we observe a dramatic change in the behavior indeed: the undesired effect quickly diminishes with increasing dilution.

Note that the distribution is shifted somewhat towards zero in comparison with the distribution of uncorrelated output neurons. The shift decreases when the number of neurons of the hidden layer increases: for $`N_\text{H}=20,000`$ there is a better resemblance to the $`Q(a)`$–curve than for $`N_\text{H}=2,000`$. For non–zero dilution the probability that none of the neurons that are presynaptic to an output neuron fire increases, resulting in a lower average activity. This causes the shift of the activity distribution to lower values.

## VI Testing the full learning rule

In the preceding section we saw that the Anti–Hebbian part of the learning rule adapts the weights in such a way that a desired input–output relation is found after a number of time steps, while, at the same time the network activities $`a_\text{H}`$ and $`a_\text{O}`$ stay within acceptable bounds. In this section we study the complete learning rule (27) with (35) and (36).

### VI.1 Comparing to other models

In order to make contact with existing literature, we also simulate systems with extremal dynamics. We consider input–output relations with the same number of active neurons $`N_\text{I}^{(a)}=N_\text{O}^{(a)}=N^{(a)}`$ in the input and output patterns. In figure 4 we plotted the performance $`R`$ of a network as a function of the number of input–output relations $`p`$ to be learned, for $`N^{(a)}=1`$, $`N^{(a)}=2`$ and $`N^{(a)}=3`$. The two pictures at the top are for the case of extremal dynamics, the pictures at the bottom correspond to simulations for the more realistic case that the neurons fire if their membrane potentials exceed their thresholds. The two pictures at the left are without reward ($`\eta =0`$), those at the right are with reward ($`\eta 0`$).

We conclude that the net is able to learn a number of input–output relations indeed. The situation $`N^{(a)}=1`$ (top, left) is analogous to the case considered by Chialvo and Bak. In the simple case that $`N^{(a)}=1`$, the performance is almost perfect, i.e., $`R`$ is close to 1. However, the performance decreases quickly for the case that more than two neurons are active in the input and output patterns. In figure 4 (top, right) we observe that rewarding significantly increases the performance $`R`$ if $`N^{(a)}`$ is larger than 1. A similar observation has been made earlier by Bosman et al. bosman .

Figure 4 (left, bottom) is to be compared with figure 4 (left, top). For $`N^{(a)}=1`$, the performance is not as good as in case of extremal dynamics, but it still works satisfactory. However, for values of $`N^{(a)}`$ larger than one, the performance is very bad. Adding reward to the learning rule, we find the results of figure 4 (right, bottom). The improvement of the performance is impressive. Hence, our model performs the task of realizing prescribed input–output relations reasonably well, although its performance is not as good as in the less realistic case that extremal dynamics is used.

We close this section with the following observation: In case only one input–output relation is to be learned by the net, i.e., $`p=1`$, the performance $`R`$ is close to 1, as can be read off from figure 4. This implies that the period of search for the correct output by means of the Anti–Hebbian term is close to the random search time. In other words, in case the feed–back is binary only ($`r=0`$ or $`r=1`$), Anti–Hebbian learning enables a way of search which is close to optimal.

### VI.2 Numerical experiments on the influence of parameters

The performance of a neural net depends on many parameters, e.g., the coefficients $`\eta `$, $`\rho `$ and $`\alpha `$ occurring in the learning rule, the dilution $`d`$ and the threshold potential $`\theta `$. In this section we take the thresholds $`\theta _\text{X}=0`$ ($`\text{X}=\text{I},\text{H},\text{O}`$). In order to get more insight regarding the behavior of the net in dependence of variations of all these parameters, we will study two particular cases.

In figure 5 we plotted the performance $`R`$ of the net as a function of the parameter $`\alpha _\text{H}`$, the coefficient occurring in the Anti–Hebbian part of the learning rule, which determines the activity of the hidden layer. There is seen to be an optimum in the performance. For values of $`\alpha _\text{H}`$ below $`0.02`$, the performance decreases rapidly. It follows that the parameter $`\alpha _\text{H}`$ occurring in the Anti–Hebbian part of the learning rule, a neuron property, has an important effect on the performance of the net. It is seen that the performance diminishes when the activity becomes too large, in agreement with earlier results bak .

In figure 6 we plotted the performance $`R`$ as a function of $`\eta /\rho `$ for $`\rho =0.05`$. For values of $`\eta `$ around two times the value of $`\rho `$, the performance is optimal. We conclude that when the rewarding part of the learning rule (proportional to $`\eta `$) becomes smaller and smaller, the performance of the net strongly decreases. On the other hand, if the rewarding part becomes larger and larger, the performance also diminishes, albeit more slowly. These observations do not come as a surprise: evidently, if the memorizing Hebbian term is too weak – relative to the Anti–Hebbian term – learning will be slow, whereas it will also be slow if the Hebbian term becomes too strong, since then the learning of any pattern will change the existing connections too wildly.

### VI.3 Non–vanishing threshold potentials

In view of section V.2 we expect that also in case of use of the full learning rule (27) with (35) and (36) we will only get satisfactory results if we take the dilution $`d`$ unequal to zero. In figure 7 the performance is plotted as a function of the number of input–output relations $`p`$ for different values of the dilution $`d_\text{O}`$. The network performance is optimal around $`d_\text{O}=0.9`$. The fact that a dilution here actually enhances the performance is a consequence of the fact that undesired correlations (see section V.2) diminish with increasing dilution.

## VII Conclusions

Our goal to model, in a biologically plausible way, neural networks capable of learning without the use of extremal dynamics bak  or some other mechanism to control the neuronal activity was reached successfully.

In section III.1 we found, on the basis of four biological assumptions, two possible learning rules, given by the equations (28) and (29). By analyzing their effects, we were able to associate them in a unique way with reward and punishment, and to formulate a plausible learning rule (27). For the rewarding part, we used a form derived in leeuwen . for the punishing part we chose two constants \[see eq. (33)\]. By studying the effects of this specific form with two constants, we found that this punishing part of our learning rule is able to control the average activity (average fraction of firing neurons) in a neuronal net. We showed that the activity remained around a desired level while, at the same time, the network is searching for the correct output by generating random patterns with a desired activity with great efficiency.

Finally, we showed that for neurons with non–zero threshold potentials, the neural net must be diluted in order to achieve a reasonable performance: dilution is found to enhance the functionality of a biological neural net. This is because the neurons start to behave correlated when the threshold potentials are non–zero. When the dilution increases, these undesired correlations decrease. In nature neural nets normally are strongly diluted: the human brain, for example, has of the order of $`10^{11}`$ neurons but there are only $`10^4`$$`10^5`$ synapses per neuron.

## VIII Outlook

Evidently, the study on the behavior of biologically inspired neural networks is far from complete.

One may, for instance, include more biologically known features into the model like time delay of the axonal signal or the refractory period of a neuron. Also, the architecture of the net could be made more realistic by including more layers and adding feed–back and lateral connections. Furthermore, the model of the surrounding world, i.e., the input–output relations to be learned could be made more realistic, for example by using a dynamic model of a realistically changing world. Also, the neuron model could be made to resemble closer real neurons by including different types of neurons and synapses, or using the fact that excitatory synapses are probably more plastic than the inhibitory ones. Another extension would be to refine the measure of success. Instead of just a binary feed–back signal indicating whether the output is *right* or *wrong* in reaction to some input, a feed–back signal that can take on a range of values is possible. Also, a relative measure could be used, telling the network if it has performed better or worse than during a previous attempt.

A most obvious first extension of our particular model would be to include (inhibitory) lateral connections inside layers and possibly also feed–back connections. It would be interesting to study the influence of these non–feed–forward connections on the behavior of the network, and, especially, the effect they will have on the (average) activity of the network. Another extension of our model could be to associate Hebbian and Anti–Hebbian learning with different types of neurons (or synapses) instead of letting the same neurons behave differently under different conditions of success or failure. Such a model could have the advantage that the Hebbian connections, in which the input–output relations are memorized on success, would not have to be changed on failure. Thus, instead of changing the same connections, different, Anti–Hebbian connections could possibly do the job of searching for successful output patterns, without changing the already learned input–output relations.

As a final remark, we mention that different extensions or alterations are possible to enhance the performance of the network, which may, however, be implausible biologically. For example, if an input–output relation would not be strengthened over and over again once it has been learned, the network generally would perform better.

###### Acknowledgements.

The authors are indebted to Bastian Wemmenhove for a number of very useful suggestions.