text:
# The long memory of the efficient market

## I Introduction

Roughly speaking, a random process is said to have long-memory if it has an autocorrelation function that is not integrable. This happens, for example, when the autocorrelation function decays asymptotically as a power law of the form $`\tau ^\alpha `$ with $`\alpha <1`$. This is important because it implies that values from the distant past can have a significant effect on the present, and implies anomalous diffusion in a stochastic process whose increments have long-memory. We present a more technical discussion of long-memory processes in Section IV.

In this paper we make use of a data set from the London Stock Exchange (LSE), which contains a full record of individual orders and cancellations, to show that stocks in the LSE display a remarkable long-memory property111Bouchaud et al. (2004) independently discovered the long-memory property of order signs for stocks in the Paris exchange. We thank them for acknowledging the oral presentation of our results in May 2003.. We label each event as either a buy or a sell order, and assign it $`\pm 1`$ accordingly. The autocorrelation of the resulting time series shows a power law autocorrelation function, with exponents that are typically about $`0.6`$, in the range $`0.36<\alpha <0.77`$. Positive autocorrelation coefficients are seen at statistically significant levels over lags of many thousand events, spanning many days. Thus the memory of the market is remarkably long.

This immediately raises a conundrum concerning market efficiency. All other things being equal, such strong long-memory behavior would imply strong predictability of the price return, easily exploitable for substantial profits. How can this be compatible with market efficiency? We show that this is at least partially solved by other properties of the market adjusting in order to compensate, making the market more efficient222For a discussion of what we mean by “efficient”, see Section VI.. In particular, the relative volume of buy and sell market orders, and the relative buy and sell liquidity, are skewed in opposition to the imbalance in order signs. For example, suppose the long-memory of previous order signs predicts that buy market orders are more likely in the near future. All things being equal, since buy market orders have a positive price response, the price should go up. But the market compensates for this: When buy orders become more likely, the ratio of buy market order size to the volume at the best ask tends to be smaller than normal. This implies that the probability of a given buy order penetrating the best price to trigger a positive price change is smaller than normal. Similarly, the opposite is true when sell orders are more likely. This is at least one of the effects that contributes to keeping the price returns roughly white. We demonstrate that market order volume and liquidity are also long-memory processes; we suggest this is because they are compensating for changes in the predictability of order signs in order to keep the market more efficient.

This brings up the interesting question of what actually causes these long-memory properties of markets. While not answering this question, we provide a clue about the answer by making use of the institutional codes associated with each order. Some institutions show long-memory quite clearly, while others do not.

The paper is organized as follows: In Section II we present a brief summary of previous work on long-memory processes in economics, and discuss our work in the context of this literature. In Section III we give a summary of the data set, and in Section IV we provide a more technical discussion of long-memory processes and the statistical techniques we use in this paper. Then in Section V we present the evidence that sequences of order signs are long-memory processes, and in particular we demonstrate that these series pass stringent tests so that we can be sure that they are long-memory with a high degree of confidence. In Section VI we show how other processes compensate for the long-memory of order signs, tending to keep price changes roughly efficient, and show that both order size and liquidity are also long-memory processes. In Section VII we break down the orders by institution and show that the behavior of some institutions shows long-memory quite clearly, while others do not show it at all. In Section VIII we discuss the implications of the long-memory properties of order flow for delayed market impact. We conclude in Section IX, discussing some of the broader issues and the remaining questions.

## II long-memory processes in economics time series

Long-memory processes have been observed in different natural and human phenomena ranging from the level of rivers to the temperature of the Earth (Beran, 1994). A good survey of the econometric approach to long-memory is given in Baillie (1996). The range of applications of long-memory processes in economics spans from macroeconomics to finance. In macroeconomics, for example, Diebold and Rudebusch (1989) find evidence of long-memory in the quarterly post World War II US real GNP data. Even if several criticisms have been raised to this work, subsequent analyses confirm the evidence of long-memory properties of GNP data. Baillie, Chung and Tieslau (1995) find that the monthly US Consumer Price Index (CPI) inflation time series has long-memory properties. A related study by Hassler and Wolters (1995) considers long-memory in inflation. Finally, Shea (1991) and Backus and Zin (1993) find evidence of long-memory in the context of the term structure of interest rates.

The study of possible long-memory properties of time series in finance is even more widespread. There has been a long-standing debate as to whether or not asset prices have long-memory properties. Several authors have claimed that the time series of stock returns for stock prices or indices display long-memory (Mandelbrot, 1971, Greene and Fielitz, 1977). More recently Lo (1991) re-examined these results and showed that the statistical R/S test used by Mandelbrot and Green and Fielitz is too weak and unable to distinguish between long and short memory (see also section IV and Campbell et al. 1997)). By introducing a modified R/S test, Lo concluded that daily stock returns do not display long-memory properties. This conclusion has been in turn criticized by Willinger, Taqqu and Teverovsky (1999). These authors showed with numerical simulations that the modified R/S test leads to the rejection of the null hypothesis of short-memory when applied to synthetic time series with a low degree of long-memory. Since financial data tipically display low degree of long-memory, Willinger, Taqqu and Teverovsky (1999) claim that the result of Lo is not conclusive. (We will not address this question).

It is more widely accepted (though still not entirely uncontroversial) that the volatility of prices is a long-memory process. Specifically Ding, Granger and Engle (1993) and Breidt, Crato and de Lima (1993) find evidence of long-memory stochastic volatility in stock returns, and Harvey (1993) finds evidence for this in exchange rates. These results led to the development of alternate models for volatility, such as FIGARCH. In particular, Baillie, Bollerslev and Mikkelsen (1996) apply the FIGARCH process to exchange rates, and Bollerslev and Mikkelsen (1996) apply FIEGARCH, a modification of FIGARCH, to stock prices. Another market property that seems to have long-memory properties is stock market trading volume (Lobato and Velasco, 2000). Models of long-memory processes include fractional Brownian noise (Mandelbrot and van Ness, 1968) and the ARFIMA process introduced by Granger and Joyeux (1980) and Hosking (1981).

In this paper we discover and study the long-memory properties of the signs of orders in a financial market. Despite the ubiquity of long-memory processes in economic time series described above, our result is, to our knowledge, the first demonstration of the presence of long-memory in a “microscopic” time series, i.e. a time series which is not the result of the aggregation of many individual events (such as the price return). Because of the rapid timescale, we have a large amount of data, and we are able to demonstrate the existence of long-memory properties with a very high degree of confidence. The order flow is a time series describing the action of trading institutions, i.e. it is an input to the price formation mechanism. From this point of view it is very interesting to see how the market responds to the existence of a highly predictable long-memory input, to form prices so as to maintain market efficiency. Because we are able to see the detailed state of the orderbook, we can study how bids, offers, and order volumes adjust to compensate, so that the long-memory properties of order signs and volumes are cancelled in the directional movements of prices. We investigate the order flow mainly in event time. This means that one time step is defined by the placement of an order of a given type (market order, limit order and cancellation). As a consequence the long-memory property we discuss here is not the effect of inhomogeneous trading during the day, but rather reflects the strategic way in which traders place their orders. Finally, we prove with statistical confidence that the long-memory property of the sign of the order flow can be present also at the level of individual trading institution.

## III Data

In order to a have a representative sample of stocks we select $`20`$ companies continuously traded at the London Stock Exchange (LSE) in the 4-year period 1999-2002. The stocks we analyzed are Astrazeneca (AZN), Bae Systems (BA.), Baa (BAA), BHP Billiton (BLT), Boots Group (BOOT), British Sky Broadcasting Group (BSY), Diageo (DGE), Gus (GUS), Hilton Group (HG.), Lloyds Tsb Group (LLOY), Prudential (PRU), Pearson (PSON), Rio Tinto (RIO), Rentokil Initial (RTO), Reuters Group (RTR), Sainsbury (SBRY), Shell Transport & Trading Co. (SHEL), Tesco (TSCO), Vodafone Group (VOD), and WPP Group (WPP). Table 1 gives a summary of the number of different events for the $`20`$ stocks.

The London Stock Exchange consists of two markets, the electronic (SETS) exchange, and the upstairs market. We study only the electronic exchange. The data set we analyze contains every action by every institution participating in this exchange. In 1999 the electronic exchange contains roughly $`57\%`$ percent of the order flow for a typical stock, and in $`2002`$ roughly $`62\%`$ percent. It is thus always a substantial fraction of the total order flow, and is believed to be the dominant mechanism for price formation. There are several types of orders allowed by the exchange, with names such as “fill or kill” and “execute or eliminate”. To place our analysis in more useful terms we label events in terms of their net effect on the limit order book. We label any component of an order that results in an immediate transaction an effective market order, and any component of an order that leaves a limit order sitting in the book an effective limit order. A single order may result in multiple effective orders. For example, consider a crossing limit order, i.e. a limit order whose limit price crosses the opposing best price quote. The part of the order that results in an immediate transaction is counted as an effective market order, while the remaining non-transacted part (if any) is counted as an effective limit order. We will call anyone who places effective market orders a liquidity taker, and anyone who places effective limit orders a liquidity provider.

We will also lump together any event that results in a queued limit order being removed without a transaction, and refer to such an event as a cancellation. Henceforth dropping the modifier “effective”, we can then classify events as one of three types: market order, limit order and cancellation. For the set of $`20`$ stocks described above there is a total of roughly $`9`$ million market orders, $`25`$ million limit orders and $`17`$ million cancellations. Throughout this paper, unless otherwise specified, we use the number of effective events as a measure of time, which we call event time. We typically do this in terms of the number of events of a given type, e.g. if we are studying market orders we measure event time in terms of the number of market orders, and if we are studying limit orders we measure event time in terms of the number of limit orders.

Trading begins each day with an opening auction. There is a period leading up to the opening auction in which orders are placed but no transactions take place. The market is then cleared and for the remainder of the day (except for occasional exceptional periods) there is a continuous auction. We remove all data associated with the opening auction, and analyze only orders placed during the continuous auction.

An analysis of the limit order placement shows that in our dataset approximately $`35\%`$ of the effective limit orders are placed behind the best price (i.e. inside the book), $`33\%`$ are placed at the best price, and $`32\%`$ are placed inside the spread. This is roughly true for all the stocks except for SHEL, for which the percentages are $`71\%`$, $`18\%`$ and $`11\%`$, respectively. Moreover for all the stocks the properties of buy and sell limit orders are approximately the same.

In our dataset cancellation occurs roughly $`32\%`$ of the time at the best price and $`68\%`$ of the time inside the book. This is quite consistent across stocks and between the cancellation of buy and sell limit orders. As for the case of the placement of limit orders, the only significant deviation is SHEL, for which the percentages are $`14\%`$ and $`86\%`$.

In the following price will indicate the mid price, i.e. $`p(t)=(a(t)+b(t))/2`$ where $`a(t)`$ and $`b(t)`$ are the best ask and best bid prices at time $`t`$, respectively.

## IV Review of methods for understanding long-memory processes

### IV.1 Definitions of long-memory

There are several way of characterizing long-memory processes. A widespread definition is in terms of the autocovariance function $`\gamma (k)`$. We define a process as long-memory if in the limit $`k\mathrm{}`$

$$\gamma (k)k^\alpha L(k)$$
(1)
where $`0<\alpha <1`$ and $`L(x)`$ is a slowly varying function333$`L(x)`$ is a slowly varying function (see Embrechts et al., 1997) if $`lim_x\mathrm{}L(tx)/L(x)=1`$. In the definition above, and for the purposes of this paper, we are considering only positively correlated long-memory processes. Negatively correlated long-memory processes also exist, but the long-memory processes we will consider in the rest of the paper are all positively correlated. at infinity. The degree of long-memory is given by the exponent $`\alpha `$; the smaller $`\alpha `$, the longer the memory.

Long-memory is also discussed in terms of the Hurst exponent $`H`$, which is simply related to $`\alpha `$. For a long-memory process $`H=1\alpha /2`$ or $`\alpha =22H`$. Short-memory processes have $`H=1/2`$, and the autocorrelation function decays faster than $`k^1`$. A positively correlated long-memory process is characterized by a Hurst exponent in the interval $`(0.5,1)`$. The use of the Hurst exponent is motivated by the relationship to diffusion properties of the integrated process. For normal diffusion, where by definition the increments do not display long-memory, the standard deviation asymptotically increases as $`t^{1/2}`$, whereas for diffusion processes with long-memory increments, the standard deviation asymptotically increases as $`t^HL(t)`$, with $`1/2<H<1`$, and $`L(t)`$ a slow-varying function.

Yet another equivalent definition of long-memory dependence can be given in terms of the behavior of the spectral density for low frequencies. A long-memory process has a spectral density which diverges for low frequencies as

$$g(f)f^{12H}L(f),$$
(2)
where $`f`$ is the frequency, and $`L(f)`$ is a slowly varying function in the limit $`f0`$. This follows immediately from the fact that the autocorrelation and the spectral density are Fourier transforms of each other.

### IV.2 Statistical tests for long-memory

The empirical determination of the long-memory property of a time series is a difficult problem. The basic reason for this is that the strong autocorrelation of long-memory processes makes statistical fluctuations very large. Thus tests for long-memory tend to require large quantities of data and can often give inconclusive results. Furthermore, different methods of statistical analysis often give contradictory results. In this section we review two such tests and discuss some of their properties. In particular we discuss the classical R/S test, which is known to be too weak, and Lo’s modified R/S test, which is known to be too strong.

The basic idea behind the classical R/S test (Hurst, 1951, Mandelbrot, 1972 and 1975) is to compare the minimum and maximum values of running sums of deviations from the sample mean, renormalized by the sample standard deviation. For long-memory processes the deviations are larger than for non-long memory processes. The classical R/S test has been proven to be too weak, i.e. it tends to indicate a time series has long-memory when it does not. In fact, Lo (1991) showed that even for a short-memory process, such as a simple AR(1) process, the classical R/S test does not reject the null hypothesis of short-memory. This fact motivated Lo (1991) to introduce a stronger test based on a modified R/S statistic.

We now describe Lo’s modified R/S test. Consider a sample time series $`X_1`$, $`X_2`$,…,$`X_n`$ with sample mean $`(1/n)_jX_j`$ as $`\overline{X}_n`$. Let $`\widehat{\sigma }_x^2`$ and $`\widehat{\gamma }_x`$ be the sample variance and autocovariance. The modified rescaled range statistic $`Q_n(q)`$ is defined by

$`Q_n(q)`$ (3)
$`{\displaystyle \frac{1}{\widehat{\sigma }_n(q)}}\left[\underset{1kn}{\mathrm{max}}{\displaystyle \underset{j=1}{\overset{k}{}}}(X_j\overline{X}_n)\underset{1kn}{\mathrm{min}}{\displaystyle \underset{j=1}{\overset{k}{}}}(X_j\overline{X}_n)\right],`$
where

$$\widehat{\sigma }_n^2(q)\widehat{\sigma }_x^2+2\underset{j=1}{\overset{q}{}}\omega _j(q)\widehat{\gamma }_j,\omega _j(q)1\frac{j}{q+1},$$
(4)
and $`q<n`$. It is worth noting that $`Q_n(q)`$ differs from the classical R/S statistics of Mandelbrot only in the denominator. In the classical R/S test $`\widehat{\sigma }_n(q)`$ is replaced by the sample standard deviation $`\widehat{\sigma }_x`$.

The optimal value of $`q`$ to be used in Eq. (3) to compute $`Q_n`$ must be chosen carefully. Lo suggested the value $`q=[k_n]`$, where

$$k_n\left(\frac{3n}{2}\right)^{\frac{1}{3}}\left(\frac{2\widehat{\rho }}{1\widehat{\rho }^2}\right)^{\frac{2}{3}},$$
(5)
$`[k_n]`$ indicates the greatest integer less than or equal to $`k_n`$ and $`\widehat{\rho }`$ is the sample first-order autocorrelation coefficient of the data. Lo was able to prove that if the process has finite fourth moment and it has a short-memory dependence (and satisfies other supplementary conditions) $`V_nQ_n/\sqrt{n}`$ tends asymptotically to a random variable distributed according to

$$F_V(v)=1+2\underset{k=1}{\overset{\mathrm{}}{}}(14k^2v^2)e^{2(kv)^2}.$$
(6)
This result makes it possible to find the boundaries of a given confidence interval under the null hypothesis that the time series is short-memory. When $`V_n`$ is outside the interval $`[0.809,1.862]`$, we can reject the null hypothesis of short range dependence with $`95\%`$ confidence.

Recently Teverovsky, Taqqu and Willinger (1999) have shown that Lo’s rescaled R/S test is too severe. They showed numerically that even for a synthetic long-memory time series with a moderate value of the Hurst exponent (like $`H=0.6`$) the Lo test cannot reject the null hypothesis of short range dependence. For our results here we are lucky that we are able to pass the rescaled R/S test for long-memory, but the stringency of this test should be borne in mind in evaluating our results.

### IV.3 Methods of measuring the Hurst exponent

The determination of the Hurst exponent of a long-memory process is not an easy task, especially when one cannot make any parametric assumptions about the investigated time series. Several heuristic methods have been introduced to estimate the Hurst exponent. Recently some authors suggested the use of a “portfolio” of estimators instead of relying on a single estimator which could be biased by the property of the time series under investigation (Taqqu, Teverovsky and Willinger, 1995). In this paper we will discuss four widespread Hurst exponent estimators which we describe below. These methods are the periodogram method, the R/S method, Detrended Fluctuation Analysis and the fit of the autocorrelation function. We find that the first three methods give reasonable agreement both in real and in surrogate time series. The fourth method appears to be more noisy and less reliable.

To use the periodogram method, one first calculates the periodogram $`I(f)`$, which is an estimate of the spectral density.

$$I(f)=\frac{1}{2\pi n}\left|\underset{j=1}{\overset{n}{}}X_je^{ijf}\right|^2$$
(7)
where, as before, $`n`$ is the size of the sample $`X_j`$. Then a regression of the logarithm of the periodogram against the logarithm of $`f`$ for small values of $`f`$ gives a slope coefficient that estimates $`12H`$ (see Eq. 2). We make our regression on the lowest $`10\%`$ of the data (Taqqu, Teverovsky and Willinger, 1995).

The second method is the R/S method (Mandelbrot, 1972 and 1975). A description of the method, which is strongly based on R/S statistics, can be found in Beran (1994). In summary, we divide a time series of length $`n`$ in $`K`$ blocks of size $`n/K`$ and we chose logarithmically spaced values of the lag $`k=1,2,4,8\mathrm{}`$. For a given value of $`k`$ we compute the classical R/S statistics (i.e. Eq.(3) with $`\widehat{\sigma }_x`$ instead of $`\widehat{\sigma }_n(q)`$ in the denominator) in each block, by using the first point as the starting point. When $`k<n/K`$, one obtains $`K`$ different values of the R/S statistics. Finally we plot the value of the R/S statistics versus $`k`$ in double logarithmic scale. The parameter $`H`$ is obtained by fitting a line to this plot.

The third method is the Detrended Fluctuation Analysis introduced in Peng et al. (1994). The time series is first integrated. The integrated time series is divided into boxes of equal length $`m`$. In each box, a least squares line is fit to the data (representing the trend in that box). The $`y`$ coordinate of the straight line segments is denoted by $`y_m(k)`$. Next, we detrend the integrated time series, $`y(k)`$, by subtracting the local trend, $`y_m(k)`$, in each box. The root-mean-square fluctuation of this integrated and detrended time series is calculated by

$$F(m)=\sqrt{\frac{1}{n}\underset{k=1}{\overset{n}{}}[y(k)y_m(k)]^2}.$$
(8)
This computation is repeated over all time scales (box sizes) to characterize the relationship between $`F(m)`$ and the box size $`m`$. Typically $`F(m)`$ will increase with box size $`m`$. The Hurst exponent is obtained by fitting $`F(m)`$ with a relation $`F(m)m^H`$. The proposers of this method claim that it is able to remove local trends due to bias in the enhanced occurrence of a class of events (Peng et al., 1994)

A fourth method is to simply compute the autocorrelation function and measure $`\alpha =22H`$ by regressing the autocorrelation function with a power law. This method, however, suffers from the problem that the sample errors in adjacent autocorrelation coefficients are strongly correlated, and so this method is less accurate than the other two methods discussed above. Thus, we only use this method as an indication. Based on tests on real and surrogate data, we find that the first three methods all give very similar results; we use either the R/S method or the periodogram method when we want to get accurate values of the exponent $`\alpha `$.

## V Demonstration of long-memory for order signs

### V.1 A quick look at the autocorrelation function

We consider the symbolic time series obtained in event time by replacing buy orders with $`+1`$ and sell orders with $`1`$, irrespective of the volume (number of shares) in the order. This can be done for market orders, limit orders, or cancellations. As we will see, all of these series show very similar behavior. We reduce these series to $`\pm 1`$ rather than analyzing the signed series of order sizes $`\omega _t`$ in order to avoid problems created by the large fluctuations in order size; analysis of the signed series of order sizes produces results that do not converge very well.

Figure 1 shows the sample autocorrelation functions of the order sign time series for Vodafone in the period 1999-2002 in double logarithmic scale.

Vodafone was chosen to illustrate the results in this paper because it is one of the most capitalized and most heavily traded stocks in the LSE during this period; we see very similar results for all the other stocks in our dataset. The autocorrelation function for market orders, limit orders and cancellations decays roughly linearly over more than $`4`$ decades, although with some break in the slope for limit orders and cancellations. This suggests that a power-law relation $`\rho (k)k^\alpha `$ is a reasonable asymptotic approximation for the empirical autocorrelation function. Of course, for larger lags there are fewer independent intervals, and the statistical fluctuations are much larger.

Estimating $`\alpha `$ from the sample autocorrelation using an ordinary least squares fit gives $`\alpha =0.39`$ for market orders. For limit orders there appears to be a break in the slope, with an exponent roughly $`0.4`$ for lags less than roughly $`500`$ and $`0.6`$ for larger lags. There is a similar break in the slope for cancellations, with a slope roughly $`0.4`$ for less than $`50`$ lags and $`0.7`$ for larger lags. As already mentioned, the sample autocorrelation is a poor method for estimating $`\alpha `$, and should only be considered an indication; later on we will use more reliable estimators. But the fact that $`\alpha `$ is much smaller than 1 in every case suggests that these might be long-memory processes. The memory is quite persistent, as is evident from the fact that the sample autocorrelations remain positive over a very long span of time. The average daily number of market orders for Vodafone in the investigated period is approximately $`1,300`$, whereas the slow decay of the autocorrelation function in Fig. 1 is seen for lags as large as $`10,000`$. This indicates that the long-memory property of the market order placement is not just an intra-day phenomenon, but rather spans multiple days, persisting on a timescale of more than a week. Similar statements are true for limit orders and cancellations. See also Section V.5, where we analyze this phenomenon in real time rather than event time.

### V.2 Statistical evidence for long-memory

In order to test the presence of long-memory properties in the time series of market order signs both longitudinally (i.e. analyzing a stock for different time periods) and cross-sectionally (i.e. analyzing different stocks) we proceed as follows: We consider the set of $`20`$ highly capitalized stocks described in Section 2 for the 4 year period 1999-2002. Since the number of orders is different for different stocks in different calendar years, we divide the data for each year and for each stock into subsets in such a way that each set contains roughly a fixed number of orders444The number of market orders ranges from $`26,438`$ for WPP in 1999 to $`415,392`$ for VOD in 2002, the number of limit orders ranges from $`51,798`$ in 1999 for BLT to $`2,552,410`$ for SHEL in 2002, and the number of cancellations ranges from $`29,395`$ for BLT in 1999 to $`2,259,526`$ for SHEL in 2002. We thus divided the market orders into $`324`$ subsets ranging in size from $`25,000`$ to $`49,999`$; we divided limit orders into $`468`$ subsets ranging in size from $`50,000`$ to $`99,999`$, and we divided cancellations into $`558`$ subsets in size ranging from $`29,000`$ to $`57,999`$.. To each set we apply the Lo test based on modified R/S statistics, obtaining a value for the statistics $`Q_n`$. Since our time series consists of $`+1`$ and $`1`$ we do not have problems with the existence of moments. Figure 2 shows the histogram of the $`324`$ values of $`Q_n`$ for the subsets of market orders.

For $`289`$ ($`89.2\%`$) subsets we can reject the null hypothesis of short-memory processes with $`95\%`$ confidence. Repeating this test for limit orders and cancellations gives even stronger results: For limit orders, based on $`468`$ subsets the short-memory hypothesis is rejected at the $`95\%`$ level in $`97\%`$ of the cases, and for cancellations using $`558`$ subsets it is rejected in $`96\%`$ of the cases. We can therefore conclude that these order sign time series are almost certainly long-memory processes. This result is even stronger when one considers the severity of this test, as pointed out in Teverovsky, Taqqu and Willinger (1999).

We have performed a similar analysis for the NYSE, using the Lee and Ready algorithm to sign the trades (Lee and Ready, 1991). Despite some technical problems associated with classifying the trades, it is quite clear that this is also a long-memory process555The Lee and Ready algorithm is not completely reliable in classifying trades. Fifteen percent of the trades remain unclassified. By random substitution of the sign of the unclassified trades, it is clear that for NYSE stocks the market order sign is a long-memory process with exponents similar to those observed in the LSE..

### V.3 Estimating the Hurst exponents

Now that we have established that these are long-memory processes we determine the Hurst exponent $`H`$ to see if there is consistency in the exponent in different years and for different stocks. Recall that for a long-memory process the Hurst exponent is related to the exponent $`\alpha `$ of the autocorrelation function through $`\alpha =22H`$.

The first estimator we used for the determination of the Hurst exponent is least squares fitting of the periodogram. The mean estimated value of the Hurst exponent is $`H=0.695\pm 0.039`$ for market orders, $`H=0.716\pm 0.054`$ for limit orders, and $`H=0.768\pm 0.059`$ for cancellations, where the error is the standard deviation. The histograms of the exponents obtained in this way are shown in Fig. 3.

We see that in every case the Hurst exponent is roughly peaked around the value $`H=0.7`$ which corresponds to $`\alpha =0.6`$.

Following the suggestion of Taqqu, Teverovsky and Willinger (1995) we also estimate the Hurst exponent for market orders through the classical R/S method. In this case the mean Hurst exponent is $`0.696\pm 0.032`$, which is consistent with the value obtained with the periodogram method. Figure 3(a) gives a comparison of the results of the two methods. In the inset we plot the Hurst exponent obtained from the periodogram against the Hurst exponent obtained from the R/S method, showing that the results are quite correlated on a case-by-case basis, with no discernable bias.

### V.4 Idiosyncratic variation of the Hurst exponents

The previous results bring up the interesting question of whether there are real variations in the Hurst exponents, or whether they have a universal value, and the variations that we see are just sample fluctuations. To compare the longtitudinal and cross-sectional variations we perform a classical ANOVA test. We assume that for each stock $`i`$ the value of the Hurst exponent in different time periods is normally distributed with mean $`m_i`$ and standard deviation $`\sigma `$. We test the null hypothesis that all the $`m_i`$ are equal. We indicate with $`H_{ij}`$ the estimated Hurst exponent of stock $`i`$ in sub-period $`j`$. There are $`r=20`$ stocks, each of them with a variable number $`n_i`$ of sub-periods. The total number of subsets is $`n=_in_i`$. As usual the sum of squares of deviations of $`H_{ij}`$ can be decomposed in the sum of squared deviations within groups (i.e. stocks) $`(nr)s_2^2=_{i=1}^r_{j=1}^{n_i}(H_{ij}\overline{H}_i)^2`$ and the sum of squared deviations between groups $`(r1)s_1^2=_{i=1}^r(\overline{H}_i\overline{H})^2`$, where $`\overline{H}`$ is the sample mean for the entire sample and $`\overline{H}_i`$ is the sample mean for stock $`i`$. Under the above null hypothesis, the sum of squared deviations within groups has a $`\chi ^2`$ distribution with $`nr`$ degrees of freedom. Likewise the sum of squared deviations between groups has a $`\chi ^2`$ distribution with $`r1`$ degrees of freedom. Therefore the logarithm of the ratio between $`s_1`$ and $`s_2`$ has Fisher’s Z-distribution with $`(r1,nr)`$ degrees of freedom. For all the three types of orders we reject the null hypothesis with $`99\%`$ confidence. Moreover in all three cases $`s_1>s_2`$, showing that the cross sectional variation of the Hurst exponent is significantly larger that the longitudinal variation, which suggests that the variations in the exponents between stocks are statistically significant. Nonetheless, it is interesting that these variations are relatively small.

### V.5 Order flow in real time

We have shown that the sequence of order signs is a long-memory process in event time. In this section we briefly consider the correlation properties in real time. This is complicated by the fact that trading is not homogeneous. There are both strong intra-day periodicities, e.g. volume tends to increase near the open and the close, and also strong temporal autocorrelations in the number of trades. Thus the number of trades in any given time interval of length $`T`$ can vary dramatically.

To understand the long-memory of orders in real time, we are seeking a quantity that gives information about imbalances in the signs of orders, but which is independent of the number of orders placed in a given time interval. We use two methods, which give similar results. Let $`n_b(t)`$ and $`n_s(t)`$ indicate the number of buy and sell market orders, respectively, in a time interval of length $`T`$ starting at time $`t`$. The first method follows a majority rule, which assigns the value $`+1`$ if $`n_b(t)>n_s(t)`$ and the value $`1`$ if $`n_b(t)<n_s(t)`$. When $`n_b(t)=n_s(t)`$ or there are no market orders in the interval we assign the value $`0`$. The main defect of this method is that it does not distinguish intervals with small or large imbalance of one type of orders. The second method is to use a continuous variable defined as $`(n_b(t)n_s(t))/(n_b(t)+n_s(t))`$ when $`n_b(t)+n_s(t)0`$ and zero elsewhere. This is bounded between $`1`$ and $`1`$. In Figure 4 we show the autocorrelation function of $`(n_b(t)n_s(t))/(n_b(t)+n_s(t))`$ for a time interval $`T=5`$ minutes for Vodafone. We note that a power law decay of the autocorrelation function fits the empirical data quite well, with an exponent $`\alpha =0.3`$, which is close to the corresponding value in event time.

This study makes it quite clear that the long-memory properties of order signs persist across trading days. There are $`102`$ intervals of length $`5`$ minutes in a trading day, which means that the last lag in Figure 4 corresponds to approximately $`10`$ trading days. Moreover the autocorrelation does not show any significant peak or break in slope near lag = $`102`$, indicating that the long-memory properties of the market persist more or less unchanged across daily boundaries.

### V.6 Autocorrelation of transaction volume

We now show that the volume of the transactions is a long memory process in event time; later on in Section VI.2 we will argue that this is connected to the long-memory properties of order signs via market efficiency. The long-memory properties of aggregated volume have been known for a long time (Lobato and Velasco, 2000, Gopikrishnan et al., 2000). We use modified R/S statistics in order to test the null hypothesis that the transaction volume is a short memory process in event time. The value of a stock changes in time because of the change in price. Therefore one could expect that the number of traded shares is non-stationary due to the non stationarity of the price. For this reason we decide to investigate the value of the transaction, defined as the product of the number of traded shares and the transaction price. The value is invariant under stock splits. In Figure 5 we show the autocorrelation function of the volume of Vodafone measured in terms of value in the period 1999-2002.

The inset shows a histogram of the transaction volume, which is well fit by a Gamma distribution. Once we adjust for scale, this seems to be roughly the same for all the stocks in the sample (see also Farmer and Lillo, 2004). Moreover this result is in contrast with what has been observed for the NYSE by Gopikrishnan et al. (2000).

We applied the modified R/S test to the $`324`$ subsets used to test the long-memory properties of market order size. One of the conditions for the applicability of the modified R/S test is that the unconditional kurtosis of the time series is finite; from the inset of Figure 5 it seems that the volume distribution does not have a power-law tail, so the modified R/S test is applicable. However there could be biases in the test due to large fluctuations in volume. In $`303`$ of the $`324`$ subsets, or $`94\%`$ of the time, we reject the null hypothesis of short-memory for the transaction volume with $`95\%`$ confidence. The Hurst exponent estimated with the periodogram method varies across subsets and the mean value is $`H=0.732\pm 0.075`$, within the sampling error of the exponent found for order signs.

## VI Market inefficiency?

At first sight the long-memory property of the market order sign time series is puzzling when considered from the perspective of market efficiency. Long-memory implies strong predictability using a simple linear model. When this is combined with the fact that orders have price impact, it naively suggests that price changes should follow a long-memory process as well. That is, buy market orders tend to drive the price up, and sell market orders tend to drive it down. Thus, all other things being equal, a run of buy orders should imply future upward price movement, and a run of sell orders should imply future downward price movement. The predictability of order signs is sufficiently strong that one would expect that profits could be made by taking advantage of it.

There are many ways to define market efficiency, and we should be clear how we are using this term. Here we mean specifically linear efficiency, i.e. that the series of price returns contains negligible temporal autocorrelations. This allows for the possibility that there might be other more complicated nonlinear patterns, and assumes a trivial reference equilibrium that supports an IID random price process. This is a strong notion of efficiency in the sense of Fama (1970), in that the information set is a sequence of recent buy or sell order signs, which for the LSE is publicly available during this period in real time. We will make a subjective judgement concerning what we mean by “negligible”, without making detailed estimates of transaction costs: If the directional movements in price returns have long-memory, then the market is unlikely to be efficient, whereas if they have short-memory, it becomes much more difficult to tell.

In this section we explore the consequences of long-memory in order signs, and show that its impact on the predictability for prices is offset by other factors. In particular, the relative size of buy and sell market orders and the relative size of the best quotes at the best bid and ask move in a way that is anti-correlated with the long memory of order signs, and compensates to make the market more linearly efficient. While order signs, market order volume, and volume at the best prices are all long-memory processes, directional price changes do not appear to have this property.

### VI.1 Inefficiency of prices in absence of liquidity fluctuations

In this subsection we show that if liquidity were fixed, the long-memory in the signs of orders would drive a strong inefficiency in prices. We first construct a series of surrogate prices assuming that the reponse of prices to new market orders depends on order size but is otherwise fixed. The relation between the volume of a market order and the consequent price shift is described by the average price impact function (also called the average market impact function). Recent studies of the impact of a single transaction (Hasbrouck, 1991, Hausman and Lo, 1992, Farmer, 1996, Potters and Bouchaud, 2002, Lillo, Farmer and Mantegna, 2003) have shown that the average market impact is a concave function of either order or transaction volume, matching other studies based on time-aggregated volume (Torre, 1997, Kempf and Korn, 1999, Plerou et al., 2001, Evans and Lyons, 2002). It appears that the average impact varies across markets and stocks. For example, for a set of $`1000`$ stocks traded at NYSE (which works with a specialist) the impact is roughly

$$E(r|V)=\frac{sign(V)|V|^\beta }{\lambda }$$
(9)
where $`r`$ is the logarithmic price return, $`V`$ is the volume of a transaction, and $`\lambda `$ is a liquidity parameter. The exponent $`\beta `$ depends on $`V`$ and is approximately $`0.5`$ for small volumes and $`0.2`$ for large volumes (Lillo, Farmer and Mantegna, 2003). The liquidity parameter $`\lambda `$ varies for each stock, and in general may also vary in time. Potters and Bouchaud (2002) analyzed a much smaller set of stocks traded at the Paris Bourse and NASDAQ and suggested a logarithmic price impact function. For the LSE, Figure 6 shows the price impact of buy market orders for 5 highly capitalized stocks, i.e. AZN, DGE, LLOY, SHEL, and VOD.

The price impact is well fit by the relation $`E(r|V)V^\beta `$, where $`V`$ should now be interpreted as the market order size $`V=|\omega |`$ and $`\beta 0.3`$.

If one assumes that the price impact is a deterministic function of order size, since market order placement constitutes a long-memory process, the generated price return time series will be long-memory too. We test this conclusion by constructing a synthetic price time series using real market order flow with a deterministic impact function of the form of Equation (9), but with $`V`$ now representing market order size. We use $`\beta =0.3`$ as measured for Vodafone in Figure 6, and arbitrarily set $`\lambda =1`$. For each real market order of volume $`V_i`$ and sign $`ϵ_i=\pm 1`$ we construct the surrogate price shift $`\mathrm{\Delta }p_i=ϵ_iV_i^{0.3}`$. In Figure 7 we plot the autocorrelation function of the surrogate time series of price shift obtained with the deterministic price impact (upper curve). The inset of Figure 7 shows the same data with a double logarithmic scale. As expected the autocorrelation decays as a power-law, implying that synthetic price returns are described by a long-memory process, in contradiction with the assumption of linear efficiency. The application of the Lo test rejects the null hypothesis of short memory and the periodogram method gives a value of the Hurst exponent $`H=0.66`$.

There are two possible explanations for how the real price series can be efficient when the surrogate price series defined above are inefficient. We have only used market orders above, so the first possible reason is that the price shift generated by limit orders and cancellations act to make the market efficient. The second possibility is that the assumption of deterministic price impact is wrong and efficiency comes about due to fluctuations in the impact. The first reason has been recently suggested by Bouchaud et al. (2004). Their argument is that the price shift due to a market order is anticorrelated with the price shift generated by limit orders and cancellations placed between market orders. We verified empirically that such an anticorrelation does exist. However, as we will show below, the market is approximately efficient, in the sense that directional price changes do not display long-memory, even when we include only price shifts driven by market orders. Instead, we show that efficiency is due to fluctuations in liquidity.

To show that efficiency does not depend on limit orders and cancellations, we plot in Figure 7 the autocorrelation function of the real price shift time series $`\mathrm{\Delta }p_i`$ due to each market order $`i`$. We note that the sample first-order autocorrelation coefficient $`\widehat{\rho }`$ is negative. For values of the lag between $`2`$ and $`10`$ the autocorrelation is positive and for lags larger than $`10`$ it fluctuates around zero. This is in strong contrast to the autocorrelation function of the order flow, which is consistently positive for at least the first $`10,000`$ lags. This makes it clear that the long-memory of the order series has been strongly suppressed, and suggests that it no longer exists.

In an attempt to test this more carefully we apply the Lo test. However, this presents two main problems. First, the distribution of price shifts due to market orders seems to have a power-law tail with a relatively small tail index (Farmer and Lillo, 2004). Therefore we are not sure that the fourth moment of the price shift is finite as needed for the statistics used in the Lo test to be valid. Second, $`\widehat{\rho }`$ is negative and we cannot use Eq. (5) to find the optimal value of $`q`$. Despite these difficulties we applied the Lo’s test to the time series of price shifts due to market orders but we find inconclusive results. Since we do not have an optimal value for $`q`$, we calculated $`Q_n(q)`$ for different values of $`q`$ and we found that for some values of $`q`$, $`Q_n(q)`$ is outside the $`95\%`$ confidence interval for the null hypothesis of short memory, whereas for other values is inside this confidence interval. A clearer result is obtained by computing the Hurst exponent of the time series with the periodogram method. The value $`H=0.53`$ is very close to the value $`0.5`$ expected for a short-memory process. In conclusion the surrogate time series obtained with the deterministic price impact is clearly long-memory, whereas the time series of market order-driven price shifts is significantly less correlated with a strong suggestion that it is probably a short memory process. Similar results are seen when we compute the Hurst exponent for returns generated by limit orders or cancellations alone, and for other stocks, see Table 2.

This shows that market order, limit order, or cancellation-driven fluctuations are approximately efficient when considered by themselves666There is a slight tendency for the Hurst exponent of all events to be a little smaller than the Hurst exponent of each type of event taken separately. This suggests a cooperative effect in which one type of event mean reverts against the other, as suggested by Bouchaud et al. (2004)..

### VI.2 The key role of fluctuations in relative liquidity

In this section we will study fluctuations in liquidity and fluctuations in market order size, and show that the ratio of the two responds to changes in order sign predictability so as to make the market more efficient.

It is clear that the liquidity, e.g. defined as the $`\lambda `$ parameter in equation 9, makes large variations in time. For most purposes it is not a good approximation to treat it as a constant. A study of the LSE makes it quite clear that fluctuations in liquidity are large in comparison to the volume dependence of $`E[\lambda _i|V]`$, and that in Equation 9 one should regard $`\lambda `$ as a random variable whose fluctuations are roughly as large in relative terms as those of $`\mathrm{\Delta }p`$ (Farmer et al., 2004).

We first show that uncorrelated fluctuations in liquidity are not sufficient to ensure linear efficiency, and then return to study the correlations. Consider Equation 9 and let us assume that the inverse of the liquidity $`\mathrm{}_i1/\lambda _i`$ is a random variable uncorrelated with market order sign and size. In the previous section we have seen that $`a_iϵ_iV_i^\beta `$ is a long-memory random process. Therefore if $`E(a_i)=0`$ then $`E(\mathrm{\Delta }p_i)=0`$, and the auto-covariance of price return is

$`\gamma _{\mathrm{\Delta }p}(\tau )=E(\mathrm{\Delta }p_{i+\tau }\mathrm{\Delta }p_i)=`$
$`E(a_{i+\tau }a_i\mathrm{}_{i+\tau }\mathrm{}_i)=E(a_{i+\tau }a_i)E(\mathrm{}_{i+\tau }\mathrm{}_i)=`$
$`\gamma _a(\tau )(\gamma _{\mathrm{}}(\tau )+E(\mathrm{})^2)`$ (10)
Now $`\mathrm{}`$ is by definition a positive quantity and $`E(\mathrm{})>0`$. Therefore the term in brackets in the last line of Eq. (10) cannot be zero and $`\gamma _{\mathrm{\Delta }p}(\tau )0`$, i.e. when the liquidity is uncorrelated with the order flow, the market cannot be efficient.

We will define the term relative liquidity to mean variations in what is offered by liquidity providers, relative to what is being asked for by liquidity takers. For our purposes here, we will define this more precisely as the size of market orders relative to the volume at the opposite best price (e.g., the size of a buy market order relative to the volume at the best ask).

Our working hypothesis is that market efficiency is strongly influenced by relative liquidity. This hypothesis is influenced by other work (Farmer et. al, 2004), in which we demonstrate the following microscopic picture of market impact. There we show that market impact varies, both because the depth of stored limit orders varies, and because the size of market orders varies with it. Market order placement and the volume (depth) at the best price are highly correlated. It is very rare for a market order to be larger than the depth at the opposite best price (Farmer et al., 2004), presumably because liquidity takers are reluctant to execute at prices worse than the best price. When market orders do trigger price changes, they almost always do so by exactly removing the volume at the (opposite) best price. The size of the resulting change in the best price is just the size of the gap between the best price and the next price occupied by a limit order. Thus, the market impact depends on two quantities: Whether or not a market order is big enough to cause a price change at all, and when it does cause a price change, the size of the gap to the next occupied price.

How can a predictable run of orders of a given sign be consistent with market efficiency? For example, consider a period in which there has been a run of buy market orders. The long-memory implies that the next order is more likely to be a buy order. Our hypothesis is that this is compensated by the fact that the next buy market order is less likely to penetrate the best price. This can be either because the volume of limit orders at the best ask is larger, or because the volume of the next buy market order is smaller, or both of the above.

In order to test this hypothesis we explicitly construct a model to predict the sign $`ϵ_t`$ of the next order. We do this by making an autoregressive model of the form

$$\widehat{ϵ}_t=\underset{i=1}{\overset{N}{}}a_iϵ_{ti}.$$
(11)
To fit the coefficients of the model we use ordinary least squares. The values of the largest lag $`N`$ vary depending on the stock, but are typically the order of fifty. $`\widehat{ϵ}_t0`$ corresponds to low predictability, whereas large values of $`|\widehat{ϵ}_t|`$ correspond to high predictability. We have tested this and demonstrated that it works very well. When $`\widehat{ϵ}_t`$ is $`0.5`$, for example, there is a $`75\%`$ probability that the next order is a buy order777A more proper approach would be to use a model specifically tailored for forecasting probabilities, but the quick and dirty approach above is sufficient for our purposes here..

We now test our hypothesis by computing the probability of a penetration of the best price as a function of the strength of the sign predictor. (We say that a market order penetrates when it is as large or larger than the opposite best, and so causes a mid-price change). We analyze this for buying and selling, and for predictions that are right or wrong. For example, when the sign predictor is positive, predicting that the next order will be a buy order, we compute the fraction of buy market orders that penetrate the best ask. We compare this to the case when the sign prediction is also positive, but the prediction is wrong, in which case we compute the fraction of sell orders penetrating the best bid. We repeat all this similarly with signs reversed when the sign predictor is negative. We bin based on the value of the sign predictor and compute the probabilities for each bin. The results are shown in Figure 8.

We see a large variation in the results. When the sign is unpredictable, the predictor has a value near zero. In this case the penetration probability is roughly $`58\%`$ when the sign predictor is correct, and about $`56\%`$ when it is incorrect (for both buys and sells). But when the predictability is high, i.e. when $`|\widehat{ϵ}_t|0.5`$, the situation is quite different. When the prediction is correct, the probability of penetration is much lower, roughly $`48\%`$, and when it is incorrect it is much higher, about $`61\%`$. This shows that when predictability is high the market acts to decrease the probability that an order of the predicted sign will penetrate. This can be achieved either through higher volume at the opposing best, or by smaller market order size, or both. In either case, this is what we mean when we say that the relative liquidity acts to oppose the trend in order flow. We see similar results for other stocks888The level of the penetration probabilities shifts from stock to stock, but the basic pattern remains the same..

To test this hypothesis in a different way we plot the expected value of the logarithm of the ratio of the market order size to the volume at the opposite best, as a function of the strength of the order sign predictor of equation 11. This includes all events, whether the sign prediction is right or wrong. The results for the stock Astrazeneca are shown in Figure 9.

Corresponding to our previous result, we see that the ratio of market order size to volume at best is larger when the sign is unpredictable, and smaller when it is predictable. This reinforces our conclusion that fluctuations in relative liquidity oppose trends in order signs. Similar results are observed for other stocks.

An obvious question is whether these results are primarily driven by changes in market order size, or by changes in the volume at the best. We have done a variety of tests for this, with inconclusive results. We have seen some indications that variation in volume at the best conditioned to the sign predictor is the dominant effect, but the effect is only $`1.5\%`$, in contrast to the effects above, which are on the order of $`20\%`$. The problem is that there are large covarying level shifts in volume and liquidity through time. Their ratio, in contrast, is not affected by such level shifts, and is a much more sensitive indicator.

The other question one naturally asks concerns the magnitude of price responses. Given that a market order penetrates the best price, are there significant variations in the size of the resulting price response? In other words, are there asymmetries in the gaps in the two sides of the limit order book that are conditioned on the sign predictor? Preliminary studies suggest that this is not as important as the change in the probability of penetration.

These results suggest that the volume of market orders and the volume at the best price are comoving with trends in order signs in order to make the market more efficient. This motivated us to test whether or not these are long-memory processes. Typical autocorrelation functions are shown in Figure 10.

These resemble the autocorrelation functions for order signs, making it quite clear that they are also long-memory processes. In presenting the results in this order, we do not mean to necessarily suggest that the long-memory of order signs is primary, and that the long-memory of volume and liquidity are consequences of it, but rather to say that these phenomena are intimately related: From the analyses presented here, one could equally well say that the long-memory of order signs adjusts in order to offset that of volume and liquidity. The key point is that to enhance linear efficiency, despite the long-memory of all three of these processes, they must be in a certain sense out of phase, so that their effect on prices roughly cancels.

We have not demonstrated that these effects are sufficient to ensure linear market efficiency. Rather we have demonstrated that they are quite strong, and they act in the right direction to make the market more efficient. In the conclusions we discuss some possible motives for this behavior.

## VII Individual institutions

In this section we consider the behavior of individual institutions in order to gain some understanding of what drives the long-memory processes described above. The LSE database allows us to track the actions of individual institutions through a numerical code which identifies the institution. For privacy reasons the code is different for different stocks and it is reshuffled each calendar month. Therefore our analysis will be limited to a single trading month.

We consider as a case study the market order placement of Vodafone in July 2002. Our choice is motivated by the fact that Vodafone is one of the most heavily traded stocks. In this month there were $`45,774`$ market orders distributed across $`155`$ trading institutions. We have found that the $`12`$ most active institutions are responsible for more than $`70\%`$ of market orders. Thus, the participation in trading is extremely inhomogeneous among the institutions, with a few institutions placing many orders and many institutions placing only a few orders.

Table 3 shows the identification code, the number of market orders, and the fraction of market orders that are buy orders for each of the twelve largest institutions. In Figure 11 we show the autocorrelation function of the time series of market order signs for four active institutions.

In panel (a) we show two institutions whose market order flow is a long-memory process. One of the two institutions (code 3589) is the most active institution, which placed buy market orders $`24\%`$ of the time, and the other one (code 1886) placed buy market orders $`51\%`$ of the time. We see that in both cases the autocorrelation function is well-described by a power law with an exponent $`\alpha 0.5`$, which corresponds to $`H=0.75`$. Panel (b) shows two active institutions (code 3007 and code 823) whose market order sign time series is a short-memory process. To test the hypothesis that the individual market order placement is a long-memory process more rigorously, we apply the modified R/S test to the time series of the market order signs of the twelve most active institutions. Table 3 reports the value of $`Q_n`$. A boldface font indicates the cases when $`Q_n`$ is outside the $`95\%`$ confidence interval of the null hypothesis of short-memory. We see that for $`7`$ of the $`12`$ active institutions we reject the null hypothesis of short-memory process.

We repeated these results for the stock AZN in August 2001 and got similar results. Out of the top ten institutions, according to the Lo test, five clearly displayed long-memory, and five did not.

This result shows that even at the institution level the placement of orders has long-memory properties. This is not true for all institutions, but rather there is an heterogeneity in their behavior. A correlated sign in the order placement could be an indication of splitting a large order in smaller size orders in order to maximize profit without paying too much in terms of price impact. On the other hand an uncorrelated (or at least short range correlated) sign in the order placement could indicate different strategies such as, for example, market making. In section IX.2 we suggest and discuss possible causes of the long-memory of order flow.

## VIII Implications for market impact

In this section we discuss a practical consequence of long-memory of order flow. We have seen in Section V.A that the market impact is a concave function of volume. We may therefore ask about the price shift in the future (in transaction time) given that an order of a given volume and sign has arrived in the present. To be more specific, let us consider a buy market order of volume $`V_1`$ occurring now. The generated price shift $`\mathrm{\Delta }p_1`$ is the difference between the midprice just after the order and just before the order. Between this market order and the next market order the midprice can change because of new limit orders and cancellations, generating a price change $`\mathrm{\Delta }p_i`$. When the next market order arrives a new midprice shift $`\mathrm{\Delta }p_2`$ occurs. The total price shift between the instant just before the first order is placed and the instant just after the second market order is placed is therefore

$$\mathrm{\Delta }p_{12}=\mathrm{\Delta }p_1+\mathrm{\Delta }p_i+\mathrm{\Delta }p_2$$
(12)
If the order flow were random we would expect that $`E(\mathrm{\Delta }p_{12}|V_1)=E(\mathrm{\Delta }p_1|V_1)`$ since the volume and the sign of the next orders is uncorrelated with the corresponding quantities of the first order. In Figure 12 we present a decomposition of the impact of two successive market orders in the terms described above. In panel (a) of Figure 12 we show the four quantities $`E(\mathrm{\Delta }p_1|V_1)`$ (the same quantity shown in Fig. 6), $`E(\mathrm{\Delta }p_i|V_1)`$, $`E(\mathrm{\Delta }p_2|V_1)`$ and $`E(\mathrm{\Delta }p_{12}|V_1)`$. We see that $`E(\mathrm{\Delta }p_i|V_1)`$ is almost zero, meaning that the price shift due to limit orders and cancellations after a market order is relatively unimportant. This result suggests that the role of price reversion due to limit orders and cancellations between two market orders is marginal in making the market efficient. On the other hand $`E(\mathrm{\Delta }p_2|V_1)`$ is clearly positive and increasing with $`V_1`$. This is due to the strong temporal correlation in market order sign and size. In fact if the first market order is a buy market order it is probable that the next market order is also a buy and the volume of the second market order is correlated with the first one. Therefore it is more probable that the price will move up due to the arrival of the second order. Figure 12 shows $`E(\mathrm{\Delta }p_{12}|V_1)`$, which is simply the sum of the three terms, as shown in Eq. 12. The distance between $`E(\mathrm{\Delta }p_{12}|V_1)`$ and $`E(\mathrm{\Delta }p_1|V_1)`$ is a measure of the effect of the correlation of order sign and size in the delayed price impact.

To extend this analysis to more orders, we study the delayed market impact $`E(\mathrm{\Delta }p_{1m}|V_1)`$ where $`m`$ is the number of future market orders. Panel (b) of Figure 12 shows this quantity as a function of $`V_1`$ for $`m=2,3,4,5,6`$ and $`m=10`$. We see that for a fixed value of $`V_1`$, $`E(\mathrm{\Delta }p_{1m}|V_1)`$ is an increasing function of $`m`$. This is clearly due to the long correlation of market order sign and size. Eventually, for large values of $`m`$, $`E(\mathrm{\Delta }p_{1m}|V_1)`$ becomes independent of $`m`$.

## IX Conclusions

### IX.1 Comparison to work of Bouchaud et al. (2004)

Now that we have presented all our results, we can compare to the work of Bouchaud et al (2004). They independently discovered the same long-memory effect we have reported here for market order flow in the Paris Stock Exchange. We have taken the analysis in a somewhat different direction than they have, and offered a different interpretation. First, to convince potential sceptics, we have gone to extensive length to demonstrate at a very high level of statistical significance that order signs are indeed a long-memory process. This is separately true for market orders, limit orders, and cancellations. We have explicitly constructed a time series forecasting model that shows the high degree of predictability that goes along with this behavior – the conditional probability of the sign of the next order is frequently as high as $`75\%`$. As required to ensure market efficiency, this predictability is not present in price movements. We have hypothesized that this is at least in part due to anti-correlated changes in relative liquidity, as defined by the ratio of market order size to volume at the best opposite price. This is large effect, involving variations in relative liquidity of the order of $`20\%`$. This does not prove that this is sufficient to ensure efficiency, but it does suggest that time varying relative liquidity plays a major role.

Bouchaud et al. (2004) have offered a different explanation. The key difference concerns the way in which liquidity is treated. They assume a constant mean-reverting propagator for market impact. Whereas we have studied the way in which liquidity varies in opposition to order flow, their assumption of a constant propagator amounts to assuming that the liquidity is not varying in any correlated manner with trends in order flow. Instead, they assume that the market impact is time dependent in a way tha causes much of it to disappear. They have pointed out that the market impact does not grow as fast as one would naively expect, and have presented evidence that it often reverts on a long timescale, corresonding to a few hours to a day. They propose that the key factor ensuring efficiency is anti-correlated limit order placement, i.e. that later placement of limit orders undoes the potentially long-memory permanent price changes that would otherwise be caused by market orders.

We have presented two pieces of evidence that seem to oppose this point of view: First, we show that market order, limit order, and cancellation driven price changes are not long-memory, even when they are considered individually. This seems to imply that correlated behavior of limit orders is not the effect that cancels the long-memory of order flow. Second, in our study of market impact in Section VIII, we showed that the expected price shift in the intervening time between two market orders due to limit orders and cancellations is relatively small. It may still be possible, however, that these small effects accumulate to become important, as suggested by Bouchaud et al. (2004).

At this stage it is quite possible that both of these effects coexist: While we have showed that time-varying liquidity is a significant effect, we have not presented any evidence that it is the only effect. It may well be that there are diverse forces working to ensure market efficiency.

### IX.2 Possible causes of long-memory order flow

We have shown that the sign of order flow, order size, and liquidity, are all long-memory processes. What might cause this? In this section we make a few speculations about the possible origin of long-memory in these fundamental inputs to price formation.

One possible explanation for long-memory in order flow is that it simply reflects news arrival. Good (or bad) news may be clustered in time, driving the sign of order flow. Such news could either be external to the market, or it could be generated by factors internal to the market. If external it could be a property of the natural world, a reflection of the environment that humans necessarily interact with. We know that the intensity of floods, hurricanes, earthquakes, and natural disasters have a power law distribution, and perhaps these are just symptoms of a ubiquitous property of the natural world that is reflected in what we consider “news”. Alternatively, this could be an internal property, due to human social dynamics. Such “news” might be internally generated, e.g. due to herding behavior (Cont and Bouchaud, 2000), or it might be caused by inattention: Time lags in the response of investors to news arrival can cause autocorrelations in order flow. However, it is not clear why this should have a power law distribution.

A different explanation is in terms of the execution of large orders, which leads to order splitting. It is well-known that institutions with large orders frequently split them into small pieces (Chan and Lakonishok, 1993 and 1995), spreading out the execution of the orders over periods that can be many months long. If such orders have a power law distribution, and the time needed to fully execute an order is proportional to the size of the order, then this might give rise to power law autocorrelations in time. The idea that order size would have a power law distribution is not implausible given that many related quantities, such as firm size (Axtell. 2001), wealth (Pareto, 1896) and mutual fund size (Gabaix et al, 2003), have power law distributions.

We have tended to discuss the autocorrelation of order signs as though this were a primary property, and the behavior of volume and liquidity are consequences, which must exist in order to maintain market efficiency. An alternative is that this reasoning is reversed, and that the autocorrelation of volume and liquidity are primary properties, and that of order signs is a consequence. However, it still remains to be determined why any of these should have such strong temporal autocorrelations. At this stage we simply don’t know what causes this phenomenon, but it is clearly a remarkable aspect of human economic activity that deserves more attention.

Even if we take the existence of long-memory as a given, we have not explained the strategic behavior that ensures efficiency. While the correlated long-memory behavior of order signs, volume, and liquidity are necessary to maintain efficiency, why are market participants motivated to make this happen? What profit-seeking or risk-avoiding motives drive market participants to place orders in such a way that order size and liquidity are anti-correlated with runs in order signs?

To illustrate the problem, we will propose and then critique a possible explanation based on market makers’ incentive to control prices. During a run of buy market orders, which would normally tend to drive the price up, market makers as a group are by definition sellers, and their positions become more negative. If this is accompanied by a price rise they will tend to lose money. To prevent this they might intentionally manipulate liquidity to prevent or reduce upward price movements, by supplying more liquidity at the ask than the bid. In a specialist system in which a single agent has a monopoly on market making this might be a reasonable explanation. However, in a competitive environment such as the LSE, where there are many market makers, this is more difficult to explain: The individual who takes the lead in controlling the price will experience the largest adverse change in position. We know that on average buy orders do tend to drive the price up (there is on average market impact), so this behavior should systematically result in losses999The key question is whether the profits from taking the spread are sufficient to offset the adverse price movements associated with long memory.. This suggests that either the market makers as a group collude to control prices, sharing the burden between them, or that this is not the correct explanation for this behavior.

An alternative hypothesis that we consider more plausible is that liquidity providers take advantage of trends in order flow to acquire a desired position. So, for example, consider a liquidity provider that wants to sell, either to unload an existing inventory or to take a new tactical position. When a buying trend develops, she takes advantage of it by placing larger sell limit orders at or near the ask. From this point of view, one might reverse the usual terminology, and think of market orders as providing liquidity for limit orders by making them more likely to be executed. Under this hypothesis, liquidity providers take advantage of an imbalance in market order flow, thereby damping the response of the price.

One clue about the long-memory behavior of order flow is that we see it for some institutions and not for others. This might be caused by differences in order splitting strategies. However, at this stage this is just speculation.

###### Acknowledgements.

We would like to thank the James S. McDonnell Foundation for their Studying Complex Systems Research Award, Credit Suisse First Boston, McKinsey Corporation, Bob Maxfield, and Bill Miller for supporting this research. We would also like to thank J-P. Bouchaud, Dick Foster,and Laszlo Gillemot for valuable conversations and Marcus Daniels for technical support