Law of Large Numbers

The law of large numbers is a theorem of probability which states that the average of a large number of independent observations of the same random quantity will be close to its expected value, and will tend to get closer as the number of observations grows. It explains why a casino can be confident of its takings over a year while having no idea whether it will win the next hand.

The result was first proved by Jakob Bernoulli, whose work was published in 1713 after his death. It underpins insurance, polling and all of sampling statistics. In finance it is the reason that strategies with a small edge per trade rely on a large number of trades, and it sets limits on how much can be learned from a short track record.

Let $X_1, X_2, \ldots, X_n$ be independent draws of a random variable with mean $\mu$. The sample mean is:

The weak form of the law says that for any small margin $\varepsilon \gt 0$:

• $\bar{X}_n$ is the average of the first $n$ observations.
• $\mu$ is the true mean, also called the population mean.
• $P(\cdot)$ is the probability of the event in brackets.
• $\varepsilon$ is any chosen tolerance, however small.

In words, the chance that the sample average misses the true mean by more than any fixed amount shrinks to zero. The strong form says more: with probability one, the sequence of averages actually converges to $\mu$.

The law says the average converges but not how quickly. When the observations have a finite standard deviation $\sigma$, the typical error of the sample mean is the standard error:

The error falls with the square root of the sample size, so four times as many observations are needed to halve it. Consider a strategy that wins or loses one unit per trade and wins 55 percent of the time. Its expected profit is $0.55 - 0.45 = 0.10$ units per trade, and the standard deviation of a single trade is about 0.995 units. After 100 trades the standard error of the average is $0.995 / 10 = 0.0995$, about the same size as the edge itself, so a losing run of 100 trades would not be unusual. After 10,000 trades it is 0.00995, a tenth of the edge, and the average result is very likely to be positive. The shape of the remaining uncertainty is described by the central limit theorem.

• Market making and high-frequency trading. A market making firm earns a small expected amount on each of a very large number of trades. Individual trades are close to a coin toss, but the daily total is far more predictable.
• Diversification. Spreading money across many independent risks makes the portfolio's outcome approach its average. This is the logic of diversification, although it removes only the risks that are specific to each holding, not those shared by all.
• Simulation. Monte Carlo simulation prices an option or estimates a risk figure by averaging the results of many random trials, relying on the law for the average to approach the true value.
• Backtests and track records. The more independent trades or periods a backtest contains, the more its average return can be trusted as an estimate of the true one.
• Position sizing. The Kelly criterion depends on the law to turn a per-bet edge into long-run growth.

The law does not say that outcomes balance out in the short run. A roulette wheel that has landed on red five times is no more likely to land on black next. The belief that it is, known as the gambler's fallacy, mistakes a statement about long-run averages for a force acting on individual trials. Nor does the law say that totals converge. The average profit per trade settles down, but the gap between actual and expected total profit typically grows in absolute terms as trades accumulate, in proportion to the square root of their number.

The law rests on conditions that markets often violate. The observations must be independent, or nearly so, yet returns on different shares fall together in a crisis, and a strategy's trades may all depend on the same market condition. The observations must come from the same distribution, yet markets change, so that an edge measured over ten years may not exist today, which is the problem of stationarity. And the mean must be approached at a useful speed. With fat tails, a handful of extreme observations carries much of the total, and convergence is so slow that a long sample can still give a misleading average.

A strategy that sells insurance against rare events illustrates the danger. It may show hundreds of small gains before a single loss wipes them out. The sample average before that loss overstates the true mean, and no number of quiet periods makes it reliable, because the events that matter most have not yet been observed.