Confidence Interval

A confidence interval is a range of values computed from a sample and used to express the uncertainty in an estimate. Instead of reporting only that a strategy's average monthly return was 1 percent, an analyst reports that the true average lies between, say, 0 and 2 percent with 95 percent confidence. The width of the interval shows how much the estimate could plausibly be off because it is based on limited data.

The method was formalized by the statistician Jerzy Neyman in the 1930s and is a standard part of statistical reporting. In finance, confidence intervals are attached to average returns, volatilities, regression coefficients such as beta, performance ratios and risk forecasts. They are the estimation counterpart of hypothesis testing and use the same ingredients.

For the mean of a sample, the interval is:

• $\bar{x}$ is the sample mean.
• $s$ is the sample standard deviation.
• $n$ is the number of observations.
• $c$ is the critical value for the chosen confidence level.

The term $s / \sqrt{n}$ is the standard error of the mean. For a 95 percent interval and a large sample, $c$ is 1.96, taken from the standard normal distribution. For a small sample, $c$ comes from Student's t-distribution with $n - 1$ degrees of freedom and is somewhat larger, which widens the interval to allow for the extra uncertainty in $s$. The use of the normal curve rests on the central limit theorem.

A strategy has 36 monthly returns with a mean of 1.0 percent and a standard deviation of 3.0 percent. The standard error is 3.0 ÷ √36 = 3.0 ÷ 6 = 0.5 percent.

Using the normal value of 1.96, the margin of error is 1.96 × 0.5 = 0.98, and the 95 percent interval runs from 0.02 to 1.98 percent. Using the more appropriate t value for 35 degrees of freedom, about 2.03, the margin is 2.03 × 0.5 = 1.015, and the interval runs from about −0.02 to 2.02 percent. The second interval includes zero. Three years of data with these figures are not enough to conclude, at this confidence level, that the strategy's true average return is positive.

Because the standard error shrinks with the square root of $n$, halving the width of an interval takes four times as much data. Twelve years of the same monthly results would give a standard error of 0.25 percent and an interval of roughly 0.5 to 1.5 percent.

The confidence level describes the method, not the particular interval. If the procedure were repeated on many independent samples, about 95 percent of the intervals produced would contain the true value. For any single interval the true value either lies inside or does not. The common reading, "there is a 95 percent probability that the true mean is in this range", is not strictly what the calculation delivers, although it is the natural shorthand.

Confidence intervals and significance tests are two views of one calculation. A 95 percent interval that excludes zero corresponds to rejecting the hypothesis of a zero mean at the 5 percent significance level, and to a p-value below 0.05. The interval is often more informative, since it shows the size of the effect and its uncertainty together.

• Strategy evaluation. After backtesting, an interval around the mean return or the Sharpe ratio shows whether the result is distinguishable from zero. A Sharpe ratio estimated from a few years of data has an interval wide enough to span both poor and excellent performance.
• Regression estimates. Software for linear regression reports an interval for each coefficient. A stock's estimated beta of 1.2 might carry an interval from 0.9 to 1.5.
• Risk models. Banks check a value at risk model by counting the days on which losses exceeded the forecast and asking whether the count falls within the range expected if the model were correct.
• Forecasts. Economic and volatility forecasts are commonly published with bands that widen as the horizon lengthens.

The standard formula assumes that observations are independent and drawn from one unchanging distribution. Market data often breaks these conditions. Returns have fat tails, their volatility clusters, and overlapping or smoothed returns are correlated through time. Each of these makes the true uncertainty larger than the formula reports. Analysts respond with standard errors adjusted for autocorrelation, or with the bootstrap, which builds an interval by resampling the data many times.

An interval also says nothing about how the sample was chosen. If a strategy was picked as the best of several hundred tried, its interval is misleadingly narrow and misplaced, because the selection favored a lucky outcome. And an interval describes the past period: if market conditions change, the true value itself moves, and an interval estimated on old data may not cover it.