Hypothesis Testing

Hypothesis testing is a statistical procedure for deciding whether the data in a sample give enough evidence to reject a stated assumption about the wider population. The assumption under test is called the null hypothesis, and it is set against an alternative hypothesis. The method is used across science, medicine and economics, and it is the standard way of judging whether a result is more than chance.

In finance the procedure is applied to questions such as whether a trading strategy's average return differs from zero, whether a fund manager has skill, whether two price series move together, or whether returns follow a normal distribution. It belongs to inferential statistics, and its modern form combines the significance tests of Ronald Fisher with the decision framework of Jerzy Neyman and Egon Pearson.

A test follows a fixed sequence.

1. The null and alternative hypotheses are stated. The null usually says there is no effect, for example that a strategy's true mean return is zero.
2. A significance level is chosen in advance, commonly 5 percent. It is the probability of rejecting a true null that the researcher is willing to accept.
3. A test statistic is computed from the sample. It measures how far the observed result lies from what the null predicts, in units of standard error.
4. The statistic is compared with a critical value, or converted into a p-value.
5. The null is rejected if the statistic lies beyond the critical value, which is the same as the p-value falling below the significance level. Otherwise the null is not rejected.

Failing to reject is not proof that the null is true. It means only that the data are not strong enough to rule it out.

For a test of whether a mean differs from a hypothesised value, the usual statistic is the t-statistic:

• $\bar{x}$ is the sample mean.
• $\mu_0$ is the value of the mean under the null hypothesis.
• $s$ is the sample standard deviation.
• $n$ is the number of observations.

Suppose a backtest of a strategy covers 1,000 trading days, with a mean daily return of 0.05 percent and a standard deviation of 1.0 percent. The null is that the true mean is zero. The standard error is 1.0 divided by the square root of 1,000, or 0.0316 percent, so $t = 0.05 / 0.0316 = 1.58$. With a sample this large the statistic can be read against the standard normal distribution, whose two-sided 5 percent critical value is 1.96. Since 1.58 is smaller, the null is not rejected: four years of data are not enough to separate this strategy's return from luck. With small samples the statistic is read against Student's t-distribution, and when the population standard deviation is known the z-test is used.

A test can go wrong in two ways. A Type I error, or false positive, rejects a null that is true; its probability is the significance level. A Type II error, or false negative, fails to reject a null that is false. The power of a test is the probability of correctly rejecting a false null, and it rises with sample size, with the size of the true effect and with lower noise in the data.

The two errors pull against each other. A stricter significance level cuts false positives but makes real effects harder to detect. In trading research a false positive means putting money behind a strategy that has no edge, while a false negative means discarding one that works. Which is worse depends on the cost of each, but the first is usually the more expensive.

Quantitative researchers meet hypothesis tests at almost every stage of their work.

• Strategy evaluation. A backtest is tested for whether its mean return or Sharpe ratio is statistically different from zero.
• Time-series properties. The augmented Dickey-Fuller test checks a price spread for stationarity, and the Johansen test checks a group of series for cointegration, before a mean-reverting trade is built on them.
• Model checks. Risk teams test whether the number of days on which losses exceeded a value-at-risk estimate is consistent with the model's stated confidence.
• Performance attribution. Regressions of fund returns on market factors test whether the intercept, the manager's alpha, differs from zero.

The largest problem in finance is multiple testing. If 100 worthless strategies are each tested at the 5 percent level, about five will pass by chance. A researcher who tries many variations and reports only the best has not found a significant result, however good the reported statistic looks. This is the statistical root of overfitting, and corrections such as a higher threshold for the t-statistic or an adjustment for the number of trials are used to offset it.

Standard tests also assume that observations are independent and drawn from a stable distribution. Market returns have fat tails, volatility that clusters and behaviour that shifts between regimes, so the stated error rates are approximate. Statistical significance is not economic significance either: with enough data a very small effect becomes significant, yet it may be too small to survive trading costs. A confidence interval around the estimate, which shows the size of the effect as well as its uncertainty, is often more informative than a bare decision to reject or not.