P-Value

A p-value is the probability of obtaining a result at least as extreme as the one actually observed, calculated on the assumption that the null hypothesis is true. It is a number between 0 and 1. A small p-value means the observed data would be unusual if there were really no effect, and it is taken as evidence against the null. A large one means the data are unremarkable under the null.

The p-value is the most commonly reported output of hypothesis testing. In finance it accompanies regression coefficients, tests of whether a strategy's returns differ from zero, and tests of time-series properties. It is also among the most misunderstood quantities in statistics, and its misuse is a leading cause of research findings that fail to hold up.

A test statistic is first computed from the data, such as a z-score or a t-statistic. The p-value is then the probability, under the null, of a statistic that far or further from zero. For a two-sided test based on the standard normal distribution, it is:

• $p$ is the p-value.
• $z$ is the observed test statistic.
• $\Phi$ is the cumulative distribution function of the standard normal distribution, the probability of a value at or below its argument.

Suppose a test gives $z = 2.2$. Tables show $\Phi(2.2) = 0.9861$, so $p = 2 \times (1 - 0.9861) = 0.028$. If the null were true, a result this far from zero in either direction would occur about 2.8 percent of the time. A one-sided test, which looks for an effect in one direction only, would give half that, 0.014. For small samples the statistic is compared with Student's t-distribution, which gives slightly larger p-values.

A researcher chooses a significance level before the test, conventionally 5 percent, and rejects the null if the p-value falls below it. This is equivalent to the test statistic exceeding the corresponding critical value: a two-sided p-value of 0.05 matches a z-score of 1.96. The threshold of 0.05 is a convention, not a law of nature. Fields that require stronger evidence use 0.01 or stricter, and reporting the p-value itself lets readers apply their own standard.

• It is not the probability that the null hypothesis is true. A p-value of 0.03 does not mean there is a 3 percent chance the effect is absent. Calculating that would require prior probabilities, as in Bayes' theorem.
• It is not the probability that the result arose by chance alone, for the same reason.
• It does not measure the size or importance of an effect. With a very large sample, a trivially small effect produces a tiny p-value.
• A value above the threshold does not show that there is no effect. It may only reflect a small sample or noisy data.

In 2016 the American Statistical Association issued a statement setting out these points, in response to widespread misinterpretation across the sciences.

A p-value is valid for a single test planned in advance. Trading research rarely works that way. A researcher tries many signals, parameters and markets, and reports the combination that performed best. If twenty unrelated strategies with no real edge are each tested at the 5 percent level, one would be expected to pass by luck. The reported p-value for that winner greatly overstates the evidence. Adjusting analyses until the p-value drops below the threshold is known as p-hacking, and it is a direct route to overfitting in backtesting.

The simplest correction is the Bonferroni adjustment, which divides the significance level by the number of tests:

Here $\alpha$ is the original significance level and $m$ is the number of tests. With 20 tests at an overall level of 5 percent, each must reach a p-value below $0.05 / 20 = 0.0025$. The adjustment is conservative, and less strict alternatives control the expected share of false discoveries instead. The difficulty in practice is that the true number of trials, including those run informally or by earlier researchers on the same data, is often unknown.

Market data raise two further issues. Standard formulas assume independent observations from a stable distribution, but returns show fat tails and clustered volatility, which make naive p-values too small. And statistical significance is not profitability: an alpha that is significant at the 1 percent level may be smaller than the cost of trading it.

Because of these weaknesses, many statisticians recommend reporting more than a p-value. A confidence interval shows the range of effect sizes consistent with the data. Out-of-sample tests show whether a finding repeats. Bayesian methods state directly how the evidence changes the probability of a hypothesis. In finance, higher thresholds for the t-statistic and measures that adjust a strategy's Sharpe ratio for the number of trials serve the same purpose.