The significance level, written , is the cut-off chosen in advance of a statistical test for deciding whether to reject the null hypothesis. It is the probability of rejecting the null hypothesis when it is in fact true, a false positive or Type I error. If the p-value of the test is smaller than , the result is called statistically significant. Common choices are 0.05 and 0.01. The level also fixes the critical value against which a test statistic is compared.
In trading research, a backtest that is significant at 0.05 has a one-in-twenty chance of looking that good when the strategy has no real edge. A researcher who tries twenty variations should therefore expect one to pass by luck alone. For this reason, studies of return anomalies have argued for much stricter thresholds in hypothesis testing when many strategies have been examined, a safeguard against overfitting.