Overfitting

Overfitting is the error of building a model or trading strategy that matches the data used to create it so closely that it captures random noise as if it were a real pattern. An overfitted model performs very well on the data it was fitted to and poorly on new data. The more flexible the model and the smaller or noisier the data set, the greater the danger.

The problem is general to statistics and machine learning, but it is especially severe in finance. Market data contain little signal and a great deal of noise, the amount of history is limited, and a researcher can test an almost unlimited number of ideas against the same record. In trading the usual form is backtest overfitting: a strategy tuned until its historical results look excellent, which then fails when real money is committed.

Any data set is a mixture of structure that will repeat and accidents that will not. A model with few adjustable parts can capture only broad structure. A model with many parameters, rules or inputs can bend to fit the accidents as well. Fitting a straight line through ten points with linear regression leaves some error, while a ninth-degree polynomial passes through all ten exactly and predicts the eleventh badly. Statisticians describe this as a trade-off between bias, the error from a model that is too simple, and variance, the error from one that is too sensitive to the particular sample.

In strategy research the parameters are things like indicator lengths, entry thresholds, stop distances, the choice of markets and the dates of the test. Each one that is adjusted after looking at results is a degree of freedom. Complex methods such as a deep decision tree or an artificial neural network have thousands or millions of them.

A strategy can be overfitted even if it has no parameters at all, simply by being the best of many that were tried. If enough random strategies are tested, some will look good by luck. For $N$ independent strategies with no true edge, the expected value of the best result, measured in standard errors, is bounded by:

• $N$ is the number of independent strategies or variations tested.
• $E[\max]$ is the expected value of the highest test statistic among them when none has any real merit.

For 100 trials the bound is $\sqrt{2 \times 4.605} = 3.03$, and the exact expected value is about 2.5. Consider what this means for the Sharpe ratio. Over a five-year backtest, the standard error of an annualised Sharpe ratio is roughly $1 / \sqrt{5} = 0.45$. The best of 100 worthless strategies would therefore be expected to show a Sharpe ratio of about $2.5 \times 0.45 = 1.1$, a figure that would look attractive in isolation.

This is why a result that passes a conventional test of the null hypothesis, with a p-value below 5 percent, means little when it was selected from many attempts. Researchers have proposed higher hurdles in response. Campbell Harvey, Yan Liu and Heqing Zhu argued in 2016 that a newly proposed return factor should clear a t-statistic of about 3, not 2. David Bailey and Marcos López de Prado proposed a deflated Sharpe ratio that adjusts for the number of trials.

• Results that are far better than those of comparable strategies run with real money.
• Performance that collapses when a parameter is changed slightly, showing a narrow peak instead of a broad plateau.
• A large number of rules or conditions relative to the number of trades.
• Rules with no economic explanation, such as trading only on particular weekdays because those happened to work.
• Profits concentrated in a handful of trades or one short period.
• A sharp drop in performance from the fitting period to data held back.

The basic defence is to judge a model on data it has not seen. The record is split into an in-sample period for development and an out-of-sample period reserved for a final check. Cross-validation repeats this over several splits. For time series, walk-forward optimization fits parameters on one window, tests on the next, and rolls forward, so that every test uses only earlier data. The held-back data lose their value once they have been consulted repeatedly, since choosing among models by their out-of-sample results is itself a form of fitting.

Other measures reduce the scope for fitting in the first place. These include starting from a stated economic hypothesis, limiting the number of parameters, penalising complexity, averaging many simple models through ensemble learning, and recording every variation tried so that the count of trials is known. Testing the same rule on other markets or periods, and a spell of paper trading before committing capital, provide further evidence.

Overfitting is often found together with other flaws that inflate a backtest. Look-ahead bias uses information that was not available at the time. Survivorship bias tests only on assets that still exist. Unrealistic assumptions about costs and execution overstate what could have been earned. Each produces a gap between simulated and live results, and a careful research backtest addresses all of them. A degree of decay after a strategy goes live is normal even without these errors, because markets change and discovered patterns attract competition.