Cross-Validation

Cross-validation is a technique for judging how well a statistical or machine learning model will perform on data it has not seen. The data set is divided into parts. The model is fitted on some parts and tested on the rest, and the process is repeated with different divisions so that every observation is used for testing once. The average test result is the estimate of out-of-sample performance.

The purpose is to detect overfitting. A model judged on the data used to fit it will always look better than it is, because it has partly memorized that data. Cross-validation is used to compare models, to choose settings and to decide how complex a model should be. In trading research it needs to be adapted, because financial data is ordered in time and standard methods can let information from the future leak into the test.

The standard form is k-fold cross-validation. The data is split into $k$ groups of equal size, called folds. The model is trained on $k - 1$ folds and tested on the remaining one, and this is done $k$ times so that each fold serves once as the test set. The overall score is the average:

Here $k$ is the number of folds, commonly 5 or 10, and $E_i$ is the error, or other performance measure, on fold $i$ when the model was trained on the other folds.

Suppose a forecasting model is tested with five folds and its errors on the five test sets are 2.0, 2.4, 1.8, 2.6 and 2.2. The cross-validated error is (2.0 + 2.4 + 1.8 + 2.6 + 2.2) ÷ 5 = 11.0 ÷ 5 = 2.2. If the same model had an error of 1.1 on its own training data, the gap between 1.1 and 2.2 shows how much of the in-sample fit was illusory. The spread of the five values also indicates how stable the model is.

• Hold-out validation. The data is split once into a training set and a test set. It is simple but wastes data, and the result depends on where the split falls.
• Leave-one-out. Each observation in turn is the test set. It uses the data fully but is slow and gives noisy estimates.
• Stratified k-fold. Folds are built so that each has the same mix of classes, which matters for a classifier when one class is rare.
• Nested cross-validation. An inner loop chooses the model's settings and an outer loop measures performance, so that the tuning itself does not bias the final estimate.

Ordinary k-fold cross-validation assigns observations to folds at random. That is sound when observations are independent. In a time series they are not: neighboring days resemble each other, and features or labels often span several days. A model trained on Monday and Wednesday and tested on Tuesday has in effect seen the answer. The test score is then inflated by look-ahead bias.

Several adaptations are used in finance:

• Walk-forward validation. The model is trained on data up to a date and tested on the period after it, then the window is moved forward and the step repeated. Every test uses only earlier data. See walk-forward optimization.
• Purging. Training observations whose labels overlap in time with the test set are removed.
• Embargo. A further gap is left after each test block before training data resumes, to allow for serial correlation.

Purged k-fold cross-validation with an embargo, set out by Marcos López de Prado in his 2018 book on financial machine learning, has become a common reference method.

Researchers use cross-validation at three points. The first is choosing settings, such as the depth of a decision tree, the penalty in a regularized linear regression, or the size of an artificial neural network. The second is choosing among candidate models or sets of features. The third is estimating how a finished model would have performed, as part of a research backtest.

Compared with a single historical backtest, cross-validation tests a strategy on several different stretches of history, which shows whether results depend on one unusual period. It is also the standard way to tune each model within ensemble learning.

Cross-validation reduces overfitting but cannot rule it out. If a researcher runs it on hundreds of model variants and keeps the best, the winning score is itself selected for luck. The cure is to hold back a final test set that is used only once, and to keep a count of how many variants were tried.

The method also assumes that the future will resemble the past. Markets change: relationships weaken as they become known, and regimes shift. A model validated on one decade can fail in the next for reasons no resampling scheme can foresee, a failure of stationarity. Financial samples are short relative to the noise in returns, so each fold contains few independent observations and the estimate carries a wide margin of error. Cross-validation results are therefore treated as an upper bound on what to expect in live trading.