Ensemble Learning

Ensemble learning is a technique in machine learning in which several models are trained on the same problem and their outputs are combined into a single prediction. The combined model is called an ensemble. Individual models make different errors, and when those errors are at least partly unrelated they tend to cancel when the predictions are averaged or put to a vote.

The idea is the statistical counterpart of asking many people and taking the average answer. It is among the most reliable ways to improve predictive accuracy and is standard practice in data competitions and in industry. In finance, ensembles are used in credit scoring, fraud screening and return forecasting, where signals are weak and single models are unstable.

The benefit can be shown with the variance of an average. If $M$ models each have prediction variance $\sigma^2$ and every pair has correlation $\rho$, the variance of their average is:

• $\sigma^2$ is the variance of a single model's prediction.
• $\rho$ is the correlation between the predictions of any two models.
• $M$ is the number of models.

Suppose each model has a variance of 4 and the models have a pairwise correlation of 0.25. With 100 models, the variance of the average is (0.25 × 4) + (0.75 × 4 ÷ 100) = 1 + 0.03 = 1.03, about a quarter of the variance of one model. The second term shrinks as models are added, but the first does not. Adding more models cannot reduce variance below $\rho \sigma^2$, so the gain depends on making the models different from one another. The logic is the same as diversification in a portfolio, where combining assets lowers risk only to the extent that they are not perfectly correlated.

• Bagging. Short for bootstrap aggregating, introduced by Leo Breiman in 1996. Each model is trained on a different random sample of the data drawn with replacement, and the predictions are averaged. Bagging mainly reduces variance.
• Random forests. A refinement of bagging for the decision tree, also due to Breiman. Each tree considers only a random subset of the features at each split, which makes the trees less alike and the average more effective.
• Boosting. Models are built one after another, each concentrating on the cases the earlier ones predicted badly. AdaBoost and gradient boosting are the main forms. Boosting mainly reduces bias and is the leading method for tabular data.
• Stacking. Models of different types are trained, and a further model learns how best to weight their predictions.
• Simple averaging and voting. The outputs of several independently built models are averaged, or the majority class is taken.

Combining forecasts has a long history in economics. A 1969 paper by Bates and Granger showed that a weighted average of two forecasts can beat both, and equal-weighted averages of economic forecasts have proved hard to improve on since. Modern ensemble methods continue that line of work.

Lenders use boosted trees and random forests alongside the traditional credit score to predict default. Payment firms use them for fraud detection, where they handle mixed data types and complex interactions well. In asset management, ensembles of trees are used to forecast stock returns from many company characteristics at once, and academic studies of machine learning in asset pricing have found tree ensembles and neural networks among the better-performing methods.

Quantitative trading firms apply the same principle one level up. Instead of relying on one signal, they combine many weak predictors, each with a small edge, into a single forecast. Several models of the same type trained on different periods or with different settings may also be averaged to make a strategy less dependent on any one choice. A trading classifier built as an ensemble tends to give steadier probabilities than a single model.

An ensemble gives up clarity for accuracy. A single small tree or a linear regression can be read and explained; a forest of five hundred trees cannot. This matters where a lender must give reasons for a decision or a risk committee must approve a model. Tools that estimate each feature's contribution help but do not fully restore transparency. Ensembles also cost more to train, store and run, which can matter where predictions are needed in microseconds.

Ensembles reduce overfitting but do not prevent it. Boosting in particular will fit noise if run for too many rounds, and the number of rounds, tree depth and learning rate are normally set by cross-validation. With financial time series, that validation must respect the order of time, or the model will be tuned on information from the future, a form of look-ahead bias.

The benefit also depends on the models being different. Models trained on the same data with the same features share the same blind spots, and averaging them removes little error. If all the component models learned a relationship that later disappears, the ensemble fails with them. Results from backtesting an ensemble strategy therefore need the same caution as those of any other model.