Portfolio Optimization

Portfolio optimization is the process of choosing the proportions in which to hold a set of assets so as to best meet a stated objective, such as the highest expected return for a given level of risk, subject to any constraints the investor faces. It turns estimates of how assets are expected to behave into a specific set of portfolio weights by solving a mathematical problem.

The discipline grew out of modern portfolio theory, which supplied the first formal statement of the problem in 1952. It is used by asset managers, pension funds, automated advice services and quantitative trading firms. The mathematics is well understood. The difficulty lies in the inputs, which must be forecast, and much of the development since the original theory has been about making the results robust to errors in those forecasts.

The classic formulation minimises portfolio variance for a required expected return:

• $w$ is the vector of portfolio weights, one for each asset.
• $\Sigma$ is the matrix of every covariance between pairs of asset returns.
• $\mu$ is the vector of expected returns.
• $R$ is the target expected return for the portfolio.

Solving the problem for a range of targets traces out the efficient frontier. Two points on it receive special attention: the minimum-variance portfolio, which has the lowest risk of all, and the tangency portfolio, which has the highest Sharpe ratio.

With only two assets, the weight of the first in the minimum-variance portfolio has a closed form:

Here $\sigma_1$ and $\sigma_2$ are the standard deviations of the two assets and $\rho$ is the correlation between them. Suppose shares have a standard deviation of 20 percent, bonds 10 percent, and the correlation is 0.2. The numerator is $0.01 - 0.2 \times 0.2 \times 0.1 = 0.006$ and the denominator is $0.04 + 0.01 - 0.008 = 0.042$, so the weight in shares is $0.006 / 0.042 = 0.143$. The minimum-variance portfolio holds 14.3 percent in shares and 85.7 percent in bonds. Its standard deviation is about 9.6 percent, slightly below that of bonds alone, because the small holding of shares diversifies the bonds.

Real portfolios are optimised under constraints. Common ones are a ban on short positions, upper and lower limits on each holding, limits on exposure to sectors or countries, a cap on turnover, and a limit on tracking error against a benchmark. Constraints reflect mandates and regulation, and they also act as a brake on extreme solutions.

The objective can differ too. Some investors minimise a measure of downside loss, such as value at risk or expected shortfall, instead of variance. Some maximise the expected long-run growth rate of wealth, the approach of the Kelly criterion. Others include the transaction cost of moving from the current portfolio to the new one, so that the optimiser trades only when the benefit outweighs the cost. This matters because an optimal portfolio drifts as prices move and must be restored by periodic rebalancing.

The optimiser treats its inputs as exact, but they are estimates. Expected returns are especially hard to forecast, and the optimiser is highly sensitive to them: it puts large weights on assets whose returns happen to be overestimated and large negative weights on those underestimated. Small changes in the inputs can produce completely different portfolios. Critics have described unconstrained mean-variance optimisation as a procedure that maximises estimation error. A well-known study published in 2009 found that, for many data sets, a portfolio with equal weights in every asset performed as well out of sample as optimised portfolios, because the gains from optimisation were offset by errors in the estimates.

The covariance matrix raises its own problem. With 500 assets there are nearly 125,000 distinct correlations to estimate, often from fewer observations than that, and the resulting matrix is unreliable. Fitting weights closely to such inputs is a form of overfitting.

• Shrinkage. Estimates are pulled towards a simple, structured target, such as equal correlations, which reduces their noise.
• Factor structure. A factor model explains the co-movement of many assets through a few common drivers, cutting the number of quantities to be estimated.
• Black-Litterman. This method, developed at Goldman Sachs around 1990, starts from the returns implied by market weights and adjusts them for the investor's own views, in proportion to the confidence held in each view.
• Risk-based allocation. Minimum-variance portfolios, risk parity and hierarchical risk parity use only risk estimates and avoid return forecasts altogether.
• Resampling and simulation. The optimisation is repeated on many simulated versions of the inputs and the results are averaged.
• Machine learning. Newer methods, covered under AI-driven portfolio optimization, apply learning algorithms to forecasting inputs or to choosing weights directly.

Few investors accept an optimiser's output without question. A portfolio manager typically uses it as one input, examines why it recommends what it does, and adjusts constraints until the result is sensible and stable. In quantitative trading, the optimiser is run frequently, sometimes daily, to combine many return forecasts into a portfolio while controlling risk, costs and exposure limits. In strategic asset allocation it is run rarely, with long-horizon assumptions, to set the broad mix of asset classes. In both settings the quality of the result depends far more on the inputs and constraints than on the solver.