Linear Regression

Linear regression is a statistical method that describes the relationship between one variable and one or more others by fitting a straight line through the data. The variable to be explained is called the dependent variable, and the variables used to explain it are called independent variables or regressors. The line is chosen to make the squared differences between the observed values and the fitted values as small as possible, a principle known as ordinary least squares.

It is the most widely used tool in regression analysis and in econometrics generally. In finance it is used to measure how a share responds to the market, to estimate hedge ratios, to test whether a signal predicts returns and to attribute a fund's performance to its sources. The version with a single regressor is called simple linear regression. With several it becomes multivariate linear regression.

The simple model is:

• $y_i$ is the $i$-th observation of the dependent variable.
• $x_i$ is the matching observation of the independent variable.
• $\alpha$ is the intercept, the value of $y$ when $x$ is zero.
• $\beta$ is the slope, the change in $y$ for a one-unit change in $x$.
• $\varepsilon_i$ is the error term, the part of $y_i$ the line does not explain.

The least squares estimate of the slope is the covariance of the two variables divided by the variance of the regressor:

The intercept follows from the averages:

Here $\bar{x}$ and $\bar{y}$ are the sample means of the two variables.

Suppose a market index returns 1, 2, 3, -1 and 0 percent in five months, and a share returns 1.5, 2.5, 4.5, -1.5 and 0.5 percent in the same months. The mean market return is 1 and the mean share return is 1.5. The deviations of the market from its mean are 0, 1, 2, -2 and -1, and those of the share are 0, 1, 3, -3 and -1. The sum of the products is $0 + 1 + 6 + 6 + 1 = 14$, and the sum of squared market deviations is $0 + 1 + 4 + 4 + 1 = 10$. The slope is $14 / 10 = 1.4$ and the intercept is $1.5 - 1.4 \times 1 = 0.1$. The share has tended to move 1.4 percent for each 1 percent move in the market, with an extra 0.1 percent a month that the market does not explain.

Three numbers are normally reported with a regression. The coefficient of determination, or R-squared, is the share of the variation in $y$ that the line explains, from 0 to 1. The standard error of each coefficient measures how precisely it is estimated. Dividing a coefficient by its standard error gives a t-statistic, and the associated p-value indicates whether the coefficient is distinguishable from zero. A high R-squared is not required for a regression to be useful: models that predict returns usually explain only a very small part of their variation, yet a small reliable edge can still be valuable.

• Beta and alpha. Regressing a share's excess returns on the market's gives its beta as the slope and its alpha as the intercept, the framework of the capital asset pricing model.
• Factor models. Adding regressors such as size, value and momentum turns the same calculation into a factor model, used for risk management and performance attribution.
• Hedge ratios. The slope from regressing one asset's price on another's estimates how many units of the second offset one unit of the first. It is the starting point for pairs trading.
• Signal testing. Regressing future returns on a current indicator tests whether the indicator has predictive power.

The standard results assume that the true relationship is linear, that the errors are unrelated to each other and to the regressors, and that their variance is constant. Financial data often break these assumptions. Volatility changes over time, errors are correlated from one period to the next, and extreme observations have a large influence because the method squares every error.

A particular trap arises with price levels. Regressing one trending series on another commonly produces a high R-squared and significant coefficients even when the two are unrelated, a result known as spurious regression. The usual remedies are to work with returns instead of prices, or to test the series for stationarity and cointegration first.

Relationships also change. A beta estimated over five years may not describe the next year, and adding regressors until the historical fit looks good leads to overfitting. Finally, a regression measures association. It does not by itself show that changes in one variable cause changes in another.