Support Vector Machine (SVM)

A support vector machine (SVM) is a machine learning algorithm used mainly for classification, the task of assigning an observation to one of two or more categories. It learns from labelled examples and draws a dividing boundary between the categories, choosing the boundary that leaves the largest gap, or margin, between itself and the nearest examples on each side. A variant, support vector regression, predicts numerical values.

The method was developed by Vladimir Vapnik and colleagues. The use of kernels to produce curved boundaries was introduced in 1992, and the soft-margin form used today was published by Corinna Cortes and Vapnik in 1995. In finance, support vector machines have been applied to forecasting the direction of prices, credit scoring, fraud detection and the classification of market conditions.

Each observation is described by a set of numerical inputs called features. For two features the observations can be plotted as points on a plane, and a classifier must find a line that separates one class from the other. Many lines may do so. The support vector machine selects the one that is as far as possible from the closest points of both classes, on the reasoning that a boundary with a wide margin is more likely to classify new data correctly.

The points lying on the edge of the margin are called support vectors. They alone determine the boundary; the other observations could be removed without changing it. With more than two features the line becomes a plane or, in general, a hyperplane.

A new observation is classified by the sign of a linear function:

• $x$ is the vector of feature values for the observation.
• $w$ is the vector of weights learned from the training data.
• $b$ is a constant, the bias term.
• $w \cdot x$ is the sum of each weight multiplied by its feature.

The width of the margin is $2 / \|w\|$, where $\|w\|$ is the length of the weight vector, so training amounts to making the weights as small as possible while still classifying the examples correctly.

Suppose a model trained to label the next day as up (+1) or down (-1) has weights of 0.8 on a momentum feature and -0.5 on a volatility feature, with a bias of -0.1. For a day with scaled feature values of 0.6 and 0.4, the function gives $0.8 \times 0.6 - 0.5 \times 0.4 - 0.1 = 0.48 - 0.20 - 0.10 = 0.18$. The result is positive, so the prediction is up. The length of the weight vector is $\sqrt{0.64 + 0.25} = 0.943$, and the margin is $2 / 0.943 = 2.12$.

Real data are rarely separable by a clean boundary. The soft-margin version allows some training points to fall inside the margin or on the wrong side, at a penalty. A parameter, usually written $C$, sets the size of the penalty: a high value insists on classifying the training data correctly and risks overfitting, while a low value accepts more errors in exchange for a wider, simpler boundary.

Where the classes cannot be divided by a straight boundary at all, the method uses a kernel. A kernel function measures the similarity between pairs of observations in a way that is equivalent to mapping the data into a space of more dimensions, where a flat boundary may exist. The radial basis function kernel is the most widely used. This device is known as the kernel trick, because the mapping never has to be computed explicitly.

• Direction forecasting. Features built from past returns, volume and technical indicators are used to predict whether an asset will rise or fall, producing a trading signal.
• Credit and default prediction. Borrowers or companies are classified as likely or unlikely to default from financial ratios and account data.
• Fraud and anomaly detection. A one-class variant learns the boundary of normal behaviour and flags transactions outside it.
• Regime classification. Periods are labelled as calm or turbulent so that a strategy can adapt.

Support vector machines were prominent in academic studies of financial prediction during the 2000s. In much current practice they have been displaced by ensemble learning methods built on the decision tree and by deep learning, though they remain a standard reference method.

The method performs well when the number of features is large relative to the number of observations, and its solution is found by a well-behaved optimization with a single best answer. It is less convenient in other respects. Features must be scaled to comparable ranges. The kernel and its parameters have to be chosen, normally by cross-validation. Training time grows quickly with the number of observations, and the basic method outputs a class, not a probability.

The larger difficulty in markets lies with the data, not the algorithm. Price data contain little predictable signal, and relationships shift over time. A model can classify its training period accurately and fail afterwards. Results must be checked on later, unseen data, with care to avoid look-ahead bias, and a classification accuracy slightly above one half does not ensure a profit once the costs found in backtesting are included.