Unsupervised Learning

Unsupervised learning is a category of machine learning in which a model is given data without any labels or correct answers and is asked to discover structure in it. The algorithm might group similar observations together, find a small number of underlying factors that explain many variables, or identify observations that do not fit the general pattern. There is no target to predict.

It is contrasted with supervised learning, where each training example comes with a known outcome, and with reinforcement learning, where an agent learns from rewards. In finance, unsupervised methods are used to group assets by behaviour, to extract common risk factors, to detect unusual activity and to recognise changes in market conditions.

• Clustering. Observations are divided into groups so that members of a group resemble one another more than they resemble members of other groups. K-means clustering assigns each observation to the nearest of a chosen number of centres. Hierarchical clustering builds a tree of nested groups.
• Dimensionality reduction. Many correlated variables are replaced by a few combinations that retain most of the information. Principal component analysis, which dates from the work of Karl Pearson in 1901 and Harold Hotelling in 1933, is the standard technique.
• Anomaly detection. The model learns what normal data look like and flags observations that depart from it. This is the basis of much anomaly detection in payments and market surveillance.
• Representation learning. A type of artificial neural network called an autoencoder learns to compress data and reconstruct it, producing compact features for other models.

Principal component analysis finds new variables, the components, that are uncorrelated with one another and ordered by how much of the data's variation each explains. They are obtained from the eigenvalues and eigenvectors of the covariance matrix. The share of total variance explained by component $i$ is:

• $\lambda_i$ is the eigenvalue associated with component $i$.
• $\sum_{j} \lambda_j$ is the sum of all eigenvalues, equal to the total variance of the data.

Suppose the returns of four assets give eigenvalues of 6.0, 2.5, 1.0 and 0.5. The total is 10. The first component explains $6.0 / 10 = 60$ percent of the variance and the second 25 percent, so two components capture 85 percent of what four variables contained. The remaining two could be dropped with little loss.

The best-known application is to interest rates. Analyses of government bond yields have repeatedly found that three components account for nearly all movement in the yield curve, and that they correspond to changes in its level, slope and curvature. Risk managers hedge these three exposures instead of each maturity separately.

In equities, components extracted from share returns serve as statistical risk factors in a factor model. The first typically represents the market as a whole. Statistical arbitrage strategies trade the part of each share's return that the main components do not explain.

Clustering is used to group shares by how their prices actually move, which can differ from the official sector classification, and to select candidates for pairs trading. Hierarchical risk parity applies hierarchical clustering to the correlation matrix before allocating capital, as an alternative to conventional portfolio optimization. Clustering and related models are also applied to market history itself, to label periods as distinct regimes, such as calm and turbulent, so that a strategy can behave differently in each.

Outside trading, unsupervised methods segment customers, group documents by topic in natural language processing, and screen transactions for fraud where confirmed examples are too few to train a supervised model.

Because there are no correct answers, there is no simple measure of whether an unsupervised model is right. Results are judged by internal measures of how compact and well separated the groups are, by stability when the data are changed slightly, and by whether the output is useful for a later task. Much depends on choices made by the analyst: the number of clusters, the measure of similarity and the scaling of the variables. Different reasonable choices can produce different structures.

Financial data add their own difficulties. Correlations between assets are estimated with considerable noise, particularly when the number of assets is large relative to the length of history, so some apparent structure is spurious. Relationships also drift, and clusters or components estimated in one period may not persist. A grouping that fits the past closely can be a form of overfitting. Components beyond the first few are often hard to interpret, which limits their use where decisions must be explained.

Unsupervised learning is frequently a preparatory step. Reduced or clustered data are passed to a supervised model, such as a classifier, which then has fewer and cleaner inputs. The large language models now used to analyse financial text are trained largely by a related approach, self-supervised learning, in which the model predicts hidden parts of unlabelled data. These methods belong to the broader field of deep learning.