Reinforcement Learning

Reinforcement learning is a type of machine learning in which a program, called an agent, learns how to act by interacting with an environment. The agent observes the current situation, chooses an action and receives a numerical reward, and over many repetitions it adjusts its behaviour to collect as much reward as possible. No one tells the agent the correct action; it must discover good behaviour by trial and error.

This differs from supervised learning, where a model learns from examples labelled with the right answer, and from unsupervised learning, where a model looks for structure in unlabelled data. Reinforcement learning is suited to problems that unfold over time, in which an action taken now affects what is possible later. In finance it has been studied for order execution, market making, hedging and portfolio allocation.

• State. The information the agent sees at each step, such as current prices, the order book and its own inventory.
• Action. The choice available to the agent, such as buying, selling or waiting.
• Reward. A number received after each action that scores the immediate result, such as profit minus costs.
• Policy. The rule the agent follows to map states to actions. Learning means improving the policy.
• Value function. An estimate of the total future reward expected from a state, or from taking a given action in a state.

The agent must balance exploration, trying actions whose results are uncertain, against exploitation, repeating actions that have worked. The mathematical setting is the Markov decision process, developed from the dynamic programming work of Richard Bellman in the 1950s.

One of the best-known methods is Q-learning, introduced by Christopher Watkins in 1989. It keeps an estimate $Q(s,a)$ of the value of taking action $a$ in state $s$ and revises it after every step:

• $\alpha$ is the learning rate, between 0 and 1, which sets how far each new experience moves the estimate.
• $r$ is the reward just received.
• $\gamma$ is the discount factor, between 0 and 1, which sets how much future rewards count relative to immediate ones.
• $s'$ is the new state, and $\max_{a'} Q(s',a')$ is the value of the best action available there.

For example, suppose the current estimate is 2.0, the reward received is 1.0, the best value in the new state is 3.0, the discount factor is 0.9 and the learning rate is 0.1. The target is $1.0 + 0.9 \times 3.0 = 3.7$. The gap between target and estimate is 1.7, and the new estimate is $2.0 + 0.1 \times 1.7 = 2.17$.

A table of values is practical only when states and actions are few. In deep reinforcement learning the table is replaced by an artificial neural network that estimates values or outputs the policy directly, an application of deep learning. The approach drew wide attention when the research company DeepMind trained agents to play Atari video games from screen images and, in 2016, when its AlphaGo program defeated the professional Go player Lee Sedol. The standard textbook on the subject is by Richard Sutton and Andrew Barto.

The clearest financial use is optimal execution. An agent that must sell a large block of shares within a time limit chooses how much to trade at each step, and is rewarded for a good average price net of market impact. This is a sequential problem with a well-defined goal, and it extends the classical models behind an execution strategy.

Other studied uses include market making, where the agent sets bid and ask quotes while controlling inventory; hedging of options under realistic costs, as an alternative to textbook delta hedging; and portfolio optimization, where the agent reallocates capital over time. The reward is often a risk-adjusted measure such as the Sharpe ratio and should include every transaction cost.

Markets are a much harder setting than games. A game can be replayed millions of times under fixed rules, but there is only one market history, and it is short relative to the data these methods need. Financial returns contain far more noise than signal, and the behaviour of markets changes over time, so a policy learned in one period may fail in the next.

Training usually takes place in a simulator built from historical data, and the simulator may not reproduce how other participants would react to the agent's own orders. An agent can learn to exploit flaws in the simulation instead of real patterns, a form of overfitting that ordinary backtesting may not reveal. Learned policies are also hard to interpret, which complicates risk control and regulatory review. For these reasons, published results should be read with caution, and production use has been concentrated in narrow, well-defined tasks such as execution.