Sentiment Analysis

Sentiment analysis is a technique that uses software to determine the attitude expressed in text, typically classifying it as positive, negative or neutral, or assigning it a numerical score. It is a branch of natural language processing. In finance it is applied to news articles, company filings, earnings call transcripts, analyst reports and social media posts, with the aim of measuring how investors and commentators feel about a company, a sector or the market as a whole.

The output is used as an input to trading models, risk monitoring and research. The term is distinct from market sentiment indicators derived from prices and positions, such as option-based volatility indices, investor surveys and short interest, although both seek to measure the mood of the market.

Three broad approaches are in use, in rising order of complexity.

• Dictionary methods. Each word is looked up in a list of positive and negative terms and the counts are combined into a score. General-purpose word lists perform poorly on financial text, where words such as liability, tax or cost are neutral. The finance-specific word lists published by Tim Loughran and Bill McDonald in 2011 were built to address this.
• Statistical classifiers. A classifier is trained by machine learning on examples that humans have labelled, and learns which words and phrases indicate each tone.
• Neural language models. Models based on deep learning read words in context, so they can handle negation and phrasing that word counts miss. FinBERT, a language model tuned on financial text, is a widely used example, and a general-purpose large language model can be instructed to score text directly.

A basic dictionary score is:

• $P$ is the number of positive words in the text.
• $N$ is the number of negative words.
• $S$ ranges from -1, entirely negative, to +1, entirely positive.

An article containing 12 positive words and 8 negative words has a score of $(12 - 8) / (12 + 8) = 0.2$, mildly positive. In practice scores are aggregated across many documents, weighted by the relevance and reach of each source, and compared with their own recent average, since some companies and outlets are habitually described in more positive terms than others.

Academic work has found that textual tone contains information about returns, though the effects are modest and often short-lived. In a 2007 study Paul Tetlock measured the pessimism of a daily Wall Street Journal market column and found that high pessimism predicted downward pressure on prices followed by a reversal. Studies of company-specific news have found that negative words in news stories forecast lower earnings and returns. The tone of earnings announcements and calls has been linked to post-earnings announcement drift.

Results from social media are more contested. A much-cited 2011 paper reported that mood measured from Twitter posts predicted moves in the Dow Jones Industrial Average, but later researchers questioned its robustness. The interpretation of sentiment effects draws on behavioral finance, which studies how investor psychology can move prices away from fundamental value.

Data vendors and trading firms convert text into structured feeds, known as machine-readable news, that tag each item with the companies mentioned, a relevance measure and a sentiment score within milliseconds of publication. Fast strategies trade on the first reading of a headline, a form of news-based trading. Slower strategies combine aggregated sentiment with other signals over days or weeks, an approach described under sentiment trading.

Sentiment is also used defensively. Risk teams monitor sudden changes in the tone or volume of coverage of a holding, and market makers may widen their quotes when a burst of news arrives.

• Language is hard. Sarcasm, irony, hedged statements and industry jargon are easily misread, and a headline may be negative for one company and positive for its competitor.
• Information may be old. Text often describes events that are already reflected in the price, so a strong score does not imply an opportunity.
• Signals decay. As more participants process the same text with similar tools, the advantage shrinks and shifts toward speed.
• Data can be manipulated. Automated accounts and coordinated posting can inflate apparent sentiment, a tactic associated with market manipulation.
• Testing is difficult. Sentiment models have many adjustable choices, which raises the risk of overfitting in backtesting. Language models trained on recent text may also have seen the outcomes they are asked to predict.