The most frequent and natural data to use for developing quantitative strategies is price and volume data. It represents the purest form of information for a stock. Whether investors fully believe the efficient market hypothesis or not, the stock price itself is meant to reflect all known information and expectations up to that point. There exist numerous price-volume data providers and depending on your use case (daily prices or higher frequency tick or bar data) there are specialist providers for all types of price data. For our use case and our longer term investing strategies, we use daily price data only for our strategy construction.

No matter who your data provider is, and arguably price data is already very accurate, it makes sense to employ some outlier control and cleaning of the data just to protect your quantitative strategy development. 

Some simple examples of price data cleaning (remember that price generates return data which is what we use for modelling in our strategies a lot of the time)

Apply checks to ensure close price is between high and low price data

Apply similar checks to ensure low < high and other such tests

Impose fixed caps on the absolute size allowed for a daily return e.g choosing something like a 75 or 100% return cap

Winsorize return data on a daily basis to cap daily return data at 99.5% e.g

Show a quick graph of how many low > high or close outside high low etc

Show a quick graph of the return trimming and how much it impacts each day. Finish with a histogram of return distribution

Corporate actions are company specific news that impacts the number of shares or number of companies in some way that changes the number of shares on issue or the number of companies in our universe. To give some specific examples, let's take a stock split. A company may split their stock for many reasons and the split may come in the form of a X-for-1 (stock split) or a 1-for-X (so-called reverse split). A standard stock split has the following properties

stock split does not change the market cap of the company

in a standard stock split, each share is replaced by X shares and lowers the old price by the ratio needed to keep the market cap of the company unchanged

stock splits are aimed to increase the overall liquidity of the stock by allowing a larger total number of shares to be traded

companies may stock split to keep the share price at a lower value and therefore keep it affordable for more investors

reverse splits are the opposite and decrease liquidity, increase share price and again keep the market cap of the company the same

Other forms of corporate actions include company spin-offs. A spin-off occurs when a company creates a new corporate identity that is investible and tradeable by the public and is treated as a fully separate new company. A parent company may spin off an internal business, or division of the company, as a separate entity with the goal that the spun-off company can be worth more as independent entity than as part of the larger parent company.

To deal with these forms of corporate actions as described, our data pipeline therefore needs to 

 all historical data to reflect the new up-to-date stock price adjustment and shares outstanding adjustments. Most price and volume data providers outline how the adjustment process should work and how the adjustment factor should be calculated and applied to create corporate-action adjusted historical data.

Volume can become very sensitive to single data points so it is common practice to also include a rolling average (be it mean or median) of the volume data over a short lookback window such as 20 or 30 days. Working with this smoothed volume makes it easier to take a more hollistic approach to volume use within signal construction and potentially keep the signal smoother with less turnover. 

Include some volume plots such as jagged vs the smoothed volume 

Include a plot of volume average per quintile etc

Discussion about price and volume data and how to clean and implement some basic outlier control. 




1. Price and Volume Data

What does it mean for data to be point-in-time ?

Point-in-time within a quantitative framework means that we have full trust that the 

date when the quantitative data was known to the market

 is accurate. Only using data that is point-in-time is a critical concept as it represents, in our backtest and other signal efficacy tests, when your strategies would have had accurate information and we can therefore judge if they should be used within your overall investment portfolio.

The common industry phrase for data that is not point-in-time is 

. The ideal quantitative world is one in which we know, with 100% certainty, that all data (be it price, volume, fundamental, alternative data) is fully point-in-time and contains 

. Unfortunately, it is nearly impossible to guarantee that with absolutely certainty no matter how careful data providers are. Sometimes it is a simple case of 

one data provider can guarantee their point-in-time data but they rely on another provider who has 99% guarantee 

Data sets may be partially point-in-time and stitched together with earlier data that they cannot guarantee is point-in-time 

Do all data providers sell point-in-time data ?

Some companies provide fully point-in-time data

A few companies provide heuristic rules as to how to approximately make their data point-in-time e.g Company X may advise to lag the data Y days to ensure it contains no lookahead

Some newer companies or alternative data companies find it a bit more difficult to guarantee point-in-time data due to their recent existence or lack of resources or experience in full scale data curation, storage and delivery

A retroactive edit occurs when our data provider notices a mistake, either on their data ingestion and dissemination pipeline or a separate data provider to them, and our data provider then seeks to modify 

 their data to their clients without an appropriate timestamp reflected when the edit occurred. 

We can give a specific example of this below, where company X notices a mistake on 20/6/2021 for a piece of data, with an orginal value of 5, that was sent to its clients with a linked timestamp for this data 13/6/2021. This real data should have been 50 instead of the mistakenly entered 5 so company X then changes the data to 50 on 20/6/2021 but keeps the original timestamp of 13/6/2021. 

For a quantitative investment strategy being researched and designed in 2022, unless you were around and actively storing daily snapshots of this data yourself, you would not be aware that on 13/6/2021 in reality your strategy would have ingested and traded on the incorrect figure of 5. We depict this graphically in the example below:

This post is all about point-in-time data and the importance of avoiding lookahead in your data. It is essential to understand the critical importance of this concept and the scale at which it can manipulate backtest results if lookahead has sneaked into your data pipeline. Lookahead can make a weak unrealistic strategy seem investible when in reality it could be loss-making at worst or at best, it's real sharpe ratio is significantly lower than your tests suggest. 

In the figure below, we deliberately inject lookahead into our data and also lag the data. For our backtest results, we plot 10 different lines starting with a lookahead of 2 days, then 1 day, then day 0 and all the way to lag 7 where we delay the data by 7 days and are therefore extra conservative in our approach. 

What is point-in-time data and why is it essential. Are there methods to make your data point-in-time ?

2. Point in Time Data

Survivorship bias represents the flawed concept of using data from companies that are only alive and correct as of today and not accounting for companies, funds or data that were alive in the past but are delisted, dead or not tradeable now. The ability to account for survivorship bias is a fundamental concept to ensure that an investment strategy is realistic and the efficacy metrics employed to evaluate it are robust and correct.

Example of a universe of stocks through time 

Show a plot that has the size of our active universe through time

Are there any other forms of survivorship bias ?

One could argue that some mathematical methods can show survivorship bias. By this we mean, if a newly invented mathematical technique is published now then one could argue that strategies relying on this method, if evaluated during the timeframe before the technique was around, could in fact be a form of survivorship bias.

At the very least, evaluation of such an investment strategy based on new techniques should be conservative in how it treats the results shown before the period when the technique was publicly known. As the results achieved would arguably be artificially high and would have been dampened by other investment managers executing similar strategies and eroding the potential alpha.    

This is a quick post on survivorship bias. It goes through examples of what it is and how it impacts quantitative strategies.

3. Survivorship Bias

This is the first post bah blah blah This is the first post bah blah blah 




4. Fundamental Data

This set of posts is all about data and the correct way to use and interpret that data.

Accurate Data and Concepts

Price and Volume Data

Point in Time Data

Survivorship Bias

Fundamental Data