Pairs trading within the S&P 500

The screen

A small pipeline pulls adjusted closes for the S&P 500 constituents, scraped from Wikipedia, and stores them in PostgreSQL for the research notebook to read.

Every possible pair is tested for correlation on the GPU, with cointegration tests alongside. The top 2,000 pairs are then backtested one by one, with a Sharpe ratio and volatility for each pair and for the strategy as a whole.

The trading rule

For each pair the spread is the difference in log prices, standardised into a z-score. The strategy goes long the spread when the score falls below minus two, short it above two, and is meant to close out once it comes back inside one.

What was wrong with the backtest

Looking at it again, two things made the results meaningless. The z-score used the mean and standard deviation of the whole sample, so every entry was judged against information from the future. And the pairs were chosen on the same years they were tested on.

There was also a plain bug. An exit and a day with no signal were both stored as zero, and the positions were then forward filled, so the exit never happened. Once a pair was opened it only ever flipped between long and short.

The manual tests on PEP and KO, SPY and TLT, and ADI and ZBRA showed cumulative returns of 73% to 127%. With look-ahead in the entries and no exits, those numbers say nothing about the strategy.

What it taught me

It is exactly the failure the rest of this site is built around: a backtest that looks good because it quietly knows the future. The point-in-time data platform and the walk-forward checks in Argus exist to catch it.

The version I would build now standardises the spread on a rolling window, screens on one period and tests on the next, charges transaction costs, and closes both legs together when a stop is hit.