The Seven Sins of Backtesting

Layer 1: The Intuition

1.1 Hindsight Bias

"If I had bought Bitcoin in 2010..." Anyone can find a perfect path in hindsight. The danger of backtesting lies in the fact that it is a systematic hindsight generator.

A backtesting result is often unreliable not because the code is written incorrectly, but because:

• It uses information that was impossible to know at the time;
• It has been tested too many times on the same historical data;
• It ignores real-world frictions.

1.2 Overview of the Seven Sins

Layer 2: Demo 1: "Good Strategies" on Random Data

2.1 Experimental Design

• Generate 2000 days of pure random walk prices: zero predictability;
• The first 1000 days serve as in-sample, the last 1000 days as out-of-sample;
• Use a moving average crossover strategy (hold when the fast line crosses above the slow line), iterating through all combinations where the fast line period is 2–48 and the slow line period is between fast line + 5 and 195.

import numpy as np, pandas as pd
rng = np.random.default_rng(7)
n = 2000
ret = pd.Series(rng.normal(0, 0.01, n))          # Pure random walk: no predictability
price = 100 * np.exp(ret.cumsum())
ins, oos = slice(0, 1000), slice(1000, n)

def sharpe(x):
    return x.mean() / x.std() * np.sqrt(252) if x.std() > 0 else 0.0

def ma_strategy(price, fast, slow):
    sig = (price.rolling(fast).mean() > price.rolling(slow).mean()).astype(float)
    return sig.shift(1) * price.pct_change()        # Trade today using yesterday's signal to avoid look-ahead bias

results = []
for fast in range(2, 50, 2):
    for slow in range(fast + 5, 200, 5):
        r = ma_strategy(price, fast, slow)
        results.append((fast, slow, sharpe(r.iloc[ins].dropna()), sharpe(r.iloc[oos].dropna())))
res = pd.DataFrame(results, columns=["fast", "slow", "is_sharpe", "oos_sharpe"])
best = res.loc[res["is_sharpe"].idxmax()]
print("Number of parameter combinations tried:", len(res))
print("In-sample best:", best.round(2).to_dict())
print("Mean out-of-sample Sharpe across all combinations:", round(res["oos_sharpe"].mean(), 2))

Output:

Number of parameter combinations tried: 826
In-sample best: {'fast': 44.0, 'slow': 84.0, 'is_sharpe': 0.61, 'oos_sharpe': -0.1}
Mean out-of-sample Sharpe for all combinations: -0.13

2.2 Interpretation

• On a sequence that is known to be unpredictable, after testing 826 parameter sets, the best set has an in-sample Sharpe ratio of 0.61;
• In the out-of-sample period, this parameter set has a Sharpe ratio of −0.1, which is no different from the average of all parameters;
• If you only saw the result "Fast 44, Slow 84, Sharpe 0.61," you would likely believe it;
• Real research is even worse: researchers often test not just parameters, but different indicators, different stock pools, and different time periods. The "number of trials" far exceeds a few hundred, and the in-sample results can be even more attractive. To see how to correct for this, see Multiple Testing and the Factor Zoo.

Layer 3: Demo 2: Forward Error Only Needs One Line

3.1 The same strategy, two ways

sig = (price.rolling(5).mean() > price.rolling(20).mean()).astype(float)
cheat = sig * price.pct_change()                     # Use today's signal to earn today's return
honest = sig.shift(1) * price.pct_change()           # Use yesterday's signal to earn today's return
print("Look-ahead version Sharpe:", round(sharpe(cheat.dropna()), 2), " Correct version Sharpe:", round(sharpe(honest.dropna()), 2))

Output:

Look-ahead version Sharpe: 0.57  Correct version Sharpe: -0.07

3.2 Why the difference is so large

Today's closing price simultaneously affects "today's moving average" (the signal) and "today's return." If the price jumps today, the moving average is more likely to cross up, making the signal more likely to be 1, thus "coincidentally" holding the asset on the day it jumps. This correlation arises entirely from the same data point being used twice, which is impossible in reality: you do not know the closing price before the market closes.

On purely random data, look-ahead bias creates a Sharpe ratio of 0.57 out of thin air. On real data, the fake returns it generates may be even larger and harder to detect because they are often hidden within complex data processing workflows:

• Using the mean and standard deviation of the full sample for normalization (see Factor Engineering);
• Using the Volume Weighted Average Price (VWAP) of the day as the execution price while simultaneously using it to calculate the signal;
• Aligning financial data by reporting period rather than announcement date;
• Shuffling time series during cross-validation in machine learning.

3.3 Defense Checklist

3.4 A Healthy Research Habit

1. Write down your hypothesis and rationale first, then look at the data (to avoid looking at results first and then inventing a story);
2. Record every attempt, including the failures;
3. Run your backtesting framework on simulated or shuffled data: if it can "make money" on random data, the framework definitely has look-ahead bias;
4. Use paper trading or small-capital live trading as the final out-of-sample test.

Summary