Multiple Testing and the Factor Zoo

Layer 1: The Intuition

1.1 The Coin-Tossing Champion

Have 1,000 people each flip a coin 10 times and record the number of heads. Almost certainly, someone will flip 9 or 10 heads. Are they a "coin-tossing genius"? Clearly not; it is simply a matter of having enough people so that someone will eventually get an extreme result by chance.

In quantitative research:

• A researcher tests 100 factors, each using a 5% significance level;
• Even if all factors are useless, there will be an average of 5 "significant" ones;
• If only these 5 are reported, 5 false discoveries have been created.

1.2 The Factor Zoo

Layer 2: Correction Methods

2.1 Raising the t-statistic Threshold

Increasing the t-statistic from 2 to 3 might not sound like much, but the corresponding p-value drops from approximately 0.05 to approximately 0.003, making the requirement over ten times stricter.

2.2 Deflated Sharpe Ratio (DSR)

Significance testing for the Sharpe Ratio needs to consider three things:

1. Number of attempts: The more attempts made, the higher the maximum Sharpe Ratio found within the noise;
2. Sample length: The shorter the sample, the less reliable the Sharpe estimate;
3. Shape of the return distribution: Strategies with negative skewness or fat tails have overestimated Sharpe Ratios.

The logic of DSR is: first calculate "what the expected maximum Sharpe Ratio is among $N$ pure noise attempts" ($SR_0$), and then test whether the observed Sharpe is significantly higher than $SR_0$:

$$SR_0 = \sqrt{V}\left[(1-\gamma)\,\Phi^{-1}\!\Big(1 - \frac{1}{N}\Big) + \gamma\,\Phi^{-1}\!\Big(1 - \frac{1}{N e}\Big)\right],$$

$$\text{DSR} = \Phi\!\left(\frac{(\widehat{SR} - SR_0)\sqrt{T-1}}{\sqrt{1 - \hat\gamma_3 \widehat{SR} + \frac{\hat\gamma_4 - 1}{4}\widehat{SR}^2}}\right),$$

Where $V$ is the variance of the Sharpe Ratios across the attempts, $\gamma \approx 0.5772$ is Euler's constant, $\hat\gamma_3, \hat\gamma_4$ is the skewness and kurtosis of the returns, and $T$ is the sample length. DSR can be understood as "the probability that this strategy's true Sharpe Ratio is greater than zero."

Layer 3: Code Implementation

import numpy as np
from scipy.stats import norm, skew, kurtosis
EULER = 0.5772156649

def expected_max_sharpe(n_trials, var_sr):
    """Expected maximum Sharpe ratio under pure noise in n independent trials (Bailey & López de Prado)"""
    return np.sqrt(var_sr) * ((1 - EULER) * norm.ppf(1 - 1 / n_trials)
                              + EULER * norm.ppf(1 - 1 / (n_trials * np.e)))

def deflated_sharpe(returns, n_trials, var_sr):
    T = len(returns)
    sr = returns.mean() / returns.std()               # Non-annualized (per-period) Sharpe
    sr0 = expected_max_sharpe(n_trials, var_sr)
    g3, g4 = skew(returns), kurtosis(returns, fisher=False)
    z = (sr - sr0) * np.sqrt(T - 1) / np.sqrt(1 - g3 * sr + (g4 - 1) / 4 * sr ** 2)
    return sr, sr0, norm.cdf(z)

rng = np.random.default_rng(3)
T = 252 * 3
r = rng.normal(0.0008, 0.01, T)                       # A "looks good" strategy
print("Annualized Sharpe:", round(r.mean() / r.std() * np.sqrt(252), 2))
for n in [2, 10, 100, 1000]:
    sr, sr0, dsr = deflated_sharpe(r, n, var_sr=1 / T)
    print(f"Trials {n:>5}: Max Sharpe from noise (annualized)={sr0*np.sqrt(252):.2f}  DSR={dsr:.3f}")

Output:

Annualized Sharpe: 2.1
Number of trials     2: Max noise Sharpe (annualized)=0.30  DSR=0.999
Number of trials    10: Max noise Sharpe (annualized)=0.91  DSR=0.979
Number of trials   100: Max noise Sharpe (annualized)=1.46  DSR=0.862
Number of trials  1000: Max noise Sharpe (annualized)=1.88  DSR=0.645

(Here var_sr = 1/T is a simplified assumption: that the variance of the Sharpe estimates for each attempt is only the sampling error variance. In actual use, you should use the sample variance of the Sharpe Ratios of the actual strategies you have tested.)

3.1 Interpretation

Consider the same strategy with a three-year annualized Sharpe Ratio of 2.1:

The credibility of the same number depends on the process used to find it. This is why serious quantitative teams require recording every single attempt.

3.2 Engineering details

Summary