Agent Factor Mining: Automated Factor Engineering and the Overfitting Defense

Layer 1: The Intuition

1.1 A Tireless Researcher

Imagine you have hired a researcher who:

• Does not need sleep and can test 2,000 ideas in a single night;
• Is excellent at writing reports, providing a fluent explanation for every "discovery";
• Does not proactively tell you how many total attempts they have made.

The next morning, they hand you 60 reports of "significantly effective" factors. How much should you believe them?

1.2 Typical Agent Factor Mining Workflow

This loop itself is a multiple testing machine: the "No → Return to Hypothesis" arrow adds another attempt on the same dataset every time it completes a cycle.

Layer 2: Simulation Experiments

2.1 Setup

• 120 months, 300 stocks per month;
• The Agent generates 2,000 candidate factors, of which only 1 has true predictive power; the remaining 1,999 are pure noise;
• The first 84 months are given to the Agent for research (in-sample), and the remaining 36 months are locked in a "safe" (out-of-sample).

import numpy as np, pandas as pd
rng = np.random.default_rng(99)
T, N, K = 120, 300, 2000          # 120 months, 300 stocks, Agent generates 2000 candidate factors
fwd = rng.normal(size=(T, N))     # future returns
true_factor = rng.normal(size=(T, N)); fwd += 0.05 * true_factor   # only one true factor is hidden

def mean_ic(F, R):
    Fz = (F - F.mean(1, keepdims=True)) / F.std(1, keepdims=True)
    Rz = (R - R.mean(1, keepdims=True)) / R.std(1, keepdims=True)
    ic = (Fz * Rz).mean(1)
    return ic.mean(), ic.mean() / ic.std() * np.sqrt(len(ic))   # IC mean and t-statistic

split = 84   # first 7 years for Agent "research", last 3 years locked in a safe
res = []
for k in range(K):
    F = true_factor if k == 0 else rng.normal(size=(T, N))
    ic_is, t_is = mean_ic(F[:split], fwd[:split])
    ic_oos, t_oos = mean_ic(F[split:], fwd[split:])
    res.append((k, ic_is, t_is, ic_oos, t_oos))
df = pd.DataFrame(res, columns=["k", "ic_is", "t_is", "ic_oos", "t_oos"])

passed = df[df["t_is"] > 2]
print("Number of candidates with in-sample t>2:", len(passed), " (of which true factor:", int((passed['k'] == 0).sum()), ")")
print("Out-of-sample IC mean of these candidates:", round(passed["ic_oos"].mean(), 4))
print("Still t>2 out-of-sample:", int((passed["t_oos"] > 2).sum()), "; true factor in-sample t =",
      round(df.loc[0, "t_is"], 1), " out-of-sample t =", round(df.loc[0, "t_oos"], 1))
strict = df[df["t_is"] > 3.0]
print("Using t>3 threshold:", len(strict), " passed, of which true factor:", int((strict['k'] == 0).sum()), ")")

Output:

Candidates with in-sample t>2: 61 (including 1 true factor)
Out-of-sample mean IC of these candidates: 0.0014
Still t>2 out-of-sample: 3; true factor in-sample t = 7.4, out-of-sample t = 5.7
Using t>3 threshold: 7 passed, including 1 true factor

2.2 Interpretation

Combining two defenses (increased threshold + out-of-sample safe) can suppress false discoveries to a very low level. However, note that even in the out-of-sample " $t > 2$ " test, about 2% of noise might pass by chance, so 2 out of those 3 are still fake.

Layer 3: Defense Design

3.1 Out-of-Sample Safe

If the Agent can see the results from the safe and then go back to modify the factor, the safe becomes in-sample. This must be guaranteed by system permissions rather than Agent self-discipline (see Security Boundaries of Agent Trading Systems, Agent Permission Design).

3.2 Increasing Thresholds Based on Attempt Count

• The system automatically records every backtest call made by the Agent, including those that were abandoned;
• Based on the total number of attempts, thresholds are dynamically calculated using Bonferroni, BHY, or the Deflated Sharpe Ratio (see Multiple Testing and the Factor Zoo);
• The Agent's report must include a line stating "Total attempts in this round: N," and this number must be filled in by the system, not self-reported by the Agent.

3.3 Economic Logic Review

Factors mined purely from data without a reasonable explanation are more likely to be noise. Every candidate factor is required to answer:

The third question is particularly powerful: a good explanation allows for falsifiable predictions. This is consistent with the "negative control" approach used in causal analysis to reduce financial hallucinations.

3.4 Other Defenses

3.5 The True Value of an Agent

Under these defenses, Agent factor mining remains highly valuable, but the value lies not in "finding more significant factors," but in:

• Quickly turning papers and ideas into executable code;
• Systematically exploring data and combinations that humans might overlook;
• Automatically completing tedious testing processes;
• Generating falsifiable hypotheses to be subjected to more rigorous testing.

Summary