Reducing Agent "Financial Hallucinations" via Causal Analysis

Layer 1: The Intuition

1.1 A Very Persuasive Error

An Agent analyzes three years of data and reaches the following conclusion:

"The news sentiment index is significantly positively correlated with the next day's returns (t = 13). Improving sentiment drives investors to buy, thereby pushing up stock prices. It is recommended to construct a sentiment factor."

It sounds airtight. However, the actual situation might be: overall market risk appetite is simultaneously driving both news sentiment (journalists become more optimistic) and stock prices; sentiment itself has no effect on stock prices. Once the driving mechanism of risk appetite changes, this "factor" will fail.

1.2 Two Types of Hallucinations

Large models learn the "because... therefore..." narrative style from vast amounts of financial text, and financial media itself is full of hindsight attribution. An Agent's causal hallucination is largely inherited.

1.3 Three Lines of Defense

Layer 2: Three Tools

2.1 Causal Graph (DAG)

Require the Agent to first draw the possible causal relationships between relevant variables before providing a conclusion:

Once drawn, paths other than "Sentiment → Returns" become obvious. Pearl's causal inference theory provides rules for determining "which variables need to be controlled" (the Backdoor Criterion).

2.2 Controlling for Confounding Variables

Include confounding variables (or their proxies) in a regression to see if the original effect remains:

• Effect decreases significantly or disappears → The original correlation was primarily from the confounder;
• Effect remains largely unchanged → At least this confounder has been ruled out (but others may exist).

2.3 Placebo Tests and Negative Controls

Key distinction: A permutation test can only prove "it is not a coincidence"; it cannot prove "it is causal." A correlation caused by a confounder will still appear "significant" under a permutation test.

Layer 3: Code Demonstration

3.1 Simulation: Returns are driven only by risk appetite; sentiment has no effect

import numpy as np, pandas as pd, statsmodels.api as sm
rng = np.random.default_rng(21)
T = 1000
# Confounder: Market risk appetite (unobservable, but can be proxied)
risk_on = rng.normal(size=T)
sentiment = 0.8 * risk_on + rng.normal(scale=0.6, size=T)     # "News sentiment factor" discovered by Agent
ret = 0.5 * risk_on + rng.normal(scale=1.0, size=T)            # Returns are actually driven only by risk appetite
df = pd.DataFrame({[["sentiment"]]: sentiment, "ret": ret,
                   "risk_proxy": risk_on + rng.normal(scale=0.3, size=T)})

naive = sm.OLS(df["ret"], sm.add_constant(df"sentiment")).fit()
print("Naive regression: sentiment coefficient", round(naive.params["sentiment"], 3), " t =", round(naive.tvalues["sentiment"], 1))
ctrl = sm.OLS(df["ret"], sm.add_constant(df[["sentiment", "risk_proxy"]])).fit()
print("After controlling for confounders: sentiment coefficient", round(ctrl.params["sentiment"], 3), " t =", round(ctrl.tvalues["sentiment"], 1))

null = []
for _ in range(500):
    s = rng.permutation(df["sentiment"].values)
    null.append(np.corrcoef(s, df["ret"])[0, 1])
obs = df["sentiment"].corr(df["ret"])
print("Observed correlation", round(obs, 3), " Placebo 95th percentile", round(np.quantile(np.abs(null), 0.95), 3))

future_only = 0.8 * risk_on + rng.normal(scale=0.6, size=T)   # Shares the same source as sentiment but cannot be a cause in time
print("Correlation between negative control and returns", round(np.corrcoef(future_only, ret)[0, 1], 3))

Output:

Naive regression: sentiment coefficient 0.45  t = 13.0
After controlling for confounders: sentiment coefficient 0.099  t = 1.8
Observed correlation 0.381  Placebo 95th percentile 0.06
Correlation of negative control variable with returns 0.384

3.2 Interpretation

The coefficient after control did not drop to zero (0.10) because the proxy variable risk_proxy carries measurement error and failed to fully absorb the confounding. This is the norm in reality: controlling for confounders can only be done "as best as possible."

3.3 Turning it into Agent Workflow Standards

Add the following requirements to the system prompts and tools for research Agents:

1. Conclusions must include a causal graph: List at least three possible confounding variables;
2. Mandatory tests: Control for major confounders (market, industry, market cap, risk appetite proxies), permutation tests, and at least one negative control;
3. Conclusion grading:

4. Prohibited phrasing: Do not use "leads to," "drives," or "because" when only correlation evidence is available;
5. Adversarial Agent: Use another Agent specifically to propose alternative explanations.

3.4 Limitations

• Unobserved confounders can never be completely ruled out;
• Financial markets are reflexive: once a factor is widely used, it changes the market itself;
• The strongest causal evidence comes from intervention (see "Interventional Training"), and in financial markets, true randomized experiments are almost impossible. What can be utilized are "natural experiments": policy changes, index rebalancing, changes in trading rules, etc.

Summary