Factor Engineering: From Raw Data to Usable Signals

Layer 1: The Intuition

1.1 Washing, Cutting, and Deboning

Ingredients cannot be thrown directly into the pot after being bought: you must wash away the sand (missing and erroneous data), cut off the bad parts (extreme values), cut them into uniform sizes (standardization), and remove unwanted flavors (neutralization).

Factor engineering is the preparation of ingredients before cooking. If the preparation is poorly done, no matter how exquisite the subsequent backtest is, it will be useless.

1.2 A Worked Example

Idea: "Stocks with low P/E ratios (cheap) will have higher future returns."

Sorting directly by P/E ratio will encounter these problems:

Layer 2: Every Step of the Pipeline

2.1 Data Acquisition: Point-in-Time Data

The most hidden and fatal error is look-ahead bias: using data on a specific day in a backtest that was impossible to know on that day.

2.2 Winsorization

"Pulling" values that exceed a reasonable range back to the boundaries, rather than deleting them. Common boundaries:

The coefficient 1.4826 makes MAD consistent with the standard deviation under a normal distribution.

2.3 Missing Values

• If possible, do not fill them (simply exclude that stock for the current period);
• If they must be filled, use the industry median, not the market-wide median;
• Do not use future data to fill (e.g., forward-filling using the next period's value).

2.4 Standardization

Perform a z-score on the cross-section for each period (each trading day or month):

$$z_i = \frac{x_i - \bar{x}}{\sigma_x}.$$

This allows comparison across different periods and different factors, and facilitates the synthesis of multiple factors. One can also use rank percentiles instead, which is less sensitive to the shape of the distribution.

2.5 Neutralization

Use regression to remove unwanted exposures. Perform a cross-sectional regression for each period:

$$x_i = \sum_k \gamma_k \, D_{ik} + \delta \ln(\text{market\_cap}_i) + \varepsilon_i,$$

$D_{ik}$ are industry dummy variables (where stock $i$ belongs to industry $k$, the value is 1). The residual $\varepsilon_i$ is the neutralized factor: it is uncorrelated with industry and market cap; what remains is "among stocks of the same industry and same size, who is cheaper."

Layer 3: Code Implementation

3.1 Engineering details: A Complete Pipeline

In the simulated data below, the raw factor intentionally mixes in industry and market cap effects, and contains heavy-tailed distributions and missing values:

import numpy as np, pandas as pd
rng = np.random.default_rng(0)
N = 500
df = pd.DataFrame({
    "industry": rng.choice(["Tech", "Finance", "Consumer", "Energy"], N),
    "mcap": np.exp(rng.normal(23, 1.2, N)),
})
ind_bias = df["industry"].map({"Tech": 0.8, "Finance": -0.5, "Consumer": 0.1, "Energy": -0.3})
df["raw"] = ind_bias + 0.4 * np.log(df["mcap"]) + rng.standard_t(3, N)  # Raw factor: mixed with industry and market cap
df.loc[rng.choice(N, 15, replace=False), "raw"] = np.nan

def winsorize_mad(x, k=3):
    med = x.median(); mad = (x - med).abs().median() * 1.4826
    return x.clip(med - k * mad, med + k * mad)

def zscore(x):
    return (x - x.mean()) / x.std()

def neutralize(y, df):
    X = pd.get_dummies(df["industry"], dtype=float)
    X["lnmcap"] = np.log(df["mcap"])
    m = y.notna()
    beta, *_ = np.linalg.lstsq(X[m].values, y[m].values, rcond=None)
    out = y.copy(); out[m] = y[m] - X[m].values @ beta
    return out

x = df["raw"]
x = x.fillna(x.groupby(df["industry"]).transform("median"))   # Fill with industry median
x = winsorize_mad(x)
x = zscore(x)
x = neutralize(x, df)
x = zscore(x)
df["factor"] = x
print("Correlation between raw factor and ln market cap:", round(df["raw"].corr(np.log(df["mcap"])), 3))
print("Correlation between processed factor and ln market cap:", round(df["factor"].corr(np.log(df["mcap"])), 3))
print("Processed mean by industry:", df.groupby("industry")["factor"].mean().round(3).to_dict())
print("Raw kurtosis:", round(df["raw"].kurt(), 2), "Processed kurtosis:", round(df["factor"].kurt(), 2))

Output:

Correlation between original factor and ln market cap: 0.266
Correlation between processed factor and ln market cap: -0.0
Processed industry means: {'Consumer': -0.0, 'Tech': -0.0, 'Energy': 0.0, 'Finance': 0.0}
Original kurtosis: 4.07 Processed kurtosis: 1.28

(-0.0 is the display of a floating-point number rounded, representing a value extremely close to zero.)

3.2 Interpretation

3.3 On Real Panel Data

Real data is a panel of "Date × Stock." Every step must be performed grouped by date; you must never calculate the mean and standard deviation by mixing all dates together (as that would use future information):

def process_one_date(g):
    x = g["raw"]
    x = x.fillna(x.groupby(g["industry"]).transform("median"))
    x = zscore(winsorize_mad(x))
    return zscore(neutralize(x, g))

panel["factor"] = panel.groupby("date", group_keys=False).apply(process_one_date)

3.4 Common Error Checklist

Summary