Overfitting: The Risk You Cannot See in the Backtest

The most dangerous result in quantitative trading is a beautiful backtest on a strategy with no real edge. It looks right. The equity curve is smooth. The Sharpe ratio is impressive. And when you trade it live, it fails immediately and systematically. This is overfitting — and it happens far more often than most people acknowledge.

What overfitting actually is

A model is overfit when it has learned the noise in historical data rather than the signal. With enough parameters, you can fit any historical dataset perfectly — but that fit is meaningless. You have memorised the past, not discovered anything about the future. In strategy development this typically happens through:

• Excessive parameter optimisation — testing hundreds of combinations and selecting the best performer.
• Look-ahead bias — rules that implicitly use information unavailable at decision time.
• Data snooping — testing many strategies on the same data, then presenting the winner as if it were the only test.

The out-of-sample test

The most important defence is the out-of-sample test. Develop and optimise on one portion of history (in-sample), then test — without modification — on a separate, untouched portion (out-of-sample). A strategy that performs well in-sample but fails out-of-sample has been overfit. The degradation between the two measures how much of the in-sample performance was noise.

Walk-forward analysis

Walk-forward analysis is a more rigorous version: the strategy is optimised over a rolling window, evaluated on the next period, then re-optimised and re-evaluated across the full history. The result is a realistic simulation of how the optimisation process would have performed in real time.

Simplicity as a bias

One of the most effective defences is simply preferring simpler strategies. Two parameters with a clear logical rationale will generalise far better than ten parameters tuned to a specific period. If the logic behind a strategy cannot be explained in plain language, the parameters are probably curve-fitted.

Honest evaluation

Pulsar’s analytics are designed to make honesty the default — showing drawdown alongside return and surfacing the statistical context that polished results presentations tend to omit.

This content is educational only — not investment advice.