filing № 002 · research memo
Confessions of a Curve Fitter
Two years ago I owned a strategy with an in-sample Sharpe ratio of 4.6. It traded momentum, yes, but only after the third Tuesday of any month containing an r, provided the previous session closed within 1.2 percent of its twenty-day average and the moon was doing something in particular. On the historical data it was flawless. The equity curve rose like a staircase to a rooftop bar. I have never been surer of anything, and I have never been wronger, and the second fact took fourteen months and a brokerage statement to fully load.
This memo is the post-mortem, filed so the desk never forgets how overfitting feels from the inside. Because here is the thing nobody warns you about: it does not feel like lying. Every one of those parameters had a story. The third Tuesday was positioning flow. The moon was volatility regimes. A story is not evidence — a story is how the noise talks you out of your stop.
| parameters | sharpe, in-sample | sharpe, out-of-sample | my confidence |
|---|---|---|---|
| 1 | 0.9 | 0.7 | cautious |
| 2 | 1.4 | 0.5 | interested |
| 4 | 2.3 | 0.1 | sleeping well |
| 6 | 3.1 | −0.4 | telling friends |
| 9 | 4.6 | −1.2 | quoting Nietzsche |
| n = 1 strategy, 1 market, 1 guy who wanted it to work | |||
Rules I purchased at full price
The desk now runs four rules, each one paid for in drawdown:
Pay for parameters. Every knob you add must be bought with data the model has never seen. If the edge needs the fourth decimal place of a stop-loss to exist, it does not exist; it is renting that decimal from luck.
Suspicion scales with beauty. The better the backtest looks, the more you should assume the builder — usually you — is the marginal buyer of their own story. Real edges are ugly and modest and arrive with paperwork.
Count degrees of freedom, then count again. The parameters you can see are the small ones. The silent ones — the dataset you chose, the period you started from, the strategies you never wrote up — fit curves too.
Walk-forward or walk away. Roll the test window forward through time and let the model re-earn itself, over and over, with no memory of the future. It is penance. It is supposed to be.
The three that survived
The ticker on this site keeps a running count: 601 curves fit, 3 unfit. The three that survived are not the clever ones. One is a slow, boring trend system I have no great story for. One is do less. One is this website: no build step, no dependencies, nothing to overfit. The beautiful strategy was a museum piece — perfect, unpublishable, and dead the moment it touched a live market.
The past is the most compliant dataset you will ever meet. It agrees with everything, which is how you know it is lying.
If you meet a flawless backtest on the road, and you will, because you are building one right now — ask it one question: how many parameters did it cost, and who paid? If the answer is nine and future me, close the position. I have the statement somewhere. It is filed under tuition.
backtest of this memo