the failure
Overfitting is what happens when a rule describes the history it was chosen on rather than the market underneath it. It is easy to describe and hard to feel, so here it is as a measurement on 144 markets.
the experiment
For each of 144 markets we cut the history at 70%. On that first part we picked the single best performing rule, which is the only evidence anyone choosing a strategy would have had at the time. Then, without changing anything, we read what that same rule did on the remaining 30%, which it had never seen.
On the half it was chosen on, the median winner returned +86%. On the half that followed, the median was -12%. It made money at all in 32% of cases.
That is the whole lesson of this site in two numbers, and it is why every market page here prints all 15 rules rather than the one that worked.
| measured on | median result |
|---|---|
| the first 70%, where it was chosen | +86% |
| the last 30%, which it had never seen | -12% |
| simply holding, same last 30% | -47% |
| how often the winner then made money | 32% |
why it happens
Any price history contains accidents. A rule with enough freedom will fit some of them, and the fit is indistinguishable from skill while you are looking at the same data the rule was fitted to. Trying more variations does not reduce this. It guarantees it, because you are selecting for whichever variation happened to match the accidents best.
Note which rules most often win the first half here: rsi(2) mean reversion on 29 markets, bollinger(20,2) mean reversion on 16 markets, 10/50 moving average on 15 markets. A rule can top that list and still be a poor bet, because topping it is partly a measure of how well a rule fits noise.
the defence
Decide the rule and its settings before you see the test. Hold back part of the history and do not touch it. When you do look, accept what it says rather than going back to adjust and retest, because a held-out set you have examined twice is no longer held out.
None of the 15 rules we publish has a tuned parameter. 50 and 200 days, 12 month momentum, RSI below 30, Bollinger bands at two standard deviations: every setting is the textbook one, chosen before any result was seen. That is a weaker-looking set of results and a more honest one. You can read any of them on the market pages.
questions
A strategy that fits the particular history it was selected on rather than any durable behaviour of the market, so it performs well on that data and poorly afterwards.
Choose the rule and its settings before testing, hold back part of the history, judge on that held-back part, and do not go back to adjust and retest against it. Publishing the rules that failed also helps, because it shows how many were tried.
Judging a strategy on data that was not used to create or select it. We split every market at 70% and report both halves separately.
Rarely, on our data. Across 144 markets, the best rule on the first 70% went on to make money in the remaining 30% only 32% of the time, with a median of -12% against +86% on the half it was picked from.
the evidence
144 markets, 15 rules, 2,160 tests, every one published with both halves and the dates. Read them at the market pages. Figures on this page were computed 2026-11-17.
more