What Is Backtest Overfitting? Definition and Example
Backtest overfitting is when a strategy is tuned so closely to past data that it captures noise instead of a real pattern. Here is how to spot it, with a real example.
A simple way to ask “could luck explain this?” about a strategy or an analyst’s record, with a worked example.
Which stocks do Wall Street’s best‑ranked analysts agree on? See them in Exent Gems →
It is a statistical test that needs no assumptions about bell curves. You break the link you care about, for example between a signal and the returns that followed, by shuffling one side at random. If the real result rarely appears in the shuffles, it is unlikely to be chance. It sits next to out-of-sample testing as a guard against fooling yourself.
Put this to work: see which stocks Wall Street’s best‑ranked analysts agree on in Exent Gems.
| Illustrative analyst test | Value |
|---|---|
| Calls made | 60 |
| Calls right | 38 (63%) |
| Shuffles run | 10,000 |
| Shuffles with 38+ right | 420 |
| p-value | 0.042 |
Illustrative only. A p-value of 0.042 says random assignment matched this record about 4 times in 100. Now the catch: among a thousand analysts with no skill, about 50 would pass a 5% test by chance alone. That is why analyst success rates need long records and comparison with the base rate.
Track records are everywhere: funds, newsletters, analysts, strategies. A permutation test is the quickest way to ask whether one is better than a coin. See skill vs luck in returns and how accurate analysts are.
It estimates how often a result as good as the observed one would appear if the signal or the person had no real skill.
Thousands are typical. More shuffles give a more precise p-value, which matters when the p-value is small.
No. A backtest measures how a rule would have performed. A permutation test asks whether that performance stands out from what chance would produce.
Exent tracks 4,000+ analysts, keeps only the elite few with proven records, and surfaces the Gems where they agree.
Educational content, not investment advice. Examples are illustrative; past performance does not guarantee future results.