# What Is a Permutation Test? Definition and Example

> A simple way to ask “could luck explain this?” about a strategy or an analyst’s record, with a worked example.

Source: https://www.exent.ai/learn/permutation-test · Reviewed by David, Senior Analyst, Exent · Published Nov 6, 2026 · Updated Nov 6, 2026

## Key takeaways

- A permutation test checks whether a result could be luck by shuffling the data many times and counting how often the shuffled versions do as well as the real one.
- The share of shuffles that match or beat the real result is the p-value.
- Base rates matter: in Exent’s 7-year study of 820,838 analyst calls with a price target (as of Nov 16, 2026), 53.0% of targets were hit, so a record a few points above that is not remarkable on its own.

## What is a permutation test?

It is a statistical test that needs no assumptions about bell curves. You break the link you care about, for example between a signal and the returns that followed, by shuffling one side at random. If the real result rarely appears in the shuffles, it is unlikely to be chance. It sits next to out-of-sample testing as a guard against fooling yourself.

## How does it work?

1. Compute the statistic on the real data, such as hit rate or average return after a signal.
2. Shuffle which outcome goes with which signal, and recompute. Repeat thousands of times.
3. **p-value = shuffles at least as good as the real result ÷ total shuffles**

## Worked example

| Illustrative analyst test | Value |
| --- | --- |
| Calls made | 60 |
| Calls right | 38 (63%) |
| Shuffles run | 10,000 |
| Shuffles with 38+ right | 420 |
| p-value | 0.042 |

Illustrative only. A p-value of 0.042 says random assignment matched this record about 4 times in 100. Now the catch: among a thousand analysts with no skill, about 50 would pass a 5% test by chance alone. That is why [analyst success rates](https://www.exent.ai/learn/analyst-success-rate) need long records and comparison with the base rate.

## Why it matters for a self‑directed investor

Track records are everywhere: funds, newsletters, analysts, strategies. A permutation test is the quickest way to ask whether one is better than a coin. See skill vs luck in returns and [how accurate analysts are](https://www.exent.ai/resources/analyst-accuracy).

## Common mistakes

- Shuffling in a way that destroys real structure, such as market-wide moves that hit all stocks at once.
- Testing hundreds of ideas and celebrating the few with low p-values.
- Reading a low p-value as proof of a large or lasting edge.
- Running too few shuffles to estimate small p-values.

## Check analysts’ track records with Exent

1. Stocks Radar ranks thousands of Wall Street analysts by past accuracy, with each analyst’s hit rate and number of calls.
2. Gems flag stocks where several of the best‑rated analysts agree.

Exent is our product: [Exent Gems](https://www.exent.ai/radar)

## Frequently asked questions

### What does a permutation test tell you in investing?

It estimates how often a result as good as the observed one would appear if the signal or the person had no real skill.

### How many shuffles do you need?

Thousands are typical. More shuffles give a more precise p-value, which matters when the p-value is small.

### Is a permutation test the same as a backtest?

No. A backtest measures how a rule would have performed. A permutation test asks whether that performance stands out from what chance would produce.

---

Educational content, not investment advice. Past performance does not guarantee future results.
