Where a p-value comes from The permutation test, the null distribution and the distribution of the p-value itself

@misc{korkmaz2026where,
  author       = {Korkmaz, Sel{\c{c}}uk},
  title        = {Where a p-value comes from: the permutation test, the null distribution and the distribution of the p-value itself},
  year         = {2026},
  month        = oct,
  date         = {2026-10-03},
  howpublished = {\url{https://selcukorkmaz.github.io/blog/permutation-test/}},
  url          = {https://selcukorkmaz.github.io/blog/permutation-test/},
  note         = {Blog post}
}

1A p-value needs a null distribution

In an earlier note I defined a p-value as the probability of getting a test statistic at least as extreme as the one observed, computed on the assumption that the null hypothesis and every other assumption of the statistical model are true ^[1]. That sentence hides a step. A probability needs a distribution: the null distribution, which says how the test statistic would vary from study to study if the null hypothesis were true. The p-value is the area in its tail beyond the observed value.

Take a small pilot trial. Twenty adults with raised blood pressure are randomised, 10 to a lifestyle programme and 10 to usual care, and the outcome is the fall in systolic blood pressure over 12 weeks (Table 1). The programme arm fell by 9.4 mmHg on average and the usual-care arm by 1.9 mmHg, a difference of 7.5 mmHg.

Student's two-sample t-test gives t = 2.25 on 18 degrees of freedom and p = 0.037, with a 95% confidence interval for the difference of 0.5 to 14.5 mmHg. The 0.037 is the area of the t distribution with 18 degrees of freedom beyond ±2.25. That distribution is the null distribution of the t statistic when the outcomes in both arms are independent draws from normal distributions with the same mean and the same standard deviation (SD). Those are assumptions about how the data arose, and the formula takes them on trust.

A permutation test reaches a null distribution by a route that needs neither the normal distribution nor a formula.

2Shuffling the labels

Suppose the programme had no effect on anybody. Then each participant's fall in blood pressure would have been the same whichever arm the randomisation had put them in. The 20 numbers in Table 1 would be fixed, and the only thing chance decided was which 10 of them carried the label "programme". The randomisation could have picked any 10 of the 20; there are 184,756 ways to do that, all equally likely. For each one we can work out the difference in means we would have seen. Those 184,756 differences are the null distribution of the difference, and no theory was needed to get them.

This null hypothesis, that the programme changed nobody's outcome, is the one a randomisation test examines directly ^[2], and it is stronger than "the two arms have the same mean". Fisher set out the idea in 1935 ^[3]. The method goes by two names: a randomisation test when, as here, the shuffling mirrors a random allocation that really took place, and a permutation test when the groups are treated as samples from populations, though the two names are often used interchangeably ^[2]. I use "permutation test" for both, as most software does, and "shuffle" for one rearrangement of the labels.

Figure 1 lets you do it. Each shuffle is one alternative history of the trial in which the programme did nothing. Most give a difference within a few mmHg of zero, and the histogram they build is the null distribution. The real difference of 7.5 mmHg sits in its right tail: exactly 7,570 of the 184,756 possible allocations give a difference at least 7.5 mmHg from zero, in either direction, so

That is the most literal reading a p-value can have. If the programme did nothing, 4.1% of the ways the randomisation could have split these 20 people would have produced a gap at least as large as the one observed. It is not the probability that the programme did nothing, nor the probability that the result was produced by chance alone ^[4]; it is computed by assuming that the programme did nothing.

Because the two arms are the same size, the shuffle distribution is symmetric about zero: swapping the two arms of any allocation turns its difference into minus that difference. Counting allocations with |T*| ≥ |T_obs| then gives exactly twice the one-sided count. With arms of different sizes the distribution need not be symmetric, and the absolute-value rule and doubling the one-sided p can disagree, so say which you used.

3Every allocation, or a random sample of them?

Listing every allocation gives the exact permutation p-value. That is feasible here because 184,756 is a small number for a computer. With 30 participants per arm there are about 1.2 × 10¹⁷ allocations, and with the 65 per arm of the earlier note about 9.5 × 10³⁷, so in practice we draw B allocations at random and estimate p from them. This is a Monte Carlo permutation test, and two details matter.

Count the real allocation. Use p = (b + 1)/(B + 1), where b is the number of random shuffles at least as extreme as the real data. The real allocation is one of those the randomisation could have produced, and with the +1 the test rejects a true null hypothesis no more often than its stated level. The plain share b/B can be zero, and it is smaller than (b + 1)/(B + 1) whenever b < B, so a test built on it can reject a true null hypothesis more often than stated ^[5]. A permutation p-value can never really be zero ^[5], and the smallest value B shuffles can give is 1/(B + 1): 999 shuffles cannot give less than 0.001.

Use enough shuffles. A Monte Carlo p-value has its own sampling error, with a standard error of about √[p(1 − p)/B] (Table 2). With 999 shuffles, a test whose exact p is 0.041 will usually report something between 0.030 and 0.055, and about one run in eleven will report a value above 0.05. With 9,999 shuffles the range shrinks to 0.037 to 0.045. Choose B so that this error is small next to the precision you report. A sensible default for a single test is 9,999 shuffles, and far more are needed when p-values will be compared with a very small threshold, as after a correction for multiple testing.

4The t distribution is a shortcut

The shuffle distribution in Figure 1 looks like a bell curve, and the comparison with the t-test is as old as the method. In The Design of Experiments Fisher applied the method to Darwin's data on 15 pairs of maize plants: he counted all 32,768 ways of flipping the signs of the within-pair differences, found that 5.3% of them were at least as extreme as the observed total, and noted that this was very nearly the result of Student's t-test ^[3]. A year later he described an imagined two-group comparison as a shuffle of cards: write the statures of a sample of 100 Englishmen and 100 Frenchmen on cards, shuffle them without regard to nationality, and deal them into two new groups of 100. The statistician's conclusions, he wrote, have no justification beyond their agreement with what this elementary method would give ^[6]. Pitman showed in 1937 how to build such tests with no assumption about the form of the populations sampled ^[7]. Figure 2 overlays the shuffle distribution and the t distribution. The bars are the exact shuffle distribution of the difference in means. The curve is the null distribution that the t-test uses, drawn on the same scale: a t distribution with 18 degrees of freedom, stretched by the standard error of the difference (3.33 mmHg), so that its area beyond ±7.5 mmHg is the t-test's p-value.

For the pilot data the two agree well: p = 0.041 from the shuffles and 0.037 from the t distribution. The confidence intervals agree too, at 0.5 to 14.5 mmHg either way (Section 7). When the data look like these, the t-test is a quick and accurate approximation to the exact answer.

They part company when the data do not fit the bell. Switch Figure 2 to the second data set, in which the largest fall in the programme arm is 45 mmHg instead of 21, say because one participant also started a blood pressure drug. The difference in means rises to 9.9 mmHg, but the SD of the programme arm rises from 7.7 to 13.4 mmHg, and the t-test gives p = 0.054 (Welch's version, 0.059). The shuffle distribution, by contrast, changes shape. The 45 lands in one arm or the other in every allocation and pushes the difference about 4 mmHg in that direction, so the distribution is wider than before but flat-topped, with short tails. Its exact p-value is 0.039. The t curve, a bell with longer tails, no longer describes what the randomisation could have produced.

That is not an argument that shuffling rescues the mean. With the 45 in the data, the difference in means still depends heavily on one person, and a statistic that resists outliers may answer the question better. Switch the statistic to the difference in medians. The shuffles build its null distribution just as easily, and give p = 0.067 for both data sets, because the median ignores how far the largest value lies from the rest. Like any shuffle test, though, it examines whether the two groups are exchangeable: when their distributions differ in shape, its error rate as a test of equal medians can be far from the stated level, even with equal group sizes ^[8]. A permutation test lets you choose the statistic, so choose it before seeing the data: trying several and reporting the one with the smallest p-value makes that p-value misleading ^[1,4].

5What a shuffle assumes

Shuffling needs no normal distribution, but it is not assumption-free. Its essential assumption is exchangeability: if the null hypothesis is true, swapping the labels does not change the probability of the data ^[9]. Where exchangeability comes from depends on how the data arose ^[2].

• In a randomised trial, the randomisation supplies it. Every allocation the randomisation could have made was equally likely, and under the null hypothesis that the treatment changed nobody's outcome the outcomes would have been the same under each, so the chance of a false positive is at most the stated level whatever the distribution of the outcome, with no assumption that the participants are a random sample from a population ^[2]. That fits most biomedical experiments: in a survey of 252 comparative studies, Ludbrook and Dudley found groups formed by randomisation in 96% and by random sampling in only 4% ^[10]. The conclusion then concerns the participants in the study ^[2]; extending it to others is a judgement about how they were recruited. The guarantee is for the null hypothesis that nobody's outcome changed. If the programme could change the spread of outcomes without changing their average, a small p-value from the raw difference may reflect the spread when the arms differ in size. To test the average effect, shuffle a studentised statistic, described below, which does so approximately in large samples ^[11]; with equal arms, as in the pilot, it gives the same test as the raw difference.
• In an observational comparison it is an assumption. Nobody assigned the labels when men are compared with women or smokers with non-smokers. Exchangeability then rests on the null hypothesis that both groups were sampled from the same distribution: the same shape and the same spread, not only the same mean ^[2].

The second point has a practical consequence. Suppose two groups have the same mean but different SDs. The null hypothesis of identical distributions is then false, the labels are not exchangeable, and a shuffle test of the difference in means can reject far more or far less often than its nominal 5%, just as Student's pooled t-test does (Table 3). Shuffling a studentised statistic, one that divides the difference by its own standard error, as Welch's t does, keeps the test exact when the groups are exchangeable and brings its error rate close to the nominal level in large samples when they are not ^[12,13]. With equal group sizes, even the raw difference behaves well in large samples ^[8,12].

When the smaller group is the more variable one, both Student's t-test and the shuffle test of the raw difference reject a true null hypothesis about one time in five; when the larger group is the more variable one, they almost never reject. Shuffling Welch's t brings the rates to 6.6% and 4.0%, close to but not at 5%, because with groups of 8 and 24 its protection is only approximate, and Welch's t-test with its own t distribution does slightly better here. Shuffling does not mend a statistic that ignores unequal spread. With equal group sizes, as in the last row and in the pilot, shuffling Welch's t gives exactly the same test as shuffling the raw difference: with equal group sizes Welch's t equals Student's t, and when only the labels change, Student's t rises steadily with the size of the difference.

The design has a consequence too: shuffle only what was randomised, in the way it was randomised (Table 4). The set of allocations the test compares with should be the set the randomisation could actually have produced, each with the probability the randomisation gave it ^[9,14].

6The p-value has a distribution too

A null distribution describes the test statistic across imagined repetitions of a study. The p-value is computed from the statistic, so it also varies from one repetition to the next, and its distribution is the clearest way to see what a single p-value can tell you ^[15,16]. Figure 3 runs the pilot trial 1,000 times on computer-generated data, with outcomes normally distributed with an SD of 8 mmHg, analyses every run with a permutation test of 999 shuffles, and plots the 1,000 p-values.

• No effect. The p-values are spread evenly between 0 and 1, and each 0.05-wide bin holds about 5% of the runs. That is what a valid test does: when the null hypothesis and the model are true and the test statistic is continuous, p is equally likely to fall anywhere between 0 and 1, so p ≤ 0.05 happens in about 5% of runs (4.3% in this simulation), which is all a 5% significance level promises ^[15,17]. For a Monte Carlo permutation test with continuous data, the real data's rank among the B + 1 values is then equally likely to be any of 1, …, B + 1, apart from the small chance that a random shuffle reproduces the real allocation or its mirror image (the two arms swapped), so (b + 1)/(B + 1) takes each of the values 1/(B + 1), 2/(B + 1), …, 1 with almost exactly equal probability.
• A real effect. The p-values pile up near zero ^[15], and the share at or below 0.05 is, by definition, the power of the test for that effect ^[17]. With a true difference of 8 mmHg, one SD, and 10 participants per arm, the power is only about 56% (the 1,000 runs in Figure 3 give 53.7%, within simulation error): the same trial, repeated with the same true effect, would give p above 0.05 nearly half the time. In the long run the median p is about 0.04, close to the pilot's, but the middle 80% of p-values run from about 0.002 to about 0.35 (the 1,000 runs in Figure 3 give 0.002 to 0.38).

The pilot's p = 0.041 is one draw from a distribution like the right-hand one. A repeat of the trial could easily give 0.3 or 0.002 ^[16]. So a single p-value says little about whether a result will replicate, and the estimate and its confidence interval, which show the size of the effect and the precision of the study, carry more of what a reader needs.

7Intervals from shuffles, and the bootstrap

A permutation test answers a question about one hypothesis, but like any test it can be turned into a confidence interval. Assume a shift model, in which the programme lowers everyone's blood pressure by the same amount δ. For a candidate δ, subtract it from each programme value and run the shuffle test of "no effect" on the shifted data. The 95% interval is the set of δ values whose p-value is above 0.05 ^[2]: the stretch where the p-value function of the earlier note, built here from shuffles instead of from the t distribution, lies above 0.05. For the pilot data it runs from 0.5 to 14.5 mmHg, the same as the t interval to one decimal place. For the data with the 45 mmHg value it runs from 0.6 to 20.0 mmHg, where the t interval is −0.2 to 20.0.

The bootstrap is the other resampling method in common use, and the two are easy to confuse. They answer different questions (Table 5).

8How to report a permutation test

A permutation p-value is still a p-value, and the reporting advice of the earlier note applies to it, starting with the estimate and its interval. The methods section needs enough extra detail for someone else to get the same number:

• The statistic, chosen in advance: difference in means, difference in medians, Welch's t, or another.
• What was shuffled, and within what: strata, blocks, pairs or clusters, matching the randomisation.
• All allocations or random shuffles. For random shuffles (a Monte Carlo test), the number of shuffles B, the formula (b + 1)/(B + 1), the software and the random seed.
• The sidedness, and for groups of different sizes, how the two-sided p was defined.
• The estimate and its interval, with the method for the interval: a t interval, a bootstrap interval, or one from inverting the permutation test.

8.1Sentences to adapt

• Methods, exact test. "The difference in mean fall in systolic blood pressure (programme minus usual care) was tested with a two-sided exact permutation test, using all 184,756 ways of allocating the 20 participants to two groups of 10; the p-value was the proportion of allocations whose difference was at least as far from zero as the observed one."
• Methods, Monte Carlo test. "The difference in means was tested with a two-sided permutation test, using 9,999 random reallocations of the treatment labels (R 4.5.2, seed 2026), made within randomisation strata if the trial was stratified. The p-value was computed as (b + 1)/(B + 1), where b is the number of reallocations giving a difference at least as far from zero as the observed one and B = 9,999."
• Results. "Systolic blood pressure fell 7.5 mmHg more in the programme group than in the usual-care group (95% CI 0.5 to 14.5, from inverting the permutation test; exact permutation p = 0.041)."

Base R does both versions in a few lines. The small tolerance in the comparisons keeps allocations whose difference equals the observed one, which rounding error could otherwise drop.

prog  <- c(21, 17, 15, 14, 13, 7, 6, 2, 2, -3)
usual <- c(11, 10, 9, 5, 2, 1, 0, -1, -9, -9)
x <- c(prog, usual); n1 <- length(prog); n2 <- length(usual)
obs <- mean(prog) - mean(usual)                       # 7.5

# Exact: every choice of the 10 participants labelled "programme"
idx  <- combn(n1 + n2, n1)                            # 184,756 allocations
sA   <- colSums(matrix(x[idx], nrow = n1))
diff <- sA / n1 - (sum(x) - sA) / n2
mean(abs(diff) >= abs(obs) - 1e-8)                    # 0.04097 (7,570 of 184,756)

# Monte Carlo: B random shuffles, counting the real allocation
set.seed(2026)
B   <- 9999
sim <- replicate(B, { s <- sample(x); mean(s[1:n1]) - mean(s[-(1:n1)]) })
(sum(abs(sim) >= abs(obs) - 1e-8) + 1) / (B + 1)      # 0.0421

t.test(prog, usual, var.equal = TRUE)                 # t = 2.253, df = 18, p = 0.03697

9Five traps

9.1Reporting p = 0

If none of 999 shuffles is as extreme as the real data, the p-value is (0 + 1)/(999 + 1) = 0.001, the smallest that 999 shuffles can give, not zero ^[5]. Report it as p = 0.001 with the number of shuffles, or use more shuffles.

9.2Too few shuffles for the question

The Monte Carlo error in Table 2 matters most near a threshold and after a multiplicity correction. With 1,000 tests and a Bonferroni threshold of 0.05/1,000 = 0.00005, a test with 999 shuffles can never reach it, because its smallest possible p is 0.001. Reaching the threshold at all takes at least 19,999 shuffles, and estimating p well near it takes many more.

9.3Shuffling what was not randomised

Shuffling individuals in a cluster-randomised trial, or across strata in a stratified one, compares the data with allocations the randomisation could never have produced. Treating the participants of a cluster trial as independent pretends there are more independent units than there are, which usually makes the p-value too small. Follow Table 4.

9.4Reading "non-parametric" as "assumption-free"

A shuffle test of the difference in means compares the data with the null hypothesis that the two groups are exchangeable. In an observational comparison, or for a treatment that may change the spread of outcomes, that means identical distributions, so a small p-value can reflect a difference in spread or shape rather than in the means, and groups of unequal size and spread can make the error rate far from nominal (Table 3).

9.5Choosing the statistic after seeing the data

For the pilot data, the shuffle test gives p = 0.041 for the difference in means and 0.067 for the difference in medians. Both are legitimate, but only one was the plan. Name the statistic in the protocol or the analysis plan, and if you report others, say so and report them all ^[1,4].

10A checklist for authors and reviewers

• The statistic was chosen in advance and is named.
• What was shuffled matches the design: individuals, pairs, clusters, and within which strata or blocks.
• The methods say whether p comes from all allocations or from random shuffles; for random shuffles they give the number of shuffles, the formula (b + 1)/(B + 1), the software and the seed.
• The number of shuffles is large enough for the precision reported and for any multiplicity threshold.
• No p-value is reported as 0.
• For a difference in means between groups of unequal size and spread, a studentised statistic was shuffled or Welch's test was used.
• Two-sided p-values say how the two tails were counted when the groups differ in size.
• The estimate and its confidence interval are reported alongside, with the method for the interval.

A permutation test turns the null distribution into something you can watch being built. Every p-value has one, whether it comes from a formula or from shuffles, and a p-value means only as much as the null distribution behind it.

@misc{korkmaz2026where,
  author       = {Korkmaz, Sel{\c{c}}uk},
  title        = {Where a p-value comes from: the permutation test, the null distribution and the distribution of the p-value itself},
  year         = {2026},
  month        = oct,
  date         = {2026-10-03},
  howpublished = {\url{https://selcukorkmaz.github.io/blog/permutation-test/}},
  url          = {https://selcukorkmaz.github.io/blog/permutation-test/},
  note         = {Blog post}
}