The p-value, the effect size and the confidence interval What each tells the reader, and how to report them in a paper

@misc{korkmaz2026p,
  author       = {Korkmaz, Sel{\c{c}}uk},
  title        = {The p-value, the effect size and the confidence interval: what each tells the reader, and how to report them in a paper},
  year         = {2026},
  month        = oct,
  date         = {2026-10-01},
  howpublished = {\url{https://selcukorkmaz.github.io/blog/p-effect-size-ci/}},
  url          = {https://selcukorkmaz.github.io/blog/p-effect-size-ci/},
  note         = {Blog post}
}

1Three numbers, three questions

Suppose a trial randomises 130 adults, 65 to a lifestyle programme and 65 to usual care, and measures systolic blood pressure after 12 weeks. The standard deviation (SD) is about 16 mmHg in both arms, and the programme arm ends up 6.2 mmHg lower. The results section could summarise this one finding with three numbers, and each answers a different question (Table 1).

The three are not independent. The p-value and the interval are both built from the same two ingredients, the estimate (which is the effect size) and its standard error (SE), which I described in an earlier note on the SD and the SE. For a difference in two means:

Here group 1 is usual care and group 2 the programme, each with mean x̄, SD s and size n, so D is usual care minus programme (positive values favour the programme); t_0.975 is the 97.5th percentile of the t distribution with the test's degrees of freedom, and T follows that distribution. In the example, SE = 16 × √[2/65] = 2.81 mmHg. The test statistic is 6.2 / 2.81 = 2.21 on 128 degrees of freedom, which gives p = 0.03, and with t_0.975 = 1.98 the 95% CI is 6.2 ± 1.98 × 2.81, or 0.6 to 11.8 mmHg.

Because the ingredients are shared, a two-sided p-value below 0.05 and a 95% confidence interval that excludes zero are the same statement, when both are computed from the same test statistic and standard error, as with the t-test and t-interval here. (For proportions, a Wald interval and a pooled test can disagree near the boundary.) So the interval already tells the reader whether p is below 0.05, and it adds the size and precision that the p-value hides. Gardner and Altman recommended reporting both, and dropping the p-value first if one had to be omitted ^[1].

2The p-value

A p-value is the probability of getting a test statistic at least as extreme as the one observed, computed on the assumption that the null hypothesis and every other assumption of the statistical model are true ^[2]. In the example, p = 0.03 says that if the programme had no effect at all and the model were right, a difference of 6.2 mmHg or more in either direction would turn up about 3 times in 100 repetitions of the trial.

Greenland and colleagues describe it as a continuous measure of compatibility between the data and the whole set of assumptions used to compute it: a small value signals that something in that set, the null hypothesis or another assumption, is doubtful ^[2]. Rafi and Greenland add a way to feel its size: the surprisal, or S-value, S = −log₂(p), in bits (written S here to keep it apart from the SD s). A p of 0.03 is about 5 bits, as surprising as five heads in a row from a fair coin ^[3].

The most useful thing to know about a p-value is that it depends on the sample size as much as on the effect. Figure 1 holds the SD at 16 mmHg and changes the group size. In one view the observed difference stays at 5 mmHg, and p nevertheless runs from 0.49 to below 0.001. In the other, every row is chosen so that p = 0.03, and the estimate runs from 16.9 mmHg down to 0.9.

Two consequences follow. A p-value below 0.05 says the data are hard to reconcile with an effect of exactly zero. It does not say the effect is big enough to matter, and in a large enough study a difference of 0.9 mmHg will reach it. A p-value above 0.05 does not say there is no effect either; it may only mean the study was small ^[2].

3The effect size

An effect size is how large the difference or association is. It answers the question the reader actually has: how much? It can be unstandardised, in the units of the outcome (6.2 mmHg, 2 days, 8 percentage points), or standardised, expressed in units of variability or as a ratio, so that it has no units (Table 2).

Prefer the unstandardised form whenever the units mean something to your readers, because they can judge it directly against what they know about blood pressure, length of stay or risk ^[5,6]. Use a standardised effect size when the outcome has no intrinsic units ^[6], when studies used different scales to measure the same construct ^[5,6], or when you need to compare or pool results across studies ^[6], and give its confidence interval as well. Standardising an arbitrary or poorly measured scale obscures its deficiencies rather than fixing them ^[5].

The best-known standardised measure for two means is Cohen's d, the difference divided by the pooled SD. Hedges' g multiplies it by a small correction, which matters when the groups are small ^[7,8]. An approximate SE for g gives its confidence interval ^[8]:

In the example, d = 6.2 / 16 = 0.39, g = 0.39 and its 95% CI, g ± 1.96 × SE(g) with SE(g) = 0.177, is 0.04 to 0.73. A standardised effect is only as interpretable as its denominator: d shrinks if the sample is more heterogeneous, even when the raw effect is unchanged ^[5]. State which SD was used.

3.1What counts as large?

Cohen's conventions are 0.2, 0.5 and 0.8 for d (small, medium, large), with matching values for other measures (Table 3) ^[9]. Cohen presented them as reference points to use only when nothing better is available ^[7,9]. Something better is almost always available: the smallest effect that would matter to patients, clinicians or the science. The Statistical Analyses and Methods in the Published Literature (SAMPL) guidelines ask authors, where possible, to say what minimum difference they consider clinically important, and not to call a correlation low, moderate or high unless they define the ranges ^[10]. For patient outcomes, that threshold is often called the minimal clinically important difference, which was originally defined as the smallest difference in the domain of interest that patients perceive as beneficial and that would mandate a change in management, in the absence of troublesome side effects and excessive cost ^[11]. It is the yardstick for reading an interval in Section 5.

4The confidence interval

A 95% confidence interval is the range of values produced by a procedure that, across repeated studies and if the model's assumptions hold, would contain the true value in 95% of them ^[2]. That statement is about the procedure, not about the one interval in front of you. The reading that works in practice is the one Rafi and Greenland propose: an interval shows the values most compatible with the data under the model, with the estimate in the middle the most compatible, and values near the ends less so ^[3]. The usual shortcut, "there is a 95% probability that the true value lies in this interval", is not what the interval means, and it is one of the misreadings that researchers commonly endorse ^[12].

Figure 2 shows how the interval, the estimate and the p-value fit together. Take every possible true difference in turn and ask how compatible the data are with it, using the same t-test that gave the p-value for zero. The result is a curve, the p-value function. Its peak is the estimate. The p-value reported for "no difference" is only the height of the curve at zero. And the 95% confidence interval is the stretch of the horizontal axis where the curve lies above 0.05 ^[3]. The curve is not a probability distribution for the true difference ^[3]: its height is the two-sided p-value of each hypothesis. It equals exactly 1 at the estimate, where the test statistic is 0, and it has a sharp point there because the p-value ignores the sign of the difference.

Three practical points follow from the picture.

• Width is precision. The interval's width shrinks in proportion to 1/√n, so four times as many participants roughly halve it, and a wide interval says the study cannot pin the effect down, whatever its p-value.
• Ratio intervals are symmetric on the log scale. Intervals for risk ratios, odds ratios and hazard ratios are usually calculated on the log scale, so they are not centred on the estimate, and the no-effect value is 1, not 0.
• An interval is only as good as its assumptions. It reflects sampling variation only ^[1]; it does not correct for bias, and it is a best-case measure of uncertainty because it depends on the model being right ^[2]. A narrow interval around a biased estimate is a precise wrong answer. And intervals of two separate groups say little about their difference ^[2]; estimate the difference itself, as the note on error bars shows.

Gardner and Altman argued in 1986 that confidence intervals should be used for the major findings, in the main text and in the abstract ^[1]. Current guidelines, discussed in Section 6, ask for the same estimate-plus-interval reporting ^[10,13,14].

5Reading the three together

Figure 3 places six illustrative intervals against a zone of trivial effects (here ±5 mmHg). The first three are statistically significant, and the last three are not, yet within each group the meaning differs a great deal (Table 4).

Rows 4 and 5 show why "not significant" must never be read as "no effect". The mistake is common: in a compilation of four published surveys covering 791 articles in five journals, Amrhein and colleagues reported that about half (402, or 51%) wrongly read a non-significant result as showing no effect ^[15]. As Altman and Bland put it, absence of evidence is not evidence of absence ^[16]. To claim that an effect is negligible, as in row 6, you need a confidence interval that lies inside a margin based on the smallest effect of interest, which is the logic of equivalence testing ^[17]. Set the margin before seeing the data. The usual two one-sided tests at the 5% level correspond to a 90% interval, and a 95% interval is more demanding ^[17]. SAMPL asks authors to report the equivalence margin ^[10].

6How to report them in a paper

Reporting guidelines agree. The SAMPL guidelines ask authors to give the estimate, or effect size, with a measure of precision, usually a 95% confidence interval, and not to rely on p-values alone ^[10]. The Consolidated Standards of Reporting Trials (CONSORT) 2025 statement, item 26, asks trial reports to give, for each primary and secondary outcome, the estimated effect size and its precision, such as a 95% CI, and for binary outcomes both the absolute and the relative effect ^[13,18]. The International Committee of Medical Journal Editors (ICMJE) recommendations ask for appropriate measures of uncertainty, such as confidence intervals, rather than reliance on hypothesis tests alone ^[14].

6.1The basic sentence

Table 5 shows one result written three ways. The weak version hides the size and the precision; the better one gives the size but not the precision; the best gives both and reads them against the smallest effect of interest.

6.2Formatting rules

• Say what the difference is. Name the direction (for example usual care minus programme, or "lower in the programme group"), the units and the confidence level (95% CI).
• Write intervals with "to". "−3.9 to 6.3" is unambiguous with negative numbers, where a dash is not. Give the estimate and both limits the same number of decimals, and no more precision than the data support.
• Give exact p-values. Write p = 0.03, not p < 0.05, and never "NS". SAMPL calls p-values less preferred than confidence intervals but, if they are given, asks for equalities to one or two decimals, with p < 0.001 as the smallest value worth reporting ^[10]. My own habit is two decimals, three below 0.01 or when the value is close to 0.05 (so 0.049 is not rounded to 0.05). Never write p = 0.000.
• Name the test and the sidedness in the methods, and say whether tests were two-sided and what α was, if you use a threshold at all ^[10].
• Do not use stars alone. Asterisks in a table are no substitute for the estimate and the interval.
• Define the standardised effect size. Say whether d or g, which SD was used ^[7], which group was subtracted from which, and give its 95% CI.
• Take a position on "significant". Some statisticians recommend dropping the phrase "statistically significant" altogether ^[15,19], and journals differ. If you use it, define α in advance and never let it stand in for the estimate.

6.3Binary outcomes: report both scales

For a binary outcome, give the absolute effect (risk difference) and the relative effect (risk ratio or odds ratio), each with its interval ^[13]; the number needed to treat (NNT) can also help ^[13]. A relative effect alone can make a rare outcome look important, and an absolute effect alone hides how it scales with baseline risk ^[13]. With 18 events among 200 participants in one arm and 34 among 200 in the other:

The interval for the NNT comes from inverting the unrounded limits of the risk difference. That is straightforward only when the risk-difference interval excludes zero, as it does here; when it does not, the interval for the NNT is not a single range, and Altman shows how to report it ^[20].

6.4Non-significant results

Report them exactly like significant ones, with the estimate, the interval and the exact p-value, and describe what the interval rules out:

6.5Tables, figures and the abstract

In a table, give one column for the estimate with its interval and, if wanted, one for the p-value; state in the footnote what the estimate is, what test gave the p-value, and whether tests were adjusted for multiplicity. For many outcomes or subgroups, a forest plot with the reference line at no effect and a second line at the smallest effect of interest lets readers do Section 5 for themselves. The abstract should carry the estimate and 95% interval of the primary outcome, not just a p-value ^[1]. Table 6 lists what to report for common analyses.

6.6Sentences to adapt

• Methods. "The primary outcome was the between-group difference in systolic blood pressure at 12 weeks, estimated with Welch's t-test. Tests were two-sided with α = 0.05, and results are reported as differences with 95% confidence intervals and exact p-values. The smallest effect of interest, 5 mmHg, was specified before data collection. No adjustment was made for secondary outcomes, which are exploratory."
• Table footnote. "Estimates are differences in means (usual care − programme; positive values favour the programme) with 95% CI from Welch's t-test; p, two-sided, unadjusted."
• Figure legend. "Squares show the reduction in systolic blood pressure (usual care − programme; positive values favour the programme) and horizontal lines show 95% confidence intervals. The dashed line marks the smallest effect of interest (5 mmHg)."

Base R gives the estimate, interval and p-value for a two-group comparison of means or proportions; the risk ratio and the NNT above are computed by hand from the counts. The effectsize package adds the standardised effect with its interval; check that its sign convention matches the direction you report. Its interval is the exact (noncentral t) one, so it can differ slightly from the approximation above.

d$arm <- factor(d$arm, levels = c("usual care", "programme"))   # first − second
fit <- t.test(sbp ~ arm, data = d)              # Welch test
unname(fit$estimate[1] - fit$estimate[2])       # usual care − programme
fit$conf.int                                    # its 95% CI
fit$p.value                                     # two-sided p
effectsize::hedges_g(sbp ~ arm, data = d)       # Hedges' g with 95% CI

# Wald CI and pooled-test p; sign is intervention − control
prop.test(c(18, 34), c(200, 200), correct = FALSE)

7Five traps

7.1Treating 0.05 as a cliff

A result with p = 0.049 and one with p = 0.051 are practically the same evidence, and no sensible reading treats them as opposites. Phrases such as "a trend towards significance" for 0.06, or "highly significant" for 0.001, treat the threshold as if it were a fact about the world. Report the number and let readers weigh it. The American Statistical Association's statement says that conclusions should not be based only on whether a p-value passes a threshold ^[4].

7.2Reading "not significant" as "no effect"

Covered in Section 5: a wide interval that includes zero is not evidence of absence ^[16]. Read the whole interval, and reserve claims of no important effect for intervals inside a margin set in advance.

7.3Comparing significance levels of two groups

The difference between "significant" and "not significant" is not itself statistically significant ^[21]. Suppose a treatment effect is 25 (SE 10) in one subgroup and 10 (SE 10) in another. The first is significant (p = 0.01) and the second is not (p = 0.32), but the difference between them is 15 with an SE of 14, and p = 0.29. To claim that subgroups differ, test the difference, with an interaction term, and report its interval.

7.4Reporting observed power

Power calculated from the observed result is a direct function of the p-value and adds no information; a non-significant result always has observed power of about 50% or less ^[22]. Report the confidence interval instead, which shows what the study could and could not exclude. Power belongs in the planning of a study, computed for a difference you named in advance.

7.5Uncounted tests

With 20 independent tests of true null hypotheses at α = 0.05, the chance of at least one p below 0.05 is 1 − 0.95²⁰ = 64%. Reporting only the significant ones, or presenting a chosen analysis as if it had been the plan, makes the p-values and the intervals misleading ^[2]. State which outcomes were primary and how multiplicity was handled ^[10].

8A checklist for authors and reviewers

• Every main result gives the estimate in original units, its direction, and a 95% confidence interval.
• The comparison is stated (A − B, or "lower in group A"), and so is the confidence level.
• p-values are exact (p = 0.03; p < 0.001 only below that), never "NS", never p = 0.000, and never stars alone.
• Standardised effect sizes are named and defined, come with a confidence interval, and are not judged by Cohen's labels alone.
• Binary outcomes give the absolute and the relative effect.
• The smallest effect of interest is stated in the methods, with how it was chosen, and the interval is read against it.
• Non-significant results are described by what their interval excludes, not as "no effect".
• The test, the sidedness, α, the primary outcome, and any adjustment for multiple comparisons are in the methods.
• The abstract has the estimate and interval of the primary outcome.

The p-value tells the reader whether zero is a comfortable value for the effect. The effect size tells them how large it is, and the interval how well the study knows it. Give them the last two first ^[1].

@misc{korkmaz2026p,
  author       = {Korkmaz, Sel{\c{c}}uk},
  title        = {The p-value, the effect size and the confidence interval: what each tells the reader, and how to report them in a paper},
  year         = {2026},
  month        = oct,
  date         = {2026-10-01},
  howpublished = {\url{https://selcukorkmaz.github.io/blog/p-effect-size-ci/}},
  url          = {https://selcukorkmaz.github.io/blog/p-effect-size-ci/},
  note         = {Blog post}
}