A p-value is not the probability that a result is real, an n of 3 is not a sample size, and a correction chosen after seeing the data is not a correction. Most statistical damage at a bench comes from those three confusions, and avoiding them takes a decision made before the first plate, not after the last.

This guide covers what a p-value licenses, why power belongs to a design rather than a result, what counts as an independent replicate, how multiplicity inflates false positives, and why pre-specifying an analysis differs from choosing one.

What a p-value measures

A p-value is computed under a statistical model containing the hypothesis being tested — usually a null of no difference — together with every other assumption in the analysis: distributional form, independence of the observations, absence of selection, a sample size fixed in advance. It is the probability, under that entire model, of a test statistic at least as extreme as the one observed. The American Statistical Association's statement on p-values, whose six principles have been set out and discussed for a clinical readership, opens with the principle that p-values can indicate how incompatible the data are with a specified statistical model [1].

Incompatible with the model. The null hypothesis is only one of the model's assumptions, and a small p-value cannot say which assumption failed. Greenland and colleagues catalog 25 distinct misinterpretations of p-values, confidence intervals and power, and emphasize that violation of often unstated analysis protocols — their example is selecting analyses for presentation on the basis of the p-values they produce — can lead to small p-values even when the declared test hypothesis is correct, and to large ones even when it is incorrect [2].

Four things a p-value is not, uncited because they follow from the definition:

  • It is not the probability that the null hypothesis is true. That requires a prior, which the calculation never supplies.
  • It is not the probability that the finding will replicate. That depends on the true effect and the replicate design, neither of which enters the calculation.
  • It is not a measure of effect size. A p-value of 0.001 on a large sample can accompany a difference too small to matter.
  • It is not a property a result has or lacks. It is continuous, and the 0.05 line through it is a convention.

The ASA's third principle, per that same article, is that scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold; the authors of that article go on to recommend that investigators report the magnitude of effect of all outcomes together with an appropriate measure of precision or variation [1].

Power is a property of the design, not of the result

Statistical power is the probability that a test will return a p-value below the chosen threshold, given a specified true effect size, variability and n. All three are required. Power is therefore something a design has before any data exist, and quoting a figure computed from the observed effect afterward is circular: observed power is a deterministic function of the p-value. Power is among the quantities Greenland and colleagues list as routinely misinterpreted [2].

The less obvious consequence, and the one Button and colleagues make central, is that low power does not only reduce the chance of detecting a true effect: it also reduces the likelihood that a significant result reflects one, and it produces overestimates of effect size and low reproducibility. Those authors show the average statistical power of studies in the neurosciences is very low [3].

The first half of that claim is arithmetic. Suppose, as a supposition rather than a measured figure, that 10% of a laboratory's tested hypotheses are true: across 1,000 hypotheses, 100 true and 900 false. At a threshold of 0.05 and power of 0.20, true positives are 100 x 0.20 = 20 and false positives 900 x 0.05 = 45, so 20 / (20 + 45) = 20 / 65 = 31% of positive results are true. Raise power to 0.80 and true positives become 80 while false positives stay at 45: 80 / 125 = 64%. The denominator is all positives, not all hypotheses — the step most often taken wrongly. The threshold did not move; the design did.

Why n = 3 is usually a decision not to find out

For a two-group comparison with a two-sided t-test at a threshold of 0.05, power depends on n and on the standardized effect size d, the difference in means divided by the pooled standard deviation. The following were computed from the noncentral t distribution for this guide:

  • n = 3 per group: power is about 0.16 at d = 1.0, 0.46 at d = 2.0, and 0.78 at d = 3.0.
  • n = 6 per group: power is about 0.35 at d = 1.0 and 0.65 at d = 1.5.
  • Reaching 0.80 power takes roughly n = 17 per group at d = 1.0, n = 26 at d = 0.8, n = 64 at d = 0.5.

At n = 3 per group, reaching 80% power requires a standardized effect of about 3.07 — group means separated by three pooled standard deviations. Effects that large do occur at a bench, and for those an n of 3 is defensible. For anything smaller it will miss a real effect most of the time, and its occasional hit will overstate that effect, because only the large sampling excursions clear the threshold.

Interval width says the same without a threshold. The half-width of a 95% confidence interval on a difference of means is t x s x sqrt(2/n): at n = 3, 2.776 x 0.8165 x s = 2.267 s; at n = 12, 2.074 x 0.4082 x s = 0.847 s — about 2.7 times narrower. An n of 3 rarely fails to produce a number; it fails to constrain one.

Where n = 3 comes from is worth stating plainly: it is convention, inherited from plate layouts and from what reviewers accept, not from any analysis identifying three as sufficient. An n should be justified by a target effect, a variability estimate and a stated power — and where those inputs do not exist, saying so beats a guess dressed as a calculation.

What counts as n

A power calculation is only as good as the unit it counts. Lazic, Clarke-Williams and Munafo argue that independent replication of the entity-intervention pair is what contributes to N, and that choosing the wrong pair means the experiment cannot address the question asked [4]. They surveyed a random sample of published animal experiments from 2011 to 2016 in which interventions were applied to parents and effects examined in the offspring: 22% (95% CI 17% to 29%) replicated the correct entity-intervention pair, 46% (38% to 53%) had pseudoreplication, and 32% (26% to 39%) reported too little to judge [4].

An earlier survey by Lazic used a single issue of Nature Neuroscience as a case study and reported 12% of papers had pseudoreplication and a further 36% were suspected of it, the suspected cases impossible to determine because insufficient information was provided; he argues sample size, degrees of freedom, test statistic and precise p-values should be a requirement for all publications [5]. Lord and colleagues make the parallel point for cell biology: the n used for statistical tests should represent biological replicates, independent measurements of the population from separate experiments [6].

A worked case. A compound is added to three flasks and vehicle to three others; after incubation each flask's contents are split across four wells for readout, and each well is injected twice. That produces 24 wells and 48 injections, but the intervention was applied six times. The experimental unit is the flask, the entity to which treatment was independently applied, so n is 3 per group. A two-sample t-test on the six flask means carries 3 + 3 - 2 = 4 degrees of freedom and a critical t of 2.776. Run on the 24 well values it would claim 12 + 12 - 2 = 22 degrees of freedom and a critical t of 2.074, about 25% smaller, with a variance pooled across the 12 wells in each group — a quantity dominated by well-to-well scatter rather than the flask-to-flask variation the comparison is actually about. Pseudoreplication inflates the sample size and the apparent evidence for a claim, giving false positives [4].

The unit is not fixed by the labware. Had the compound been applied independently to each well from cultures prepared on different days, the well could be the experimental unit — the criterion is independent application of the entity-intervention pair [4], not the vessel. The injections are technical replicates either way: they measure instrument precision and are averaged before the biological comparison, by convention rather than any finding.

Multiplicity: the family, not the test

Testing more than one null hypothesis from a single experiment inflates the risk of at least one false positive, and multiple comparison procedures exist to control that inflation [7]. With m independent tests each at a 0.05 threshold, the probability of at least one false positive when all nulls are true is 1 - 0.95^m: 14% at m = 3, 40% at m = 10, and 64% at m = 20.

Those are not hypothetical rates. Reviewing 195 original clinical research articles with more than 100 subjects published in 2012 across four general otolaryngology journals, Kirkham and Weaver found 72% met their criterion for multiple testing — more than five hypotheses in at least one family of inferences — with a mean 41% chance of a type I error in those studies and, on average, 18% of significant results likely to be false positives; after Bonferroni correction only 57% of the reported significant results remained significant [8].

Two different things to control

Family-wise error rate is the probability of one or more false positives anywhere in a declared family. Bonferroni and Sidak one-step procedures control it, as do Tukey-Kramer for all pairwise contrasts of means and Dunnett for contrasts against a control mean, the latter two for continuous data under normal distribution theory [7]. Ludbrook commends Holm's step-down procedures for accuracy, power and versatility: they control the family-wise rate as effectively as the one-step procedures, are more powerful than one-step Bonferroni or Sidak — especially when hypotheses are logically related — and apply to ordinal and categorical data as well as continuous [7]. Curran-Everett, reviewing the three procedures cited most often in his search (Newman-Keuls, Bonferroni and least significant difference), concludes each is of limited practical value [9].

False discovery rate is the expected proportion of false positives among the tests declared positive, the better target when the family is large and false leads are tolerable. Storey and Tibshirani proposed the q-value for genomewide studies, a measure of significance in terms of the false discovery rate rather than the false positive rate, describing it as avoiding a flood of false positives while offering a more liberal criterion than genome scans for linkage had used [10]. Curran-Everett called the false discovery rate procedure a promising development that may be the best practical solution to multiple comparisons in physiology and other disciplines [9].

The difference is large in practice. Take ten sorted p-values: 0.001, 0.008, 0.019, 0.032, 0.041, 0.060, 0.30, 0.44, 0.68, 0.91. Bonferroni at a family-wise 0.05 compares each to 0.05 / 10 = 0.005; only 0.001 survives. Benjamini-Hochberg at a false discovery rate of 0.10 compares the i-th sorted value to (i / 10) x 0.10 — thresholds of 0.010, 0.020, 0.030, 0.040, 0.050 and 0.060 for the first six. Each of those six clears its own threshold (0.060 meets 0.060 exactly), the seventh does not, and the procedure rejects all six. One discovery or six, from identical data — which is why the choice belongs in the protocol rather than the discussion.

Pre-specification versus post-hoc selection

Every procedure above assumes the family of tests was fixed before the data were seen. When it was not, the arithmetic does not apply and no correction repairs that. Head and colleagues used text mining to demonstrate that p-hacking — collecting or selecting data or analyses until nonsignificant results become significant — is widespread throughout science, though they also report that its effect seems to be weak relative to the real effect sizes being measured and suggest it probably does not drastically alter consensuses drawn from meta-analyses [11]. That hedge is theirs: the practice is common, which is not the same as every literature built on it being wrong.

Sample loss is the version that looks innocent. Holman and colleagues simulated random and biased attrition in preclinical experiments and found random loss decreased statistical power, as expected, while biased removal — including the removal of outliers — dramatically increased the probability of false positives; their accompanying meta-analysis of stroke and cancer animal studies found most papers did not adequately report attrition, and, extrapolating from the simulations, they suggest those effect sizes were likely overestimated [12]. An exclusion rule written before unblinding is a design feature. The same rule applied after seeing which points sit awkwardly is a selection effect.

The remedy is to write the analysis down first: the primary outcome, the comparison, the test, the family over which multiplicity will be controlled, the exclusion criteria, and n with its justification. Preregistration makes that record external and dated. The Animal Study Registry, launched in January 2019 free of charge, was designed for exploratory and confirmatory studies in applied, basic and preclinical research; registration assigns a DOI, and visibility can be restricted for up to five years [13]. Analyses not written down first are still worth running — but they are hypothesis-generating, and saying so costs nothing.

What is not settled

The 0.05 threshold and the 80% power target are conventions with no derivation behind them; both are choices about the relative cost of the two errors. There is no general rule for what constitutes a family — whether a panel of readouts on one plate, one figure or one paper is the right boundary is the analyst's judgment, and it changes the correction. Whether family-wise or false-discovery control fits depends on the cost of a false lead in that program, which no procedure supplies. And the effect size feeding a power calculation is usually estimated from a small pilot, so it carries the same wide interval as any small-n estimate; under low power the effects clearing the threshold are systematically the overestimated ones [3], making a pilot's own significant result a poor basis for the calculation it feeds.