In plain English
Most laboratory papers compare groups: control wells against wells exposed to a peptide, or one assay method against another. Statistics help answer two questions. How big is the difference? And could a difference this big easily appear by chance alone?
The p-value addresses the second question, and only in a narrow way. The confidence interval addresses both. The number of independent repeats, called n, decides how much either can be trusted [2, 3].
The short version
- A small p-value flags data as unusual if the model, including “no difference”, were true. It does not prove an effect [1, 2].
- A non-significant p-value does not show that there is no effect [2, 4].
- Look at the effect size and its confidence interval, not only the p-value [1, 2].
- Check what n counts. Triplicate wells from one experiment are one experiment [3].
What a p-value is, and what it is not
In 2016 the American Statistical Association (ASA) issued a formal statement on p-values. Its six principles, lightly shortened, are [1]:
- P-values can indicate how incompatible the data are with a specified statistical model.
- P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone.
- Scientific conclusions should not be based only on whether a p-value passes a specific threshold.
- Proper inference requires full reporting and transparency.
- A p-value, or statistical significance, does not measure the size of an effect or the importance of a result.
- By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis.
Greenland and colleagues list 25 common misinterpretations of p-values, confidence intervals and power [2]. The ones that turn up most often in laboratory papers are:
| Common reading | Why it is wrong [2] |
|---|---|
| “P = 0.01 means a 1% chance the null hypothesis is true.” | The p-value is calculated assuming the test hypothesis is true; it is not a probability for that hypothesis. |
| “P > 0.05 shows there is no effect.” | A large p-value only says the data are not unusual under the model. Many other hypotheses fit the data too. |
| “Significant means important.” | In large studies, very small effects can be statistically significant. |
| “Not significant means the effect is small.” | In small studies, even large effects can be lost in noise. |
| “P = 0.05 and P ≤ 0.05 mean the same thing.” | Reporting only “P ≤ 0.05” lumps borderline results with very strong ones. Exact p-values should be given unless very small, such as below 0.001. |
| “Two studies with P values on either side of 0.05 conflict.” | Tests are sensitive to differences such as group size; the two results may agree closely. |
Greenland and colleagues also stress that a p-value depends on every assumption behind it, including the assumption that the analysis was not chosen because of the p-value it produced [2].
Confidence intervals
A 95% confidence interval (CI) is a range calculated so that, over many repetitions of the study, 95% of such intervals would contain the true value, if the model’s assumptions hold [2, 3]. It does not mean there is a 95% chance that this particular interval contains the true value: a given interval either contains it or does not [2].
- Width is precision. A narrow interval means a precise estimate; a wide one means a rough estimate [2, 3].
- Read both ends. An interval from 0.5% to 30% says the data are compatible with a tiny effect and a large one.
- Overlap is a poor test. Two 95% intervals can overlap substantially and the difference between the groups can still give P < 0.05 [2].
SD, SEM and CI: which error bar is which?
Error bars can show standard deviations, standard errors, confidence intervals or other quantities, and different types give quite different information [3].
| Bar | What it describes | Gets smaller with more samples? |
|---|---|---|
| Standard deviation (SD) | Spread of individual measurements | No, it estimates the natural scatter |
| Standard error of the mean (SEM) | Precision of the mean: SD ÷ √n | Yes |
| 95% confidence interval | Range of plausible values for the mean | Yes |
Cumming, Fidler and Vaux set out eight rules for error bars [3]. The most useful for a reader are:
- Rule 1: the figure legend must say what the error bars are.
- Rule 2: the value of n must be stated, and kept distinct from the number of replicates.
- Rule 3: error bars and statistics should be shown only for independently repeated experiments, never for replicates. A “representative experiment” has n = 1.
- Rule 5: with n of 10 or more, doubling the SE bars gives roughly the 95% CI. With n = 3, SE bars must be multiplied by 4 [3].
Motulsky lists over-reliance on standard errors, which are often misunderstood, among the common mistakes in data analysis [5]. SEM bars look reassuringly small, especially when n is small.
What does n count?
n is the number of independent results. A replicate is a repeated measurement of the same sample or condition [3].
Example from Cumming and colleagues: an experiment with triplicate cultures, repeated four independent times, has n = 4, not 3 or 12 [3]. The scatter within each set of triplicates reflects how well the replicates were made, not the effect being tested.
For a typical peptide experiment, the same logic applies:
| Set-up | n for a comparison of conditions |
|---|---|
| One plate, one cell passage, 6 wells per condition | 1 (the wells are technical replicates) |
| Three plates on three days from separate cell passages, 6 wells per condition each | 3 |
For instrument measurements, repeated readings of the same prepared solution measure the instrument’s repeatability, not variation between preparations. Independent preparations are what count as n for a comparison between materials.
Pseudoreplication is the error of analysing measurements that are not independent as if they were, for example by running a t-test on 12 wells from one plate. It shrinks the error bars and the p-value without adding any independent evidence. Cumming and colleagues add that for repeated measurements on the same group, such as the same cultures over time, CIs or SE bars are irrelevant to comparisons within that group [3].
Sample size and statistical power
Statistical power is the probability that a study detects an effect of a given size, if that effect is real. A study with low power has a reduced chance of detecting a true effect. Button and colleagues add a less obvious point: low power also reduces the likelihood that a statistically significant result reflects a true effect, and leads to overestimated effect sizes and low reproducibility [6].
Ioannidis modelled the wider problem. A research finding is less likely to be true when studies are smaller, when effect sizes are smaller, when more relationships are tested with less preselection, and when there is more flexibility in designs, definitions, endpoints and analysis [7].
Worked example (illustrative numbers)
A cell-viability assay compares control wells with wells exposed to a peptide. The readout is expressed as a percentage of the control signal. Four independent experiments are run for each condition.
| Condition | Mean signal | SD | n |
|---|---|---|---|
| Control | 100% | 8 | 4 |
| Exposed | 85% | 10 | 4 |
- Difference: −15 percentage points.
- Standard error of the difference (pooled SD 9.06): 9.06 × √(1/4 + 1/4) = 6.40.
- t statistic: 15 ÷ 6.40 = 2.34, with 6 degrees of freedom, giving P ≈ 0.058 (two-sided).
- 95% CI for the difference: −15 ± 2.447 × 6.40, or −30.7 to +0.7 percentage points.
The p-value misses 0.05. But the interval runs from a 31-point drop to essentially no change, so “no effect” would be the wrong conclusion: absence of evidence is not evidence of absence [4]. With a true difference this size (about 1.5 SD), four experiments per condition give only about a 43% chance of reaching P < 0.05. About 8 per condition would be needed for 80% power (a normal approximation gives 7). The honest summary is that the experiment was too small to tell.
Multiple comparisons and p-hacking
Every test run at the 0.05 level carries a 5% false-positive rate when there is no true effect. Across 20 independent tests of true null hypotheses, the chance of at least one P < 0.05 is 1 − 0.95²⁰, about 64%.
- The Bonferroni method controls this by dividing the significance level by the number of tests, so 20 tests would each be judged at 0.0025 [8]. It is simple but conservative.
- p-hacking is collecting or selecting data or analyses until non-significant results become significant. Head and colleagues used text-mining to show that it is widespread throughout science, although its effect on meta-analytic conclusions seemed weak relative to the real effect sizes [9]. Motulsky gives a common form: reanalysing a data set in many ways, or adding replicates, until the desired result appears [5].
- The defence is a plan written before the experiment: the primary readout, the sample size and the analysis. Reporting every test performed lets readers judge the rest [1].
Show the data, not just the bars
In a systematic review of 703 papers in physiology journals, most papers presented continuous data in bar and line graphs, and scatterplots, box plots and histograms were rare. Many different distributions of data can lead to the same bar graph, so for small studies Weissgerber and colleagues recommend showing the individual points [10]. With n = 4, there is no reason not to plot all four.
Reporting standards
For animal studies, the ARRIVE 2.0 guidelines divide reporting items into the “ARRIVE Essential 10”, the minimum requirement, and a “Recommended Set” [11]. They are intended to let others scrutinise the work, evaluate its methodological rigour and reproduce the methods. In vitro studies have no single equivalent, but the same principles apply: state n and what it counts, name the test, give exact p-values and intervals [2, 3]. Pooling results across papers is covered in Systematic Reviews and Meta-Analyses, and potency values such as EC50 in EC50, IC50, Kd and Ki.
A checklist for the statistics in a paper
- Is n stated, and is it independent experiments rather than wells or readings [3]?
- Are the error bars defined in the legend [3]?
- Is the effect size given with a confidence interval, not only a p-value [1, 2]?
- Are exact p-values reported [2]?
- Was the sample size justified in advance [6]?
- How many tests were run, and was there a correction for multiple comparisons [8]?
- Are individual data points shown for small samples [10]?
- Is “not significant” being read as “no effect” [2, 4]?
- Was the material characterised? Identity and peptide content matter as much as statistics; see How to Read a Peptide Certificate of Analysis. Evidence levels are explained in How to Read Peptide Research.
What this means in Australia
- Significance is not approval. A statistically significant result in a cell line or animal model says nothing about whether a substance may be supplied in Australia. The TGA states that unapproved peptide products have not been assessed by the TGA for safety, quality or effectiveness, and that a “research use only” disclaimer does not make supply lawful [12].
- Check the register. Products on the Australian Register of Therapeutic Goods can be looked up in the TGA’s ARTG search [13]; see What Is the ARTG?.
This is general information, not legal advice. Laws change; check the current Poisons Standard and TGA guidance.
Frequently asked questions
What does p < 0.05 mean?
It means that, if the statistical model (usually “no difference”) and all its assumptions were true, data at least this extreme would be expected less than 5% of the time. It does not mean there is a 95% chance the effect is real [1, 2].
What is the difference between SD and SEM?
The SD describes how spread out individual measurements are. The SEM describes how precisely the mean is estimated and shrinks as n grows. Figure legends should say which is shown [3].
What is a 95% confidence interval?
It is a range calculated by a method that, over many repetitions, captures the true value 95% of the time if its assumptions hold. Its width shows how precise the estimate is [2, 3].
What is the difference between biological and technical replicates?
Independent experiments, such as separate cell passages on separate days, are what n counts. Repeated wells or readings within one experiment are technical replicates and show measurement precision, not independent results [3].
Why do small studies give unreliable results?
Small studies have low power. They often miss real effects, and when they do reach significance the effect size tends to be overestimated and harder to reproduce [6, 7].
What is p-hacking?
It is collecting or selecting data or analyses until a non-significant result becomes significant, for example by adding replicates or trying several tests. A pre-specified analysis plan and full reporting guard against it [5, 9].
References
- Wasserstein RL, Lazar NA. The ASA statement on p-values: context, process, and purpose. Am Stat. 2016;70(2):129–133. doi:10.1080/00031305.2016.1154108
- Greenland S, Senn SJ, Rothman KJ, et al. Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. Eur J Epidemiol. 2016;31(4):337–350. doi:10.1007/s10654-016-0149-3. PMID: 27209009
- Cumming G, Fidler F, Vaux DL. Error bars in experimental biology. J Cell Biol. 2007;177(1):7–11. doi:10.1083/jcb.200611141. PMID: 17420288
- Altman DG, Bland JM. Absence of evidence is not evidence of absence. BMJ. 1995;311(7003):485. doi:10.1136/bmj.311.7003.485. PMID: 7647644
- Motulsky HJ. Common misconceptions about data analysis and statistics. Pharmacol Res Perspect. 2015;3(1):e00093. doi:10.1002/prp2.93. PMID: 25692012
- Button KS, Ioannidis JP, Mokrysz C, et al. Power failure: why small sample size undermines the reliability of neuroscience. Nat Rev Neurosci. 2013;14(5):365–376. doi:10.1038/nrn3475. PMID: 23571845
- Ioannidis JP. Why most published research findings are false. PLoS Med. 2005;2(8):e124. doi:10.1371/journal.pmed.0020124. PMID: 16060722
- Bland JM, Altman DG. Multiple significance tests: the Bonferroni method. BMJ. 1995;310(6973):170. doi:10.1136/bmj.310.6973.170. PMID: 7833759
- Head ML, Holman L, Lanfear R, Kahn AT, Jennions MD. The extent and consequences of p-hacking in science. PLoS Biol. 2015;13(3):e1002106. doi:10.1371/journal.pbio.1002106. PMID: 25768323
- Weissgerber TL, Milic NM, Winham SJ, Garovic VD. Beyond bar and line graphs: time for a new data presentation paradigm. PLoS Biol. 2015;13(4):e1002128. doi:10.1371/journal.pbio.1002128. PMID: 25901488
- Percie du Sert N, Hurst V, Ahluwalia A, et al. The ARRIVE guidelines 2.0: updated guidelines for reporting animal research. PLoS Biol. 2020;18(7):e3000410. doi:10.1371/journal.pbio.3000410. PMID: 32663219
- Therapeutic Goods Administration. Understanding your responsibilities when importing, compounding and supplying unapproved peptide products. Safety advisory, 13 April 2026. tga.gov.au. Accessed 25 October 2026.
- Therapeutic Goods Administration. Australian Register of Therapeutic Goods (ARTG) search. tga.gov.au/resources/artg. Accessed 25 October 2026.
