Systematic Reviews and Meta-Analyses: How Evidence Is Pooled

In plain English

A review article sums up many studies. In a traditional review (a narrative review), the authors choose which studies to discuss, so the picture can lean towards what they already think. A systematic review follows a written plan instead. It states the question, searches databases in a way others can repeat, and uses fixed rules to decide which studies count ^[1].

A meta-analysis is a maths step that can come at the end. It combines the numbers from several studies into one overall estimate, and the more precise studies count for more ^{[7, 9]}. Not every systematic review has one.

Pooling does not repair weak studies. If the original experiments were small or biased, the pooled number carries the same problems ^[13]. A review of animal studies is still animal evidence.

The short version

• A systematic review is defined by its methods, not its length ^[1].
• A meta-analysis is the statistics, and it is optional ^[1].
• Check for a registered plan, a full search, risk-of-bias ratings and a measure of inconsistency (I²) ^{[1, 2, 9]}.
• A pooled estimate is only as good as the studies inside it.

Narrative review, systematic review, meta-analysis: what is the difference?

PRISMA 2020, the main reporting guideline for systematic reviews, defines a systematic review as one that “uses explicit, systematic methods to collate and synthesise findings” of studies addressing a clearly formulated question ^[1]. It counts meta-analysis as one form of statistical synthesis among several; others include combining P values and vote counting based on the direction of effect ^[1].

A systematic review is best seen as a way of appraising and combining evidence, not as a level of evidence in its own right ^[14]. Evidence levels are covered in How to Read Peptide Research.

How a systematic review is done

1. A plan written and registered first

Before screening begins, reviewers write down the question, the eligibility criteria and the planned analysis, and register them publicly. PROSPERO is an international prospective register of systematic reviews, developed by the Centre for Reviews and Dissemination at the University of York. It offers free registration and free public access, and its public records include an audit trail of major changes to planned methods ^[2]. When it launched, registration was ideally done before formal screening, and the stated aims were to reduce unplanned duplication of reviews and to minimise reporting bias ^[2]. PRISMA 2020 asks authors to give registration details and to describe any amendments to the registered plan ^[1].

2. A search that others can repeat

PRISMA 2020 asks reviewers to list every database, register and website searched, the date each was last searched, and the full search strategies with any filters and limits ^[1]. A reader should be able to rerun the search. For the practical side, see How to Search PubMed.

3. Screening and selection

Records (titles and abstracts) are screened first, then full reports. Records that refer to the same report are duplicates and are removed ^[1]. PRISMA 2020 asks authors to state how many reviewers screened each record and report, whether they worked independently, and to present the numbers at each stage in a flow diagram ^[1].

4. Risk-of-bias assessment

Each included study is rated with a named tool. The choice of tool depends on the study design.

Risk of bias is about how a study was done, not how well it was written up.

Meta-analysis basics

Effect sizes

Each study contributes an effect estimate (for example a mean difference or a ratio) and a measure of its precision, such as a standard error ^[1]. The statistics behind these quantities are explained in Statistics in Research Papers.

Fixed-effect or random-effects?

The two common models make different assumptions ^{[7, 9]}:

• Fixed-effect: the true effect is the same in every study, so the differences between studies are due to chance alone ^[9].
• Random-effects: the true effects vary between studies, usually assumed to follow a normal distribution, so differences reflect both chance and genuine variation ^[9].

The models can give similar numbers, but they rest on different assumptions, and the choice frames how the pooled estimate should be interpreted ^[7].

Heterogeneity: Q, I² and the prediction interval

Heterogeneity means the studies disagree by more than chance would explain.

• Cochran’s Q test checks whether there is any heterogeneity. It has low power when studies are small or few, so a non-significant Q is not evidence of no heterogeneity; a P value of 0.10 is sometimes used as the threshold instead of 0.05 ^[9].
• I² describes the proportion of the total variation in study estimates that is due to heterogeneity rather than chance ^[5]. It was developed so that it does not depend on the number of studies or the effect measure ^[5], and by 2003 Cochrane reviews had started reporting it ^[6]. It is calculated as I² = (Q − df) / Q, where df is the number of studies minus one, and set to 0 if negative.
• A rough guide from the Cochrane Handbook: 0–40% might not be important, 30–60% may be moderate, 50–90% may be substantial, and 75–100% is considerable. The ranges overlap, and I² is itself very uncertain when there are few studies ^[9].
• A prediction interval gives the expected range of true effects in similar studies. In a survey of 479 statistically significant Cochrane meta-analyses with some heterogeneity, the 95% prediction interval included no effect, or an effect in the opposite direction, in 72.4% of them ^[15].

Worked example (illustrative numbers)

Five hypothetical method-comparison studies each measured the same reference materials by two quantification methods and reported the mean difference in peptide content (method A minus method B, in percentage points of mass fraction), with its standard error (SE).

1. Inverse-variance pooling. Each study counts in proportion to 1/SE². Study 4 gives 1/0.4² = 6.25 out of a total of 16.30, so it contributes 38.3% of the fixed-effect estimate.
2. Fixed-effect estimate: 1.56 percentage points (95% CI 1.08 to 2.05).
3. Heterogeneity: Q = 9.29 with 4 degrees of freedom (P ≈ 0.054). I² = (9.29 − 4) / 9.29 = 57%. The Q test misses the 0.05 threshold, yet more than half the variation is between studies: a good illustration of the test’s low power ^[9].
4. Random-effects estimate: with an estimated between-study variance (τ²) of 0.44, the estimate becomes 1.79 percentage points (95% CI 1.01 to 2.57). The interval widens, and the small studies gain share.
5. 95% prediction interval: −0.67 to 4.25 percentage points. A new, similar comparison could plausibly find no difference at all, even though the pooled estimate is clearly above zero.

Reading a forest plot

A forest plot “displays effect estimates and confidence intervals for both individual studies and meta-analyses” ^[9]. Each study is a square on a horizontal line (its confidence interval), and a larger square means a larger share of the pooled estimate. The pooled estimate is a diamond whose width is its confidence interval. Check whether the study intervals overlap, whether one study dominates, and whether the diamond crosses the line of no effect.

Publication bias and funnel plots

If studies with unremarkable findings stay unpublished, a meta-analysis of the published ones can overstate the effect ^{[8, 11]}. A funnel plot is a scatter plot of the effect estimates from individual studies against a measure of their size or precision ^{[8, 11]}. In the absence of bias, the points are expected to form a roughly symmetrical, inverted funnel, because small studies scatter more widely than large ones.

• Egger’s test (1997) measures funnel-plot asymmetry with a regression. In that study, asymmetry indicated bias in 14 of 37 meta-analyses (38%) from four general journals and in 5 of 38 Cochrane reviews (13%). In the four pairs where a meta-analysis and a single large study disagreed, the meta-analysis had shown the larger effect every time ^[8].
• Use with care. Asymmetry tests have limited ability to detect bias when a meta-analysis rests on a few small studies ^[8]. As a rule of thumb, they should be used only when there are at least 10 studies ^[11].
• Asymmetry is not proof of publication bias. Small-study effects can also come from true heterogeneity, artefacts and chance ^[11].

Certainty of evidence: GRADE

GRADE (Grading of Recommendations Assessment, Development and Evaluation) rates the certainty of a body of evidence for each measured endpoint ^{[1, 12, 16]}. It uses four categories, high, moderate, low and very low, applied to the whole body of evidence rather than to individual studies. Randomised trials start as high and observational studies as low. Certainty can be rated down for risk of bias, imprecision, inconsistency, indirectness and publication bias ^[12]. Early GRADE papers called this “quality”; PRISMA 2020 notes the shift in terminology from assessing “quality” to assessing “certainty” ^[1].

Systematic reviews of laboratory and animal studies

Systematic reviews of preclinical studies have become more common. Their authors argue that empirical evidence shows too many preclinical experiments lack methodological rigour, which inflates effect estimates, and that reviews can push the field towards better conduct and reporting ^[13].

• Small, varied studies. Preclinical studies are often small, and heterogeneity between them is often substantial. Both affect how effect sizes are calculated and pooled ^[17].
• Dedicated tools. SYRCLE’s risk-of-bias tool adapts the Cochrane approach to animal studies ^[4]. The CAMARADES group at the University of Edinburgh produces tools and resources to help scientists conduct preclinical reviews ^[18].
• Evidence level does not change. Pooling 30 rat studies gives a more precise estimate of what happened in rats, not human evidence.

Between 1991 and 2014, annual publication of systematic reviews rose by 2,728% and of meta-analyses by 2,635%, against 153% for all PubMed-indexed items. The author of that analysis concluded that few systematic reviews and meta-analyses are both non-misleading and useful ^[19].

A checklist for reading a systematic review

1. Was the plan registered in advance, and are amendments explained ^{[1, 2]}?
2. Is the full search strategy given for every database, with the date of each search ^[1]?
3. Did at least two people screen independently, and is there a flow diagram ^[1]?
4. Was a named risk-of-bias tool applied to every included study ^{[3, 4]}?
5. Is the choice of fixed-effect or random-effects model justified ^[7]?
6. Is heterogeneity reported with I², τ² and, ideally, a prediction interval ^{[9, 15]}?
7. Was publication bias assessed only where there were enough studies (about 10 or more) ^[11]?
8. Is certainty rated for each endpoint, for example with GRADE ^[12]?
9. Was the molecule in each study characterised? Pooling studies that used poorly identified material adds noise; see How to Read a Peptide Certificate of Analysis.
10. Are competing interests and data availability declared ^[1]? For judging the journals themselves, see Peer Review, Retractions and Predatory Journals.

What this means in Australia

• An Australian-led guideline. PRISMA 2020 was developed by an international group whose lead author is based at Monash University in Melbourne ^[1].
• A review is not an approval. However consistent its findings, a systematic review does not change a product’s regulatory status. Products on the Australian Register of Therapeutic Goods can be checked in the TGA’s online ARTG search ^[20]; see What Is the ARTG?.
• Unapproved peptides. The TGA states that unapproved peptide products have not been assessed by the TGA for safety, quality or effectiveness, and that a “research use only” disclaimer does not make supply lawful, change regulatory status, permit importation, or remove advertising or supply obligations ^[21].

This is general information, not legal advice. Laws change; check the current Poisons Standard and TGA guidance.

Frequently asked questions

What is the difference between a systematic review and a meta-analysis?

A systematic review is the whole process of finding, selecting and appraising studies with explicit methods. A meta-analysis is the optional statistical step that combines their effect estimates into one summary ^[1].

What does I² mean?

I² is the percentage of the variation between study estimates that is due to genuine differences between studies rather than chance. Higher values mean the studies disagree more, but I² is very uncertain when there are few studies ^{[5, 9]}.

Should a meta-analysis use a fixed-effect or random-effects model?

Fixed-effect assumes one true effect shared by every study; random-effects assumes the true effect varies between studies. The choice should match what the reviewers believe about their studies and be justified in advance ^{[7, 9]}.

What does a funnel plot show?

It plots each study’s effect estimate against its size or precision. Asymmetry can suggest publication bias, but it can also arise from heterogeneity or chance, and tests need about 10 or more studies ^{[8, 11]}.

Is a meta-analysis always the strongest evidence?

No. It pools the included studies, so its reliability depends on their quality and on how consistent they are. A meta-analysis of animal studies is still preclinical evidence ^{[13, 14]}.

References

1. Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71. doi:10.1136/bmj.n71. PMID: 33782057
2. Booth A, Clarke M, Dooley G, et al. The nuts and bolts of PROSPERO: an international prospective register of systematic reviews. Syst Rev. 2012;1:2. doi:10.1186/2046-4053-1-2. PMID: 22587842
3. Sterne JAC, Savović J, Page MJ, et al. RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ. 2019;366:l4898. doi:10.1136/bmj.l4898. PMID: 31462531
4. Hooijmans CR, Rovers MM, de Vries RB, et al. SYRCLE’s risk of bias tool for animal studies. BMC Med Res Methodol. 2014;14:43. doi:10.1186/1471-2288-14-43. PMID: 24667063
5. Higgins JP, Thompson SG. Quantifying heterogeneity in a meta-analysis. Stat Med. 2002;21(11):1539–1558. doi:10.1002/sim.1186. PMID: 12111919
6. Higgins JP, Thompson SG, Deeks JJ, Altman DG. Measuring inconsistency in meta-analyses. BMJ. 2003;327(7414):557–560. doi:10.1136/bmj.327.7414.557. PMID: 12958120
7. Borenstein M, Hedges LV, Higgins JP, Rothstein HR. A basic introduction to fixed-effect and random-effects models for meta-analysis. Res Synth Methods. 2010;1(2):97–111. doi:10.1002/jrsm.12. PMID: 26061376
8. Egger M, Davey Smith G, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test. BMJ. 1997;315(7109):629–634. doi:10.1136/bmj.315.7109.629. PMID: 9310563
9. Deeks JJ, Higgins JPT, Altman DG, McKenzie JE, Veroniki AA. Chapter 10: Analysing data and undertaking meta-analyses. In: Cochrane Handbook for Systematic Reviews of Interventions, version 6.5. Cochrane; 2024. cochrane.org. Accessed 25 October 2026.
10. Higgins JPT, Savović J, Page MJ, Elbers RG, Sterne JAC. Chapter 8: Assessing risk of bias in a randomized trial. In: Cochrane Handbook for Systematic Reviews of Interventions, version 6.5. Cochrane; 2024. cochrane.org. Accessed 25 October 2026.
11. Page MJ, Higgins JPT, Sterne JAC. Chapter 13: Assessing risk of bias due to missing evidence in a meta-analysis. In: Cochrane Handbook for Systematic Reviews of Interventions, version 6.5. Cochrane; 2024. cochrane.org. Accessed 25 October 2026.
12. Balshem H, Helfand M, Schünemann HJ, et al. GRADE guidelines: 3. Rating the quality of evidence. J Clin Epidemiol. 2011;64(4):401–406. doi:10.1016/j.jclinepi.2010.07.015. PMID: 21208779
13. Sena ES, Currie GL, McCann SK, Macleod MR, Howells DW. Systematic reviews and meta-analysis of preclinical studies: why perform them and how to appraise them critically. J Cereb Blood Flow Metab. 2014;34(5):737–742. doi:10.1038/jcbfm.2014.28. PMID: 24549183
14. Murad MH, Asi N, Alsawas M, Alahdab F. New evidence pyramid. Evid Based Med. 2016;21(4):125–127. doi:10.1136/ebmed-2016-110401. PMID: 27339128
15. IntHout J, Ioannidis JP, Rovers MM, Goeman JJ. Plea for routinely presenting prediction intervals in meta-analysis. BMJ Open. 2016;6(7):e010247. doi:10.1136/bmjopen-2015-010247. PMID: 27406637
16. GRADE Working Group. About GRADE. gradeworkinggroup.org. Accessed 25 October 2026.
17. Vesterinen HM, Sena ES, Egan KJ, et al. Meta-analysis of data from animal studies: a practical guide. J Neurosci Methods. 2014;221:92–102. doi:10.1016/j.jneumeth.2013.09.010. PMID: 24099992
18. University of Edinburgh. CAMARADES. clinical-brain-sciences.ed.ac.uk. Accessed 25 October 2026.
19. Ioannidis JP. The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses. Milbank Q. 2016;94(3):485–514. doi:10.1111/1468-0009.12210. PMID: 27620683
20. Therapeutic Goods Administration. Australian Register of Therapeutic Goods (ARTG) search. tga.gov.au/resources/artg. Accessed 25 October 2026.
21. Therapeutic Goods Administration. Understanding your responsibilities when importing, compounding and supplying unapproved peptide products. Safety advisory, 13 April 2026. tga.gov.au. Accessed 25 October 2026.