Attention should be given to multiplicity issues in systematic reviews



      The objective of this paper is to describe the problem of multiple comparisons in systematic reviews and to provide some guidelines on how to deal with it in practice.

      Study Design and Setting

      We describe common reasons for multiplicity in systematic reviews, and present some examples. We provide guidance on how to deal with multiplicity when it is unavoidable.


      We identified six common reasons for multiplicity in systematic reviews: multiple outcomes, multiple groups, multiple time points, multiple effect measures, subgroup analyses, and multiple looks at accumulating data. The existing methods to deal with multiplicity in single trials can not always be applied in systematic reviews.


      There is no simple and completely satisfactory solution to the problem of multiple comparisons in systematic reviews. More research is required to develop multiple comparison procedures for use in systematic reviews. Authors and consumers of systematic reviews should give serious attention to multiplicity in systematic reviews when presenting, interpreting and using the results of these reports.


      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to Journal of Clinical Epidemiology
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Bauer P.
        Multiple testing in clinical trials.
        Stat Med. 1991; 10: 871-890
        • Ottenbacher K.J.
        Quantitative evaluation of multiplicity in epidemiology and public health research.
        Am J Epidemiol. 1998; 147: 615-619
      1. Biester K, Lange S. The multiplicity problem in systematic reviews [Abstract]. XIII Cochrane Colloquium, Melbourne, 22–26 October 2005. Program and Abstracts; 2005: 153. Available at Accessed May 13, 2008.

        • Bender R.
        • Lange S.
        Adjusting for multiple testing—when and how?.
        J Clin Epidemiol. 2001; 54: 343-349
        • Cook R.J.
        • Dunnett C.W.
        Multiple comparisons.
        in: Armitage P. Colton T. Encyclopedia of biostatistics. Wiley, Chichester2005: 3383-3393
        • Dmitrienko A.
        • Hsu J.C.
        Multiple testing in clinical trials.
        in: Kotz S. Read C.B. Balakrishnan N. Vidakovic B. Encyclopedia of statistical sciences. Wiley, Hoboken, New Jersey2006: 5111-5117
        • Chen T.
        • Hoppe F.M.
        Simultaneous confidence intervals.
        in: Armitage P. Colton T. Encyclopedia of biostatistics. Wiley, Chichester2005: 4953-4955
        • Bland J.M.
        • Altman D.G.
        Multiple significance tests: the Bonferroni method.
        BMJ. 1995; 310: 170
        • Rabiu M.
        • Alhassan M.B.
        • Agbabiaka I.O.
        Interventions for Mooren's ulcer (Protocol).
        Cochrane Database Syst Rev. 2006; 3 (CD006131)
        • Rolim de Moura C.
        • Paranhos Jr., A.
        Laser trabeculoplasty for open angle glaucoma (Protocol).
        Cochrane Database Syst Rev. 2002; 4 (CD003919)
        • van Houwelingen H.C.
        • Arends L.R.
        • Stijnen T.
        Advanced methods in meta-analysis: multivariate approach and meta-regression.
        Stat Med. 2002; 21: 589-624
        • Arends L.R.
        • Voko Z.
        • Stijnen T.
        Combining multiple outcome measures in a meta-analysis: an application.
        Stat Med. 2003; 22: 1335-1353
        • Riley R.D.
        • Abrams K.R.
        • Sutton A.J.
        • Lambert P.C.
        • Thompson J.R.
        Bivariate random-effects meta-analysis and the estimation of between-study correlation.
        BMC Med Res Methodol. 2007; 7: 3
        • Ishak K.J.
        • Platt R.W.
        • Joseph L.
        • Hanley J.A.
        Impact of approximating or ignoring within-study covariances in multivariate meta-analyses.
        Stat Med. 2008; 27: 670-686
        • Ong C.K.
        • Lirk P.
        • Seymour R.A.
        • Jenkins B.J.
        The efficacy of preemptive analgesia for acute postoperative pain management: a meta-analysis.
        Anesth Analg. 2005; 100: 757-773
        • Hsu J.C.
        Multiple comparisons: Theory and methods.
        Chapman & Hall, Boca Raton, FL1996
        • Hasselblad V.
        Meta-analysis of multitreatment studies.
        Med Decis Making. 1998; 18: 37-43
        • Riley R.D.
        • Abrams K.R.
        • Lambert P.C.
        • Sutton A.J.
        • Thompson J.R.
        An evaluation of bivariate random-effects meta-analysis for the joint synthesis of two correlated outcomes.
        Stat Med. 2007; 26: 78-97
        • Bouza C.
        • Lopez T.
        • Magro A.
        • Navalpotro L.
        • Amate J.M.
        Systematic review and meta-analysis of transurethral needle ablation in symptomatic benign prostatic hyperplasia.
        BMC Urol. 2006; 6: 14
        • Parmar M.K.
        • Torri V.
        • Stewart L.
        Extracting summary statistics to perform meta-analyses of the published literature for survival endpoints.
        Stat Med. 1998; 17: 2815-2834
        • Dear K.B.G.
        Iterative generalized least squares for meta-analysis of survival data at multiple times.
        Biometrics. 1994; 50: 989-1002
        • Deeks J.J.
        • Altman D.G.
        Effect measures for meta-analysis of trials with binary outcomes.
        in: Egger M. Davey Smith G. Altman D.G. Systematic reviews in health care: Meta-analysis in context. BMJ Books, London2001: 313-335
        • Clarke M.
        • Hopewell S.
        • Juszczak E.
        • Eisinga A.
        • Kjeldstrom M.
        Compression stockings for preventing deep vein thrombosis in airline passengers.
        Cochrane Database Syst Rev. 2006; 2 (CD004002)
        • Sweeting M.J.
        • Sutton A.J.
        • Lambert P.C.
        What to add to nothing? Use and avoidance of continuity corrections in meta-analysis of sparse data.
        Stat Med. 2004; 23: 1351-1375
        • Davey Smith G.
        • Egger M.
        Going beyond the grand mean: subgroup analysis in meta-analysis of randomised trials.
        in: Egger M. Davey Smith G. Altman D.G. Systematic reviews in health care: Meta-analysis in context. BMJ Books, London2001: 143-156
        • Pocock S.J.
        • Assmann S.E.
        • Enos L.E.
        • Kasten L.E.
        Subgroup analysis, covariate adjustment and baseline comparisons in clinical trial reporting: current practice and problems.
        Stat Med. 2002; 21: 2917-2930
        • Austin P.C.
        • Mamdani M.M.
        • Juurlink D.N.
        • Hux J.E.
        Testing multiple statistical hypotheses resulted in spurious associations: a study of astrological signs and health.
        J Clin Epidemiol. 2006; 59: 964-969
        • Roberts D.
        • Dalziel S.
        Antenatal corticosteroids for accelerating fetal lung maturation for women at risk of preterm birth.
        Cochrane Database Syst Rev. 2006; 3 (CD004454)
        • Brookes S.T.
        • Whitley E.
        • Peters T.J.
        • Mulheran P.A.
        • Egger M.
        • Davey Smith G.
        Subgroup analyses in randomised controlled trials: quantifying the risks of false-positives and false-negatives.
        Health Technol Assess. 2001; 5: 1-56
      2. Higgins J.P.T. Green S. Cochrane handbook for systematic reviews of interventions 4.2.6 [updated September 2006]. The Cochrane Library, Issue 4, 2006. Wiley, Chichester, UK2006
        • Lau J.
        • Antman E.M.
        • Jimenez-Silva J.
        • Kupelnick B.
        • Mosteller F.
        • Chalmers T.C.
        Cumulative meta-analysis of therapeutic trials for myocardial infarction.
        N Engl J Med. 1992; 327: 248-254
        • Lau J.
        • Schmid C.H.
        • Chalmers T.C.
        Cumulative meta-analysis of clinical trials builds evidence for exemplary medical care.
        J Clin Epidemiol. 1995; 48: 45-57
        • McPherson K.
        Statistics: The problem of examining accumulating data more than once.
        N Engl J Med. 1974; 290: 501-502
        • O'Brien P.C.
        Data and safety monitoring.
        in: Armitage P. Colton T. Encyclopedia of biostatistics. Wiley, Chichester2005: 1362-1371
        • Pogue J.M.
        • Yusuf S.
        Cumulating evidence from randomized trials: utilizing sequential monitoring boundaries for cumulative meta-analysis.
        Control Clin Trials. 1997; 18: 580-593
        • Devereaux P.J.
        • Beattie W.S.
        • Choi P.T.
        • Badner N.H.
        • Guyatt G.H.
        • Villar J.C.
        • et al.
        How strong is the evidence for the use of perioperative beta blockers in non-cardiac surgery? Systematic review and meta-analysis of randomised controlled trials.
        BMJ. 2005; 331: 313-321
        • Wetterslev J.
        • Thorlund K.
        • Brok J.
        • Gluud C.
        Trial sequential analysis may establish when firm evidence is reached in cumulative meta-analysis.
        J Clin Epidemiol. 2008; 61: 64-75
        • Whitehead A.
        A prospectively planned cumulative meta-analysis applied to a series of concurrent clinical trials.
        Stat Med. 1997; 16: 2901-2913
        • Lan K.K.G.
        • Hu M.
        • Cappalleri J.C.
        Applying the law of iterated logarithm to cumulative meta-analysis of a continuous endpoint.
        Statistica Sinica. 2003; 13: 1135-1145
        • Hu M.
        • Cappelleri J.C.
        • Lan K.K.
        Applying the law of iterated logarithm to control type I error in cumulative meta-analysis of binary outcomes.
        Clin Trials. 2007; 4: 329-340
        • Jüni P.
        • Nartey L.
        • Reichenbach S.
        • Sterchi R.
        • Dieppe P.A.
        • Egger M.
        Risk of cardiovascular events and rofecoxib: cumulative meta-analysis.
        Lancet. 2004; 364: 2021-2029
        • Ioannidis J.P.A.
        • Contopoulos-Ioannidis D.G.
        • Lau J.
        Recursive cumulative meta-analysis: a diagnostic for the evolution of total randomized evidence from group and individual patient data.
        J Clin Epidemiol. 1999; 52: 281-291