Journal of Clinical Epidemiology
Volume 61, Issue 9 , Pages 857-865 , September 2008

Attention should be given to multiplicity issues in systematic reviews

  • Ralf Bender

      Affiliations

    • Institute for Quality and Efficiency in Health Care, Dillenburger Street 27, Cologne D-51105, Germany
    • Faculty of Medicine, University of Cologne, Joseph-Stelzmann-Street 20, Cologne D-50931, Germany
    • Corresponding Author InformationCorresponding author. Department of Medical Biometry, Institute for Quality and Efficiency in Health Care, Dillenburger Street 27, D–51105 Cologne, Germany. Tel.: +49-221-35685-451; fax: +49-221-35685-891.
    • ,
  • Catey Bunce

      Affiliations

    • Research & Development, Moorfields Eye Hospital, City Road, London EC1V 2PD, UK
    • ,
  • Mike Clarke

      Affiliations

    • UK Cochrane Centre, Summertown Pavilion, Middle Way, Oxford OX2 7LG, UK
    • ,
  • Simon Gates

      Affiliations

    • Warwick Clinical Trials Unit, Warwick Medical School, Coventry CV4 7AL, UK
    • ,
  • Stefan Lange

      Affiliations

    • Institute for Quality and Efficiency in Health Care, Dillenburger Street 27, Cologne D-51105, Germany
    • ,
  • Nathan L. Pace

      Affiliations

    • Department of Anesthesiology, University of Utah, 30 North 1900 East, Salt Lake City, Utah 84132, USA
    • ,
  • Kristian Thorlund

      Affiliations

    • Copenhagen Trial Unit, Copenhagen University Hospital, Blegdamsvej 9, Copenhagen DK-2100, Denmark

,Accepted 7 March 2008.

References 

  1. Bauer P. Multiple testing in clinical trials. Stat Med. 1991;10:871–890
  2. Ottenbacher KJ. Quantitative evaluation of multiplicity in epidemiology and public health research. Am J Epidemiol. 1998;147:615–619
  3. Biester K, Lange S. The multiplicity problem in systematic reviews [Abstract]. XIII Cochrane Colloquium, Melbourne, 22–26 October 2005. Program and Abstracts; 2005: 153. Available at http://www.cochrane.org/colloquia/abstracts/melbourne/P-155.htm. Accessed May 13, 2008.
  4. Bender R, Lange S. Adjusting for multiple testing—when and how?. J Clin Epidemiol. 2001;54:343–349
  5. Cook RJ, Dunnett CW. Multiple comparisons. In:  Armitage P,  Colton T editor. Encyclopedia of biostatistics. Chichester: Wiley; 2005;p. 3383–3393
  6. Dmitrienko A, Hsu JC. Multiple testing in clinical trials. In:  Kotz S,  Read CB,  Balakrishnan N,  Vidakovic B editor. Encyclopedia of statistical sciences. Hoboken, New Jersey: Wiley; 2006;p. 5111–5117
  7. Chen T, Hoppe FM. Simultaneous confidence intervals. In:  Armitage P,  Colton T editor. Encyclopedia of biostatistics. Chichester: Wiley; 2005;p. 4953–4955
  8. Bland JM, Altman DG. Multiple significance tests: the Bonferroni method. BMJ. 1995;310:170
  9. Rabiu M, Alhassan MB, Agbabiaka IO. Interventions for Mooren's ulcer (Protocol). Cochrane Database Syst Rev. 2006;3:CD006131
  10. Rolim de Moura C, Paranhos A. Laser trabeculoplasty for open angle glaucoma (Protocol). Cochrane Database Syst Rev. 2002;4:CD003919
  11. van Houwelingen HC, Arends LR, Stijnen T. Advanced methods in meta-analysis: multivariate approach and meta-regression. Stat Med. 2002;21:589–624
  12. Arends LR, Voko Z, Stijnen T. Combining multiple outcome measures in a meta-analysis: an application. Stat Med. 2003;22:1335–1353
  13. Riley RD, Abrams KR, Sutton AJ, Lambert PC, Thompson JR. Bivariate random-effects meta-analysis and the estimation of between-study correlation. BMC Med Res Methodol. 2007;7:3
  14. Ishak KJ, Platt RW, Joseph L, Hanley JA. Impact of approximating or ignoring within-study covariances in multivariate meta-analyses. Stat Med. 2008;27:670–686
  15. Ong CK, Lirk P, Seymour RA, Jenkins BJ. The efficacy of preemptive analgesia for acute postoperative pain management: a meta-analysis. Anesth Analg. 2005;100:757–773
  16. Hsu JC. Multiple comparisons: Theory and methods. Boca Raton, FL: Chapman & Hall; 1996;
  17. Hasselblad V. Meta-analysis of multitreatment studies. Med Decis Making. 1998;18:37–43
  18. Riley RD, Abrams KR, Lambert PC, Sutton AJ, Thompson JR. An evaluation of bivariate random-effects meta-analysis for the joint synthesis of two correlated outcomes. Stat Med. 2007;26:78–97
  19. Bouza C, Lopez T, Magro A, Navalpotro L, Amate JM. Systematic review and meta-analysis of transurethral needle ablation in symptomatic benign prostatic hyperplasia. BMC Urol. 2006;6:14
  20. Parmar MK, Torri V, Stewart L. Extracting summary statistics to perform meta-analyses of the published literature for survival endpoints. Stat Med. 1998;17:2815–2834
  21. Dear KBG. Iterative generalized least squares for meta-analysis of survival data at multiple times. Biometrics. 1994;50:989–1002
  22. Deeks JJ, Altman DG. Effect measures for meta-analysis of trials with binary outcomes. In:  Egger M,  Davey Smith G,  Altman DG editor. Systematic reviews in health care: Meta-analysis in context. London: BMJ Books; 2001;p. 313–335
  23. Clarke M, Hopewell S, Juszczak E, Eisinga A, Kjeldstrom M. Compression stockings for preventing deep vein thrombosis in airline passengers. Cochrane Database Syst Rev. 2006;2:CD004002
  24. Sweeting MJ, Sutton AJ, Lambert PC. What to add to nothing? Use and avoidance of continuity corrections in meta-analysis of sparse data. Stat Med. 2004;23:1351–1375
  25. Davey Smith G, Egger M. Going beyond the grand mean: subgroup analysis in meta-analysis of randomised trials. In:  Egger M,  Davey Smith G,  Altman DG editor. Systematic reviews in health care: Meta-analysis in context. London: BMJ Books; 2001;p. 143–156
  26. Pocock SJ, Assmann SE, Enos LE, Kasten LE. Subgroup analysis, covariate adjustment and baseline comparisons in clinical trial reporting: current practice and problems. Stat Med. 2002;21:2917–2930
  27. Austin PC, Mamdani MM, Juurlink DN, Hux JE. Testing multiple statistical hypotheses resulted in spurious associations: a study of astrological signs and health. J Clin Epidemiol. 2006;59:964–969
  28. Roberts D, Dalziel S. Antenatal corticosteroids for accelerating fetal lung maturation for women at risk of preterm birth. Cochrane Database Syst Rev. 2006;3:CD004454
  29. Brookes ST, Whitley E, Peters TJ, Mulheran PA, Egger M, Davey Smith G. Subgroup analyses in randomised controlled trials: quantifying the risks of false-positives and false-negatives. Health Technol Assess. 2001;5:1–56
  30. In:  Higgins JPT,  Green S editor. Cochrane handbook for systematic reviews of interventions 4.2.6 [updated September 2006]. The Cochrane Library, Issue 4, 2006. Chichester, UK: Wiley; 2006;
  31. Lau J, Antman EM, Jimenez-Silva J, Kupelnick B, Mosteller F, Chalmers TC. Cumulative meta-analysis of therapeutic trials for myocardial infarction. N Engl J Med. 1992;327:248–254
  32. Lau J, Schmid CH, Chalmers TC. Cumulative meta-analysis of clinical trials builds evidence for exemplary medical care. J Clin Epidemiol. 1995;48:45–57
  33. McPherson K. Statistics: The problem of examining accumulating data more than once. N Engl J Med. 1974;290:501–502
  34. O'Brien PC. Data and safety monitoring. In:  Armitage P,  Colton T editor. Encyclopedia of biostatistics. Chichester: Wiley; 2005;p. 1362–1371
  35. Pogue JM, Yusuf S. Cumulating evidence from randomized trials: utilizing sequential monitoring boundaries for cumulative meta-analysis. Control Clin Trials. 1997;18:580–593
  36. Devereaux PJ, Beattie WS, Choi PT, Badner NH, Guyatt GH, Villar JC, et al. How strong is the evidence for the use of perioperative beta blockers in non-cardiac surgery? Systematic review and meta-analysis of randomised controlled trials. BMJ. 2005;331:313–321
  37. Wetterslev J, Thorlund K, Brok J, Gluud C. Trial sequential analysis may establish when firm evidence is reached in cumulative meta-analysis. J Clin Epidemiol. 2008;61:64–75
  38. Whitehead A. A prospectively planned cumulative meta-analysis applied to a series of concurrent clinical trials. Stat Med. 1997;16:2901–2913
  39. Lan KKG, Hu M, Cappalleri JC. Applying the law of iterated logarithm to cumulative meta-analysis of a continuous endpoint. Statistica Sinica. 2003;13:1135–1145
  40. Hu M, Cappelleri JC, Lan KK. Applying the law of iterated logarithm to control type I error in cumulative meta-analysis of binary outcomes. Clin Trials. 2007;4:329–340
  41. Jüni P, Nartey L, Reichenbach S, Sterchi R, Dieppe PA, Egger M. Risk of cardiovascular events and rofecoxib: cumulative meta-analysis. Lancet. 2004;364:2021–2029
  42. Ioannidis JPA, Contopoulos-Ioannidis DG, Lau J. Recursive cumulative meta-analysis: a diagnostic for the evolution of total randomized evidence from group and individual patient data. J Clin Epidemiol. 1999;52:281–291

PII: S0895-4356(08)00093-0

doi: 10.1016/j.jclinepi.2008.03.004

Journal of Clinical Epidemiology
Volume 61, Issue 9 , Pages 857-865 , September 2008

Article Tools