Flaws in the application and interpretation of statistical analyses in systematic reviews of therapeutic interventions were common: a cross-sectional analysis

  • Matthew J. Page
    Correspondence
    Corresponding author. Tel.: +61 9903 0248.
    Affiliations
    School of Public Health and Preventive Medicine, Monash University, 553 St Kilda Road, Melbourne, Victoria 3004, Australia
    Search for articles by this author
  • Douglas G. Altman
    Affiliations
    UK EQUATOR Centre, Centre for Statistics in Medicine, NDORMS, University of Oxford, Oxford OX3 7LD, United Kingdom
    Search for articles by this author
  • Joanne E. McKenzie
    Affiliations
    School of Public Health and Preventive Medicine, Monash University, 553 St Kilda Road, Melbourne, Victoria 3004, Australia
    Search for articles by this author
  • Larissa Shamseer
    Affiliations
    Centre for Journalology, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, Ontario, K1H 8L6, Canada

    School of Epidemiology, Public Health and Preventive Medicine, Faculty of Medicine, University of Ottawa, Ottawa, Ontario, K1H 8M5, Canada
    Search for articles by this author
  • Nadera Ahmadzai
    Affiliations
    Knowledge Synthesis Group, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, Ontario, K1H 8L6, Canada
    Search for articles by this author
  • Dianna Wolfe
    Affiliations
    Knowledge Synthesis Group, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, Ontario, K1H 8L6, Canada
    Search for articles by this author
  • Fatemeh Yazdi
    Affiliations
    Knowledge Synthesis Group, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, Ontario, K1H 8L6, Canada
    Search for articles by this author
  • Ferrán Catalá-López
    Affiliations
    Knowledge Synthesis Group, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, Ontario, K1H 8L6, Canada

    Department of Medicine, University of Valencia/INCLIVA Health Research Institute and CIBERSAM, Valencia, 46010, Spain
    Search for articles by this author
  • Andrea C. Tricco
    Affiliations
    Knowledge Translation Program, Li Ka Shing Knowledge Institute, St Michael's Hospital, Toronto, Ontario, M5B 1W8, Canada

    Epidemiology Division, Dalla Lana School of Public Health, University of Toronto, Toronto, Ontario, M5T 3M7, Canada
    Search for articles by this author
  • David Moher
    Affiliations
    Centre for Journalology, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, Ontario, K1H 8L6, Canada

    School of Epidemiology, Public Health and Preventive Medicine, Faculty of Medicine, University of Ottawa, Ottawa, Ontario, K1H 8M5, Canada
    Search for articles by this author
Published:December 01, 2017DOI:https://doi.org/10.1016/j.jclinepi.2017.11.022

      Abstract

      Objectives

      The objective of the study was to investigate the application and interpretation of statistical analyses in a cross-section of systematic reviews (SRs) of therapeutic interventions, without restriction by journal, clinical condition, or specialty.

      Study Design and Setting

      We evaluated a random sample of SRs assembled previously, which were indexed in MEDLINE® during February 2014, focused on a treatment or prevention question, and reported at least one meta-analysis. The reported statistical methods used in each SR were extracted from articles and online appendices by one author, with a 20% random sample extracted in duplicate.

      Results

      We evaluated 110 SRs; 78/110 (71%) were non-Cochrane SRs and 55/110 (50%) investigated a pharmacological intervention. The SRs presented a median of 13 (interquartile range: 5–27) meta-analytic effects. When considering the index (primary or first reported) meta-analysis of each SR, just over half (62/110 [56%]) used the random-effects model, but few (5/62 [8%]) interpreted the meta-analytic effect correctly (as the average of the intervention effects across all studies). A statistical test for funnel plot asymmetry was reported in 17/110 (15%) SRs; however, in only 4/17 (24%) did the test include the recommended number of at least 10 studies of varying size. Subgroup analyses accompanied 42/110 (38%) index meta-analyses, but findings were not interpreted with respect to a test for interaction in 29/42 (69%) cases, and the issue of potential confounding in the subgroup analyses was not raised in any SR.

      Conclusions

      There is scope for improvement in the application and interpretation of statistical analyses in SRs of therapeutic interventions. The involvement of statisticians on the SR team and establishment of partnerships between researchers with specialist expertise in SR methods and journal editors may help overcome these shortcomings.

      Keywords

      To read this article in full you will need to make a payment

      References

        • Deeks J.J.
        • Higgins J.P.T.
        • Altman D.G.
        Analysing data and undertaking meta-analyses.
        in: Higgins J.P.T. Green S. Cochrane handbook for systematic reviews of interventions version 5.1.0. The Cochrane Collaboration, 2011 (Available at: www.cochrane-handbook.org. Accessed July 1, 2017)
        • Ioannidis J.P.
        • Patsopoulos N.A.
        • Rothstein H.R.
        Reasons or excuses for avoiding meta-analysis in forest plots.
        BMJ. 2008; 336: 1413-1415
        • Murad M.H.
        • Montori V.M.
        • Ioannidis J.P.
        • Jaeschke R.
        • Devereaux P.J.
        • Prasad K.
        • et al.
        How to read a systematic review and meta-analysis and apply the results to patient care: users' guides to the medical literature.
        JAMA. 2014; 312: 171-179
        • McKenzie J.E.
        • Beller E.M.
        • Forbes A.B.
        Introduction to systematic reviews and meta-analysis.
        Respirology. 2016; 21: 626-637
        • Fisher D.J.
        • Carpenter J.R.
        • Morris T.P.
        • Freeman S.C.
        • Tierney J.F.
        Meta-analytical methods to identify who benefits most from treatments: daft, deluded, or deft approach?.
        BMJ. 2017; 356: j573
        • Riley R.D.
        • Gates S.
        • Neilson J.
        • Alfirevic Z.
        Statistical methods can be improved within Cochrane pregnancy and childbirth reviews.
        J Clin Epidemiol. 2011; 64: 608-618
        • Donegan S.
        • Williams L.
        • Dias S.
        • Tudur-Smith C.
        • Welton N.
        Exploring treatment by covariate interactions using subgroup analysis and meta-regression in cochrane reviews: a review of recent practice.
        PLoS One. 2015; 10: e0128804
        • Koletsi D.
        • Valla K.
        • Fleming P.S.
        • Chaimani A.
        • Pandis N.
        Assessment of publication bias required improvement in oral health systematic reviews.
        J Clin Epidemiol. 2016; 76: 118-124
        • Atakpo P.
        • Vassar M.
        Publication bias in dermatology systematic reviews and meta-analyses.
        J Dermatol Sci. 2016; 82: 69-74
        • Hedin R.J.
        • Umberham B.A.
        • Detweiler B.N.
        • Kollmorgen L.
        • Vassar M.
        Publication bias and nonreporting found in majority of systematic reviews and meta-analyses in anesthesiology journals.
        Anesth Analg. 2016; 123: 1018-1025
        • Herrmann D.
        • Sinnett P.
        • Holmes J.
        • Khan S.
        • Koller C.
        • Vassar M.
        Statistical controversies in clinical research: publication bias evaluations are not routinely conducted in clinical oncology systematic reviews.
        Ann Oncol. 2017; 28: 931-937
        • Richardson M.
        • Garner P.
        • Donegan S.
        Cluster randomised trials in cochrane reviews: evaluation of methodological and reporting practice.
        PLoS One. 2016; 11: e0151818
        • Nolan S.J.
        • Hambleton I.
        • Dwan K.
        The use and reporting of the cross-over study design in clinical trials and systematic reviews: a systematic assessment.
        PLoS One. 2016; 11: e0159014
        • Page M.J.
        • Shamseer L.
        • Altman D.G.
        • Tetzlaff J.
        • Sampson M.
        • Tricco A.C.
        • et al.
        Epidemiology and reporting characteristics of systematic reviews of biomedical research: a cross-sectional study.
        PLoS Med. 2016; 13: e1002028
        • Moher D.
        • Shamseer L.
        • Clarke M.
        • Ghersi D.
        • Liberati A.
        • Petticrew M.
        • et al.
        Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement.
        Syst Rev. 2015; 4: 1
        • Shamseer L.
        • Moher D.
        • Clarke M.
        • Ghersi D.
        • Liberati A.
        • Petticrew M.
        • et al.
        Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation.
        BMJ. 2015; 349: g7647
        • Moher D.
        • Tetzlaff J.
        • Tricco A.C.
        • Sampson M.
        • Altman D.G.
        Epidemiology and reporting characteristics of systematic reviews.
        Plos Med. 2007; 4: e78
        • Shemilt I.
        • Khan N.
        • Park S.
        • Thomas J.
        Use of cost-effectiveness analysis to compare the efficiency of study identification methods in systematic reviews.
        Syst Rev. 2016; 5: 140
        • Higgins J.P.T.
        • Lasserson T.
        • Chandler J.
        • Tovey D.
        • Churchill R.
        Methodological expectations of Cochrane intervention reviews.
        Cochrane, London2016
        • Schriger D.L.
        • Altman D.G.
        • Vetter J.A.
        • Heafner T.
        • Moher D.
        Forest plots in reports of systematic reviews: a cross-sectional study reviewing current practice.
        Int J Epidemiol. 2010; 39: 421-429
        • StataCorp
        Stata statistical software: release 14.
        StataCorp LP, College Station, TX2015
        • The Nordic Cochrane Centre (The Cochrane Collaboration)
        Review Manager (RevMan). 5.1.
        The Nordic Cochrane Centre, The Cochrane Collaboration, Copenhagen2011
        • R Development Core Team
        R: a language and environment for statistical computing.
        R Foundation for Statistical Computing, Vienna, Austria2012
        • Riley R.D.
        • Higgins J.P.
        • Deeks J.J.
        Interpretation of random effects meta-analyses.
        BMJ. 2011; 342: d549
        • Higgins J.P.
        • Thompson S.G.
        • Deeks J.J.
        • Altman D.G.
        Measuring inconsistency in meta-analyses.
        BMJ. 2003; 327: 557-560
      1. (AHRQ Publication No. 10(14)-EHC063-EF)Methods guide for effectiveness and comparative effectiveness reviews. Agency for Healthcare Research and Quality, Rockville, MD2014 (Chapters available at: www.effectivehealthcare.ahrq.gov. Accessed July 1, 2017)
        • IOM (Institute of Medicine)
        Finding what works in health care: standards for systematic reviews.
        The National Academies Press, Washington, DC2011
        • DerSimonian R.
        • Laird N.
        Meta-analysis in clinical trials.
        Control Clin Trials. 1986; 7: 177-188
        • Sidik K.
        • Jonkman J.N.
        Simple heterogeneity variance estimation for meta-analysis.
        J R Stat Soc Ser C Appl Stat. 2005; 54: 367-384
        • Peters J.L.
        • Sutton A.J.
        • Jones D.R.
        • Abrams K.R.
        • Rushton L.
        Contour-enhanced meta-analysis funnel plots help distinguish publication bias from other causes of asymmetry.
        J Clin Epidemiol. 2008; 61: 991-996
        • Egger M.
        • Davey Smith G.
        • Schneider M.
        • Minder C.
        Bias in meta-analysis detected by a simple, graphical test.
        BMJ. 1997; 315: 629-634
        • Sterne J.A.
        • Sutton A.J.
        • Ioannidis J.P.
        • Terrin N.
        • Jones D.R.
        • Lau J.
        • et al.
        Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials.
        BMJ. 2011; 343: d4002
        • Borenstein M.
        • Hedges L.V.
        • Higgins J.P.T.
        • Rothstein H.R.
        Introduction to meta-analysis.
        John Wiley & Sons, Ltd, West Sussex, UK2009
        • Altman D.G.
        • Bland J.M.
        Interaction revisited: the difference between two estimates.
        BMJ. 2003; 326: 219
        • Ioannidis J.P.
        The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses.
        Milbank Q. 2016; 94: 485-514
        • Petropoulou M.
        • Nikolakopoulou A.
        • Veroniki A.A.
        • Rios P.
        • Vafaei A.
        • Zarin W.
        • et al.
        Bibliographic study showed improving statistical methodology of network meta-analyses published between 1999 and 2015.
        J Clin Epidemiol. 2017; 82: 20-28
        • Hutton B.
        • Wolfe D.
        • Moher D.
        • Shamseer L.
        Reporting guidance considerations from a statistical perspective: overview of tools to enhance the rigour of reporting of randomised trials and systematic reviews.
        Evid Based Ment Health. 2017; 20: 46-52
        • Moher D.
        • Liberati A.
        • Tetzlaff J.
        • Altman D.G.
        • The PRISMA Group
        Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement.
        Plos Med. 2009; 6: e1000097
        • Page M.J.
        • Moher D.
        Mass production of systematic reviews and meta-analyses: an exercise in mega-silliness?.
        Milbank Q. 2016; 94: 515-519
        • Ioannidis J.P.A.
        • Greenland S.
        • Hlatky M.A.
        • Khoury M.J.
        • Macleod M.R.
        • Moher D.
        • et al.
        Increasing value and reducing waste in research design, conduct, and analysis.
        Lancet. 2014; 383: 166-175
        • Moher D.
        • Glasziou P.
        • Chalmers I.
        • Nasser M.
        • Bossuyt P.M.
        • Korevaar D.A.
        • et al.
        Increasing value and reducing waste in biomedical research: who's listening?.
        Lancet. 2016; 387: 1573-1586
        • IntHout J.
        • Ioannidis J.P.
        • Rovers M.M.
        • Goeman J.J.
        Plea for routinely presenting prediction intervals in meta-analysis.
        BMJ Open. 2016; 6: e010247