Advertisement

Reproducible research practices are underused in systematic reviews of biomedical interventions

  • Matthew J. Page
    Correspondence
    Corresponding author. Tel.: +61 3 9903 0248.
    Affiliations
    School of Public Health and Preventive Medicine, Monash University, 553 St Kilda Road, Melbourne, Victoria 3004, Australia
    Search for articles by this author
  • Douglas G. Altman
    Affiliations
    UK EQUATOR Centre, Centre for Statistics in Medicine, NDORMS, University of Oxford, Windmill Road, Oxford OX3 7LD, UK
    Search for articles by this author
  • Larissa Shamseer
    Affiliations
    Centre for Journalology, Clinical Epidemiology Program, Ottawa Hospital Research Institute, 501 Smyth Road, Ottawa, Ontario K1H 8L6, Canada

    School of Epidemiology, Public Health and Preventive Medicine, Faculty of Medicine, University of Ottawa, 451 Smyth Road, Ottawa, Ontario K1H 8M5, Canada
    Search for articles by this author
  • Joanne E. McKenzie
    Affiliations
    School of Public Health and Preventive Medicine, Monash University, 553 St Kilda Road, Melbourne, Victoria 3004, Australia
    Search for articles by this author
  • Nadera Ahmadzai
    Affiliations
    Knowledge Synthesis Group, Clinical Epidemiology Program, Ottawa Hospital Research Institute, 451 Smyth Road, Ottawa, Ontario K1H 8M5, Canada
    Search for articles by this author
  • Dianna Wolfe
    Affiliations
    Knowledge Synthesis Group, Clinical Epidemiology Program, Ottawa Hospital Research Institute, 451 Smyth Road, Ottawa, Ontario K1H 8M5, Canada
    Search for articles by this author
  • Fatemeh Yazdi
    Affiliations
    Knowledge Synthesis Group, Clinical Epidemiology Program, Ottawa Hospital Research Institute, 451 Smyth Road, Ottawa, Ontario K1H 8M5, Canada
    Search for articles by this author
  • Ferrán Catalá-López
    Affiliations
    Knowledge Synthesis Group, Clinical Epidemiology Program, Ottawa Hospital Research Institute, 451 Smyth Road, Ottawa, Ontario K1H 8M5, Canada

    Department of Medicine, University of Valencia/INCLIVA Health Research Institute and CIBERSAM, Valencia 46010, Spain
    Search for articles by this author
  • Andrea C. Tricco
    Affiliations
    Knowledge Translation Program, Li Ka Shing Knowledge Institute, St Michael's Hospital, Toronto, 30 Bond Street, Ontario M5B 1W8, Canada

    Epidemiology Division, Dalla Lana School of Public Health, University of Toronto, 155 College Street, Toronto, Ontario M5T 3M7, Canada
    Search for articles by this author
  • David Moher
    Affiliations
    Centre for Journalology, Clinical Epidemiology Program, Ottawa Hospital Research Institute, 501 Smyth Road, Ottawa, Ontario K1H 8L6, Canada

    School of Epidemiology, Public Health and Preventive Medicine, Faculty of Medicine, University of Ottawa, 451 Smyth Road, Ottawa, Ontario K1H 8M5, Canada
    Search for articles by this author
Published:November 04, 2017DOI:https://doi.org/10.1016/j.jclinepi.2017.10.017

      Abstract

      Objectives

      To evaluate how often reproducible research practices, which allow others to recreate the findings of studies, given the original data, are used in systematic reviews (SRs) of biomedical research.

      Study Design and Setting

      We evaluated a random sample of SRs indexed in MEDLINE during February 2014, which focused on a therapeutic intervention and reported at least one meta-analysis. Data on reproducible research practices in each SR were extracted using a 26-item form by one author, with a 20% random sample extracted in duplicate. We explored whether the use of reproducible research practices was associated with an SR being a Cochrane review, as well as with the reported use of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement.

      Results

      We evaluated 110 SRs of therapeutic interventions, 78 (71%) of which were non-Cochrane SRs. Across the SRs, there were 2,139 meta-analytic effects (including subgroup meta-analytic effects and sensitivity analyses), 1,551 (73%) of which were reported in sufficient detail to recreate them. Systematic reviewers reported the data needed to recreate all meta-analytic effects in 72 (65%) SRs only. This percentage was higher in Cochrane than in non-Cochrane SRs (30/32 [94%] vs. 42/78 [54%]; risk ratio 1.74, 95% confidence interval 1.39–2.18). Systematic reviewers who reported imputing, algebraically manipulating, or obtaining some data from the study author/sponsor infrequently stated which specific data were handled in this way. Only 33 (30%) SRs mentioned access to data sets and statistical code used to perform analyses.

      Conclusion

      Reproducible research practices are underused in SRs of biomedical interventions. Adoption of such practices facilitates identification of errors and allows the SR data to be reanalyzed.

      Keywords

      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'

      Subscribe:

      Subscribe to Journal of Clinical Epidemiology
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect

      References

        • Iqbal S.A.
        • Wallach J.D.
        • Khoury M.J.
        • Schully S.D.
        • Ioannidis J.P.
        Reproducible research practices and transparency across the biomedical literature.
        PLoS Biol. 2016; 14: e1002333
        • Nosek B.A.
        • Alter G.
        • Banks G.C.
        • Borsboom D.
        • Bowman S.D.
        • Breckler S.J.
        • et al.
        Scientific standards. Promoting an open research culture.
        Science. 2015; 348: 1422-1425
        • Errington T.M.
        • Iorns E.
        • Gunn W.
        • Tan F.E.
        • Lomax J.
        • Nosek B.A.
        An open investigation of the reproducibility of cancer biology research.
        Elife. 2014; 3: e04333
        • Goodman S.N.
        • Fanelli D.
        • Ioannidis J.P.
        What does research reproducibility mean?.
        Sci Transl Med. 2016; 8: 341ps12
        • Lakens D.
        • Hilgard J.
        • Staaks J.
        On the reproducibility of meta-analyses: six practical recommendations.
        BMC Psychol. 2016; 4: 24
        • Elliott J.H.
        • Turner T.
        • Clavisi O.
        • Thomas J.
        • Higgins J.P.
        • Mavergames C.
        • et al.
        Living systematic reviews: an emerging opportunity to narrow the evidence-practice gap.
        PLoS Med. 2014; 11: e1001603
        • Crequit P.
        • Trinquart L.
        • Ravaud P.
        Live cumulative network meta-analysis: protocol for second-line treatments in advanced non-small-cell lung cancer with wild-type or unknown status for epidermal growth factor receptor.
        BMJ Open. 2016; 6: e011841
        • Moher D.
        • Liberati A.
        • Tetzlaff J.
        • Altman D.G.
        • The PRISMA group
        Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement.
        PLoS Med. 2009; 6: e1000097
        • Stevens A.
        • Shamseer L.
        • Weinstein E.
        • Yazdi F.
        • Turner L.
        • Thielman J.
        • et al.
        Relation of completeness of reporting of health research to journals' endorsement of reporting guidelines: systematic review.
        BMJ. 2014; 348: g3804
        • Tam W.W.
        • Lo K.K.
        • Khalechelvam P.
        Endorsement of PRISMA statement and quality of systematic reviews and meta-analyses published in nursing journals: a cross-sectional study.
        BMJ Open. 2017; 7: e013905
        • Peters J.P.
        • Hooft L.
        • Grolman W.
        • Stegeman I.
        Reporting quality of systematic reviews and meta-analyses of otorhinolaryngologic articles based on the PRISMA statement.
        PLoS One. 2015; 10: e0136540
        • Liu Y.
        • Zhang R.
        • Huang J.
        • Zhao X.
        • Liu D.
        • Sun W.
        • et al.
        Reporting quality of systematic reviews/meta-analyses of acupuncture.
        PLoS One. 2014; 9: e113172
        • Tunis A.S.
        • McInnes M.D.
        • Hanna R.
        • Esmail K.
        Association of study quality with completeness of reporting: have completeness of reporting and quality of systematic reviews and meta-analyses in major radiology journals changed since publication of the PRISMA statement?.
        Radiology. 2013; 269: 413-426
        • Gagnier J.J.
        • Kellam P.J.
        Reporting and methodological quality of systematic reviews in the orthopaedic literature.
        J Bone Joint Surg Am. 2013; 95: e771-e777
        • Wolfenden L.
        • Grimshaw J.
        • Williams C.M.
        • Yoong S.L.
        Time to consider sharing data extracted from trials included in systematic reviews.
        Syst Rev. 2016; 5: 185
        • Loder E.
        • Groves T.
        The BMJ requires data sharing on request for all trials.
        BMJ. 2015; 350: h2373
        • Page M.J.
        • Shamseer L.
        • Altman D.G.
        • Tetzlaff J.
        • Sampson M.
        • Tricco A.C.
        • et al.
        Epidemiology and reporting characteristics of systematic reviews of biomedical research: a cross-sectional study.
        PLoS Med. 2016; 13: e1002028
        • Moher D.
        • Shamseer L.
        • Clarke M.
        • Ghersi D.
        • Liberati A.
        • Petticrew M.
        • et al.
        Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement.
        Syst Rev. 2015; 4: 1
        • Shamseer L.
        • Moher D.
        • Clarke M.
        • Ghersi D.
        • Liberati A.
        • Petticrew M.
        • et al.
        Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation.
        BMJ. 2015; 349: g7647
        • Moher D.
        • Tetzlaff J.
        • Tricco A.C.
        • Sampson M.
        • Altman D.G.
        Epidemiology and reporting characteristics of systematic reviews.
        PLoS Med. 2007; 4: e78
        • Liberati A.
        • Altman D.G.
        • Tetzlaff J.
        • Mulrow C.
        • Gotzsche P.C.
        • Ioannidis J.P.
        • et al.
        The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration.
        BMJ. 2009; 339: b2700
        • Higgins J.P.T.
        • Lasserson T.
        • Chandler J.
        • Tovey D.
        • Churchill R.
        Methodological expectations of Cochrane intervention reviews.
        Cochrane, London2016
        • StataCorp
        Stata Statistical Software: Release 14.
        StataCorp LP, College Station, TX2015
        • DerSimonian R.
        • Laird N.
        Meta-analysis in clinical trials.
        Control Clin Trials. 1986; 7: 177-188
        • Sidik K.
        • Jonkman J.N.
        A comparison of heterogeneity variance estimators in combining results of studies.
        Stat Med. 2007; 26: 1964-1981
        • The Nordic Cochrane Centre (The Cochrane Collaboration)
        Review Manager (RevMan). 5.1.
        The Nordic Cochrane Centre, The Cochrane Collaboration, Copenhagen2011
        • Ioannidis J.P.
        The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses.
        Milbank Q. 2016; 94: 485-514
        • Kidwell M.C.
        • Lazarevic L.B.
        • Baranski E.
        • Hardwicke T.E.
        • Piechowski S.
        • Falkenberg L.S.
        • et al.
        Badges to acknowledge open practices: a simple, low-cost, effective method for increasing transparency.
        PLoS Biol. 2016; 14: e1002456
        • McKiernan E.C.
        • Bourne P.E.
        • Brown C.T.
        • Buck S.
        • Kenall A.
        • Lin J.
        • et al.
        How open science helps researchers succeed.
        Elife. 2016; 5: e16800
        • Nosek B.A.
        • Errington T.M.
        Making sense of replications.
        Elife. 2017; 6: e23383
        • Open Science Collaboration
        PSYCHOLOGY. Estimating the reproducibility of psychological science.
        Science. 2015; 349: aac4716
        • Hill J.
        • Hoyt J.
        • van Eijk A.M.
        • D'Mello-Guyett L.
        • Ter Kuile F.O.
        • Steketee R.
        • et al.
        Factors affecting the delivery, access, and use of interventions to prevent malaria in pregnancy in sub-Saharan Africa: a systematic review and meta-analysis.
        PLoS Med. 2013; 10: e1001488
        • Altman D.G.
        • Cates C.
        The need for individual trial results in reports of systematic reviews.
        BMJ. 2001; 323: 776
        • Gotzsche P.C.
        • Hrobjartsson A.
        • Maric K.
        • Tendal B.
        Data extraction errors in meta-analyses that use standardized mean differences.
        JAMA. 2007; 298: 430-437
        • Barnes C.
        • Boutron I.
        • Giraudeau B.
        • Porcher R.
        • Altman D.G.
        • Ravaud P.
        Impact of an online writing aid tool for writing a randomized trial report: the COBWEB (Consort-based WEB tool) randomized controlled trial.
        BMC Med. 2015; 13: 221
        • Shamseer L.
        • Roberts J.
        Disclosure of data and statistical commands should accompany completely reported studies.
        J Clin Epidemiol. 2016; 70: 272-274
        • Ip S.
        • Hadar N.
        • Keefe S.
        • Parkin C.
        • Iovin R.
        • Balk E.M.
        • et al.
        A Web-based archive of systematic review data.
        Syst Rev. 2012; 1: 15
        • Li T.
        • Vedula S.S.
        • Hadar N.
        • Parkin C.
        • Lau J.
        • Dickersin K.
        Innovations in data collection, management, and archiving for systematic reviews.
        Ann Intern Med. 2015; 162: 287-294
        • Moher D.
        • Goodman S.N.
        • Ioannidis J.P.
        Academic criteria for appointment, promotion and rewards in medical research: where's the evidence?.
        Eur J Clin Invest. 2016; 46: 383-385