Abstract
Objectives
To evaluate how often reproducible research practices, which allow others to recreate
the findings of studies, given the original data, are used in systematic reviews (SRs)
of biomedical research.
Study Design and Setting
We evaluated a random sample of SRs indexed in MEDLINE during February 2014, which
focused on a therapeutic intervention and reported at least one meta-analysis. Data
on reproducible research practices in each SR were extracted using a 26-item form
by one author, with a 20% random sample extracted in duplicate. We explored whether
the use of reproducible research practices was associated with an SR being a Cochrane
review, as well as with the reported use of the Preferred Reporting Items for Systematic
Reviews and Meta-Analyses statement.
Results
We evaluated 110 SRs of therapeutic interventions, 78 (71%) of which were non-Cochrane
SRs. Across the SRs, there were 2,139 meta-analytic effects (including subgroup meta-analytic
effects and sensitivity analyses), 1,551 (73%) of which were reported in sufficient
detail to recreate them. Systematic reviewers reported the data needed to recreate
all meta-analytic effects in 72 (65%) SRs only. This percentage was higher in Cochrane
than in non-Cochrane SRs (30/32 [94%] vs. 42/78 [54%]; risk ratio 1.74, 95% confidence
interval 1.39–2.18). Systematic reviewers who reported imputing, algebraically manipulating,
or obtaining some data from the study author/sponsor infrequently stated which specific
data were handled in this way. Only 33 (30%) SRs mentioned access to data sets and
statistical code used to perform analyses.
Conclusion
Reproducible research practices are underused in SRs of biomedical interventions.
Adoption of such practices facilitates identification of errors and allows the SR
data to be reanalyzed.
Keywords
To read this article in full you will need to make a payment
Purchase one-time access:
Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online accessOne-time access price info
- For academic or personal research use, select 'Academic and Personal'
- For corporate R&D use, select 'Corporate R&D Professionals'
Subscribe:
Subscribe to Journal of Clinical EpidemiologyAlready a print subscriber? Claim online access
Already an online subscriber? Sign in
Register: Create an account
Institutional Access: Sign in to ScienceDirect
References
- Reproducible research practices and transparency across the biomedical literature.PLoS Biol. 2016; 14: e1002333
- Scientific standards. Promoting an open research culture.Science. 2015; 348: 1422-1425
- An open investigation of the reproducibility of cancer biology research.Elife. 2014; 3: e04333
- What does research reproducibility mean?.Sci Transl Med. 2016; 8: 341ps12
- On the reproducibility of meta-analyses: six practical recommendations.BMC Psychol. 2016; 4: 24
- Living systematic reviews: an emerging opportunity to narrow the evidence-practice gap.PLoS Med. 2014; 11: e1001603
- Live cumulative network meta-analysis: protocol for second-line treatments in advanced non-small-cell lung cancer with wild-type or unknown status for epidermal growth factor receptor.BMJ Open. 2016; 6: e011841
- Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement.PLoS Med. 2009; 6: e1000097
- Relation of completeness of reporting of health research to journals' endorsement of reporting guidelines: systematic review.BMJ. 2014; 348: g3804
- Endorsement of PRISMA statement and quality of systematic reviews and meta-analyses published in nursing journals: a cross-sectional study.BMJ Open. 2017; 7: e013905
- Reporting quality of systematic reviews and meta-analyses of otorhinolaryngologic articles based on the PRISMA statement.PLoS One. 2015; 10: e0136540
- Reporting quality of systematic reviews/meta-analyses of acupuncture.PLoS One. 2014; 9: e113172
- Association of study quality with completeness of reporting: have completeness of reporting and quality of systematic reviews and meta-analyses in major radiology journals changed since publication of the PRISMA statement?.Radiology. 2013; 269: 413-426
- Reporting and methodological quality of systematic reviews in the orthopaedic literature.J Bone Joint Surg Am. 2013; 95: e771-e777
- Time to consider sharing data extracted from trials included in systematic reviews.Syst Rev. 2016; 5: 185
- The BMJ requires data sharing on request for all trials.BMJ. 2015; 350: h2373
- Epidemiology and reporting characteristics of systematic reviews of biomedical research: a cross-sectional study.PLoS Med. 2016; 13: e1002028
- Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement.Syst Rev. 2015; 4: 1
- Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation.BMJ. 2015; 349: g7647
- Epidemiology and reporting characteristics of systematic reviews.PLoS Med. 2007; 4: e78
- The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration.BMJ. 2009; 339: b2700
- Methodological expectations of Cochrane intervention reviews.Cochrane, London2016
- Stata Statistical Software: Release 14.StataCorp LP, College Station, TX2015
- Meta-analysis in clinical trials.Control Clin Trials. 1986; 7: 177-188
- A comparison of heterogeneity variance estimators in combining results of studies.Stat Med. 2007; 26: 1964-1981
- Review Manager (RevMan). 5.1.The Nordic Cochrane Centre, The Cochrane Collaboration, Copenhagen2011
- The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses.Milbank Q. 2016; 94: 485-514
- Badges to acknowledge open practices: a simple, low-cost, effective method for increasing transparency.PLoS Biol. 2016; 14: e1002456
- How open science helps researchers succeed.Elife. 2016; 5: e16800
- Making sense of replications.Elife. 2017; 6: e23383
- PSYCHOLOGY. Estimating the reproducibility of psychological science.Science. 2015; 349: aac4716
- Factors affecting the delivery, access, and use of interventions to prevent malaria in pregnancy in sub-Saharan Africa: a systematic review and meta-analysis.PLoS Med. 2013; 10: e1001488
- The need for individual trial results in reports of systematic reviews.BMJ. 2001; 323: 776
- Data extraction errors in meta-analyses that use standardized mean differences.JAMA. 2007; 298: 430-437
- Impact of an online writing aid tool for writing a randomized trial report: the COBWEB (Consort-based WEB tool) randomized controlled trial.BMC Med. 2015; 13: 221
- Disclosure of data and statistical commands should accompany completely reported studies.J Clin Epidemiol. 2016; 70: 272-274
- A Web-based archive of systematic review data.Syst Rev. 2012; 1: 15
- Innovations in data collection, management, and archiving for systematic reviews.Ann Intern Med. 2015; 162: 287-294
- Academic criteria for appointment, promotion and rewards in medical research: where's the evidence?.Eur J Clin Invest. 2016; 46: 383-385
Article info
Publication history
Published online: November 04, 2017
Accepted:
October 30,
2017
Footnotes
Funding: There was no direct funding for this study. M.J.P. is supported by an Australian National Health and Medical Research Council (NHMRC) Early Career Fellowship (1088535). J.E.M. is supported by an NHMRC Australian Public Health Fellowship (1072366). F.C.-L. is supported by the Generalitat Valenciana (PROMETEOII/2015/021). A.C.T. is funded by a Tier 2 Canada Research Chair in Knowledge Synthesis. D.M. is supported in part by a University Research Chair, University of Ottawa. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the article.
Conflicts of interest: The authors have read the journal's policy and the authors of this article have the following competing interests: A.C.T. is an associate editor for Journal of Clinical Epidemiology but had no involvement in the peer review process or decision for publication. M.J.P. and J.E.M. are affiliates of Cochrane Australia. M.J.P. is a co-convenor of the Cochrane Bias Methods Group. J.E.M. is a co-convenor of the Cochrane Statistical Methods Group. A.C.T. is an author of two of the systematic reviews included in this study but was not involved in eligibility assessment or data collection. D.G.A. is a senior investigator of National Institute for Health Research.
Data availability: The study protocol, data-collection form, and the raw data and statistical analysis code for this study are available on the Open Science Framework: https://osf.io/523bq/.
Identification
Copyright
© 2017 Elsevier Inc. All rights reserved.
