Highlights
- •In this large sample of 1,567 interventions studied within Cochrane reviews, effects of most interventions (94%) interventions were not supported by high-quality evidence.
- •Potential harms of healthcare interventions were measured more rarely than benefits.
- •Patients, doctors, and policy makers should consider the lack of high-quality evidence supporting the benefits and harms of many interventions in their decision-making.
Abstract
Objective
To estimate the proportion of healthcare interventions tested within Cochrane Reviews
that are effective according to high-quality evidence.
Methods
We selected a random sample of 2,428 (35%) of all Cochrane Reviews published between
1 January 2008 and 5 March 2021. We extracted data about interventions within these
reviews that were compared with placebo, or no treatment, and whose outcome quality
was rated using the Grading of Recommendations Assessment, Development and Evaluation
(GRADE) system. We calculated the proportion of interventions whose benefits were
based on high-quality evidence (defined as having high quality GRADE rating for at
least one primary outcome, statistically significant positive results, and being judged
by review authors as effective. We also calculated the proportion of interventions
that suggested harm.
Results
Of 1,567 eligible interventions, 87 (5.6%) had high-quality evidence supporting their
benefits. Harms were measured for 577 (36.8%) interventions. There was statistically
significant evidence for harm in 127 (8.1%) of these. Our dependence on the reliability
of Cochrane author assessments (including their GRADE assessments) was the main potential
limitation of our study.
Conclusion
More than 9 in 10 healthcare interventions studied within recent Cochrane Reviews
are not supported by high-quality evidence, and harms are under-reported.
Keywords
To read this article in full you will need to make a payment
Purchase one-time access:
Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online accessOne-time access price info
- For academic or personal research use, select 'Academic and Personal'
- For corporate R&D use, select 'Corporate R&D Professionals'
Subscribe:
Subscribe to Journal of Clinical EpidemiologyAlready a print subscriber? Claim online access
Already an online subscriber? Sign in
Register: Create an account
Institutional Access: Sign in to ScienceDirect
References
- Deadly Medicine: Why tens of thousands of heart patients died in America's worst drug disaster.Simon & Schuster, NY; London1995
- Baby and child care.New English Library, London1961
- Infant sleeping position and the sudden infant death syndrome: systematic review of observational studies and historical review of recommendations from 1940 to 2002.Int J Epidemiol. 2005; 34: 874-887
- Oseltamivir for influenza in adults and children: systematic review of clinical study reports and summary of regulatory comments.BMJ. 2014; 348: g2545
- What evidence in evidence-based medicine?.Philos Sci. 2002; 69: S316-S330
- The limits of evidence-based medicine.Respir Care. 2001; 46 (discussion 40-1): 1435-1440
- The role of medicine: dream, mirage or nemesis?.Nuffield Provincial Hospitals Trust, London1976
- Mapping the Cochrane evidence for decision making in health care.J Eval Clin Pract. 2007; 13: 689-692
- What to do about CAM: how much of orthodox medicine is evidence based?.BMJ. 2007; 335: 951
- The philosophy of evidence-based medicine.Wiley-Blackwell, Oxford2011
- Schünemann H. Brożek J. Guyatt G. Oxman A. GRADE handbook for grading quality of evidence and strength of recommendations. Updated October 2013. The GRADE Working Group. 2013 (Available at)
- Use of the GRADE approach in systematic reviews and guidelines.Br J Anaesth. 2019 Nov; 123: 554-559
- High quality of the evidence for medical and other health-related interventions was uncommon in cochrane systematic reviews.J Clin Epidemiol. 2016; 78: 34-42
- The quality of evidence for medical interventions does not improve or worsen: a metaepidemiological study of Cochrane reviews.J Clin Epidemiol. 2020; 126: 154-159
- The PRISMA statement for reporting systematic reviews and meta-Analyses of studies that evaluate health care interventions: Explanation and Elaboration.Ann Intern Med. 2009; 151: W65-W94
- Furey E 2021;Pages. Accessed at Calculator Soup.(Available at)https://www.calculatorsoup.comDate accessed: October 28, 2021
- Systems CR 2012;Pages. Accessed at Creative Research Systems.(Available at)https://www.surveysystem.com/sscalc.htmDate accessed: May 12, 2021
- Buprenorphine maintenance versus placebo or methadone maintenance for opioid dependence.Cochrane Database Syst Rev. 2014; 3: CD002207
- Sauer S 2021;Pages. Accessed at Sebastian Sauer.(Available at)https://github.com/sebastiansauer/Cochrane-ParsingDate accessed: October 3, 2021
- 2021;Pages. Accessed at Kleijnen Systematic Reviews.(Available at)https://ksrevidence.comDate accessed: November 4, 2021
- Stata statistical software: release 14.StataCorp LP, College Station, TX2015
- Reviewing the reviews. How strong is the evidence? How clear are the conclusions?.Int J Technol Assess Health Care. 2001; 17: 457-466
- Summaries of harms in systematic reviews are unreliable Paper 1: An introduction to research on harms.J Clin Epidemiol. 2021; 143: 186-196
- Cochrane handbook for systematic reviews of interventions version 6.0.Version 6.0 ed. The Cochrane Collaboration, Chichester2019
- The GRADE approach is reproducible in assessing the quality of evidence of quantitative evidence syntheses.J Clin Epidemiol. 2013; 66 (quiz 42 e1-5): 736-742
- Improving GRADE evidence tables part 2: a systematic survey of explanatory notes shows more guidance is needed.J Clin Epidemiol. 2016; 74: 19-27
- When are randomised trials unnecessary? Picking signal from noise.BMJ. 2007; 334: 349-351
- Availability of large-scale evidence on specific harms from systematic reviews of randomized trials.Am J Med. 2004; 117: 582-589
- Selective reporting bias of harm outcomes within studies: findings from a cohort of systematic reviews.BMJ. 2014; 349: g6501
- Reporting of adverse events in systematic reviews can be improved: survey results.J Clin Epidemiol. 2008; 61: 597-602
- Reporting of safety results in published reports of randomized controlled trials.Arch Intern Med. 2009; 169: 1756-1761
- Quality of reporting in systematic reviews of adverse events: systematic review.BMJ. 2014; 348: f7668
Article info
Publication history
Published online: April 17, 2022
Accepted:
April 12,
2022
Footnotes
Registration: PROSPERO: CRD42021240989.
Funding source: This study was not externally funded.
Declaration of interest: None of the authors have any personal financial interests or professional relationships to disclose.
Contributions: JH (guarantor) conceived of the idea (together with StS), and wrote the first draft of the protocol all authors contributed to developing the protocol. JH piloted the data extraction form and all authors made suggestions for improvement. JH, DK, TJ, CM, ML, HW, SeS, JS, NP, StS, and JPAI contributed to the data extraction. SeS developed a computerized quality check; HW and JH resolved discrepancies. JH, JPAI, CM, and DK developed a plan for and analyzed the data. JH drafted the final manuscript, with contributions from all authors.
Identification
Copyright
© 2022 Elsevier Inc. All rights reserved.
