Advertisement

Most healthcare interventions tested in Cochrane Reviews are not effective according to high quality evidence: a systematic review and meta-analysis

      Highlights

      • In this large sample of 1,567 interventions studied within Cochrane reviews, effects of most interventions (94%) interventions were not supported by high-quality evidence.
      • Potential harms of healthcare interventions were measured more rarely than benefits.
      • Patients, doctors, and policy makers should consider the lack of high-quality evidence supporting the benefits and harms of many interventions in their decision-making.

      Abstract

      Objective

      To estimate the proportion of healthcare interventions tested within Cochrane Reviews that are effective according to high-quality evidence.

      Methods

      We selected a random sample of 2,428 (35%) of all Cochrane Reviews published between 1 January 2008 and 5 March 2021. We extracted data about interventions within these reviews that were compared with placebo, or no treatment, and whose outcome quality was rated using the Grading of Recommendations Assessment, Development and Evaluation (GRADE) system. We calculated the proportion of interventions whose benefits were based on high-quality evidence (defined as having high quality GRADE rating for at least one primary outcome, statistically significant positive results, and being judged by review authors as effective. We also calculated the proportion of interventions that suggested harm.

      Results

      Of 1,567 eligible interventions, 87 (5.6%) had high-quality evidence supporting their benefits. Harms were measured for 577 (36.8%) interventions. There was statistically significant evidence for harm in 127 (8.1%) of these. Our dependence on the reliability of Cochrane author assessments (including their GRADE assessments) was the main potential limitation of our study.

      Conclusion

      More than 9 in 10 healthcare interventions studied within recent Cochrane Reviews are not supported by high-quality evidence, and harms are under-reported.

      Keywords

      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'

      Subscribe:

      Subscribe to Journal of Clinical Epidemiology
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect

      References

        • Moore T.J.
        Deadly Medicine: Why tens of thousands of heart patients died in America's worst drug disaster.
        Simon & Schuster, NY; London1995
        • Spock B.
        Baby and child care.
        New English Library, London1961
        • Gilbert R.
        • Salanti G.
        • Harden M.
        • See S.
        Infant sleeping position and the sudden infant death syndrome: systematic review of observational studies and historical review of recommendations from 1940 to 2002.
        Int J Epidemiol. 2005; 34: 874-887
        • Jefferson T.
        • Jones M.
        • Doshi P.
        • Spencer E.A.
        • Onakpoya I.
        • Heneghan C.J.
        Oseltamivir for influenza in adults and children: systematic review of clinical study reports and summary of regulatory comments.
        BMJ. 2014; 348: g2545
        • Worrall J.
        What evidence in evidence-based medicine?.
        Philos Sci. 2002; 69: S316-S330
        • Tonelli M.R.
        The limits of evidence-based medicine.
        Respir Care. 2001; 46 (discussion 40-1): 1435-1440
        • McKeown T.
        The role of medicine: dream, mirage or nemesis?.
        Nuffield Provincial Hospitals Trust, London1976
        • El Dib R.P.
        • Atallah A.N.
        • Andriolo R.B.
        Mapping the Cochrane evidence for decision making in health care.
        J Eval Clin Pract. 2007; 13: 689-692
        • Garrow J.S.
        What to do about CAM: how much of orthodox medicine is evidence based?.
        BMJ. 2007; 335: 951
        • Howick J.
        The philosophy of evidence-based medicine.
        Wiley-Blackwell, Oxford2011
      1. Schünemann H. Brożek J. Guyatt G. Oxman A. GRADE handbook for grading quality of evidence and strength of recommendations. Updated October 2013. The GRADE Working Group. 2013 (Available at)
        • Granholm A.
        • Alhazzani W.
        • Møller M.H.
        Use of the GRADE approach in systematic reviews and guidelines.
        Br J Anaesth. 2019 Nov; 123: 554-559
        • Fleming P.S.
        • Koletsi D.
        • Ioannidis J.P.
        • Pandis N.
        High quality of the evidence for medical and other health-related interventions was uncommon in cochrane systematic reviews.
        J Clin Epidemiol. 2016; 78: 34-42
        • Howick J.
        • Koletsi D.
        • Pandis N.
        • Fleming P.S.
        • Loef M.
        • Walach H.
        • et al.
        The quality of evidence for medical interventions does not improve or worsen: a metaepidemiological study of Cochrane reviews.
        J Clin Epidemiol. 2020; 126: 154-159
        • Liberati A.
        • Altman D.G.
        • Tetzlaff J.
        • Mulrow C.
        • Gotzsche P.C.
        • Ioannidis J.P.A.
        • et al.
        The PRISMA statement for reporting systematic reviews and meta-Analyses of studies that evaluate health care interventions: Explanation and Elaboration.
        Ann Intern Med. 2009; 151: W65-W94
      2. Furey E 2021;Pages. Accessed at Calculator Soup.
        (Available at)
        https://www.calculatorsoup.com
        Date accessed: October 28, 2021
      3. Systems CR 2012;Pages. Accessed at Creative Research Systems.
        (Available at)
        https://www.surveysystem.com/sscalc.htm
        Date accessed: May 12, 2021
        • Mattick R.P.
        • Breen C.
        • Kimber J.
        • Davoli M.
        Buprenorphine maintenance versus placebo or methadone maintenance for opioid dependence.
        Cochrane Database Syst Rev. 2014; 3: CD002207
      4. Sauer S 2021;Pages. Accessed at Sebastian Sauer.
        (Available at)
      5. 2021;Pages. Accessed at Kleijnen Systematic Reviews.
        (Available at)
        https://ksrevidence.com
        Date accessed: November 4, 2021
        • Corporation S
        Stata statistical software: release 14.
        StataCorp LP, College Station, TX2015
        • Ezzo J.
        • Bausell B.
        • Moerman D.E.
        • Berman B.
        • Hadhazy V.
        Reviewing the reviews. How strong is the evidence? How clear are the conclusions?.
        Int J Technol Assess Health Care. 2001; 17: 457-466
        • Qureshi R.
        • Mayo-Wilson E.
        • Li T.
        Summaries of harms in systematic reviews are unreliable Paper 1: An introduction to research on harms.
        J Clin Epidemiol. 2021; 143: 186-196
        • Higgins J.J.
        • Thomas J.C.
        • Chandler J.
        • Cumpston M.
        • Li T.
        • Page M.J.
        • et al.
        Cochrane handbook for systematic reviews of interventions version 6.0.
        Version 6.0 ed. The Cochrane Collaboration, Chichester2019
        • Mustafa R.A.
        • Santesso N.
        • Brozek J.
        • Akl E.A.
        • Walter S.D.
        • Norman G.
        • et al.
        The GRADE approach is reproducible in assessing the quality of evidence of quantitative evidence syntheses.
        J Clin Epidemiol. 2013; 66 (quiz 42 e1-5): 736-742
        • Langendam M.
        • Carrasco-Labra A.
        • Santesso N.
        • Mustafa R.A.
        • Brignardello-Petersen R.
        • Ventresca M.
        • et al.
        Improving GRADE evidence tables part 2: a systematic survey of explanatory notes shows more guidance is needed.
        J Clin Epidemiol. 2016; 74: 19-27
        • Glasziou P.
        • Chalmers I.
        • Rawlins M.
        • McCulloch P.
        When are randomised trials unnecessary? Picking signal from noise.
        BMJ. 2007; 334: 349-351
        • Papanikolaou P.N.
        • Ioannidis J.P.
        Availability of large-scale evidence on specific harms from systematic reviews of randomized trials.
        Am J Med. 2004; 117: 582-589
        • Saini P.
        • Loke Y.K.
        • Gamble C.
        • Altman D.G.
        • Williamson P.R.
        • Kirkham J.J.
        Selective reporting bias of harm outcomes within studies: findings from a cohort of systematic reviews.
        BMJ. 2014; 349: g6501
        • Hopewell S.
        • Wolfenden L.
        • Clarke M.
        Reporting of adverse events in systematic reviews can be improved: survey results.
        J Clin Epidemiol. 2008; 61: 597-602
        • Pitrou I.
        • Boutron I.
        • Ahmad N.
        • Ravaud P.
        Reporting of safety results in published reports of randomized controlled trials.
        Arch Intern Med. 2009; 169: 1756-1761
        • Zorzela L.
        • Golder S.
        • Liu Y.
        • Pilkington K.
        • Hartling L.
        • Joffe A.
        • et al.
        Quality of reporting in systematic reviews of adverse events: systematic review.
        BMJ. 2014; 348: f7668