The quality of evidence for medical interventions does not improve or worsen: a metaepidemiological study of Cochrane reviews

Published:September 02, 2020DOI:



      The objective of the study was to determine the change in quality of evidence in updates of Cochrane reviews that were initially published between January 1, 2013 and June 30, 2014. We used the Grading of Recommendations, Assessment, Development and Evaluation (GRADE) system to document evidence quality.

      Study Design and Setting

      We searched the Cochrane Database of Systematic Reviews on March 20, 2020 to identify which of the reviews from the initial (2013/14) sample had been updated. Using the same methods to determine the quality of evidence in the previous analysis, we assessed the quality of evidence for the first-listed primary outcomes in the updated reviews.


      Of the 608 reviews in the original sample, 154 had been updated with and 151 contained available data for both original and updated systematic reviews (24.8%). The updated reviews included: 15 (9.9%) with high-quality evidence, 56 (37.1%) with moderate-quality evidence, 47 (31.1%) with low-quality evidence, and 33 (21.9%) with very low-quality evidence. No change in the GRADE quality of evidence was found for most (103, 68.2%) of the updated reviews. The quality of evidence rating was downgraded in 28 reviews (58.3%) and upgraded in 20 (41.7%), although only six reviews were promoted to high quality.


      Updated systematic reviews continued to suggest that only a minority of outcomes for health care interventions are supported by high-quality evidence. The quality of the evidence did not consistently improve or worsen in updated reviews.


      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to Journal of Clinical Epidemiology
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Ezzo J.
        • Bausell B.
        • Moerman D.E.
        • Berman B.
        • Hadhazy V.
        Reviewing the reviews. How strong is the evidence? How clear are the conclusions?.
        Int J Technol Assess Health Care. 2001; 17: 457-466
        • Garrow J.S.
        What to do about CAM: how much of orthodox medicine is evidence based?.
        BMJ. 2007; 335: 951
        • El Dib R.P.
        • Atallah A.N.
        • Andriolo R.B.
        Mapping the Cochrane evidence for decision making in health care.
        J Eval Clin Pract. 2007; 13: 689-692
        • Balshem H.
        • Helfand M.
        • Schunemann H.J.
        • Oxman A.D.
        • Kunz R.
        • Brozek J.
        • et al.
        GRADE guidelines: 3. Rating the quality of evidence.
        J Clin Epidemiol. 2011; 64: 401-406
        • Fleming P.S.
        • Koletsi D.
        • Ioannidis J.P.
        • Pandis N.
        High quality of the evidence for medical and other health-related interventions was uncommon in Cochrane systematic reviews.
        J Clin Epidemiol. 2016; 78: 34-42
        • Chalmers I.
        • Haynes B.
        Reporting, updating, and correcting systematic reviews of the effects of health care.
        BMJ. 1994; 309: 862-865
        • Garritty C.
        • Tsertsvadze A.
        • Tricco A.C.
        • Sampson M.
        • Moher D.
        Updating systematic reviews: an international survey.
        PLoS One. 2010; 5: e9914
        • Moher D.
        • Tsertsvadze A.
        Systematic reviews: when is an update an update?.
        Lancet. 2006; 367: 881-883
        • French S.D.
        • McDonald S.
        • McKenzie J.E.
        • Green S.E.
        Investing in updating: how do conclusions change when Cochrane systematic reviews are updated?.
        BMC Med Res Methodol. 2005; 5: 33
        • Higgins J.J.
        • Thomas J.C.
        • Chandler J.
        • Cumpston M.
        • Li T.
        • Page M.J.
        • et al.
        Cochrane Handbook for Systematic Reviews of Interventions version 6.0.
        The Cochrane Collaboration, Chichester2019
        • Shojania K.G.
        • Sampson M.
        • Ansari M.T.
        • Ji J.
        • Doucette S.
        • Moher D.
        How quickly do systematic reviews go out of date? A survival analysis.
        Ann Intern Med. 2007; 147: 224-233
        • Mattick R.P.
        • Breen C.
        • Kimber J.
        • Davoli M.
        Buprenorphine maintenance versus placebo or methadone maintenance for opioid dependence.
        Cochrane Database Syst Rev. 2014; : CD002207
        • Higgins J.J.
        • Green S.
        The Cochrane Handbook for Systematic Reviews of Interventions. Version 5.1.0 [updated March 2011].
        The Cochrane Collaboration, Chichester2011
        • Berkman N.D.
        • Lohr K.N.
        • Morgan L.C.
        • Kuo T.M.
        • Morton S.C.
        Interrater reliability of grading strength of evidence varies with the complexity of the evidence in systematic reviews.
        J Clin Epidemiol. 2013; 66: 1105-1107.e1
        • Hartling L.
        • Hamm M.P.
        • Milne A.
        • Vandermeer B.
        • Santaguida P.L.
        • Ansari M.
        • et al.
        Testing the risk of bias tool showed low reliability between individual reviewers and across consensus assessments of reviewer pairs.
        J Clin Epidemiol. 2013; 66: 973-981
        • Boardman H.M.
        • Hartley L.
        • Eisinga A.
        • Main C.
        • Roqué i Figuls M.
        • Bonfill Cosp X.
        • et al.
        Hormone therapy for preventing cardiovascular disease in post-menopausal women.
        Cochrane Database Syst Rev. 2015; : CD002229
        • Hakoum M.B.
        • Kahale L.A.
        • Tsolakian I.G.
        • Matar C.F.
        • Yosuico V.E.
        • Terrenato I.
        • et al.
        Anticoagulation for the initial treatment of venous thromboembolism in people with cancer.
        Cochrane Database Syst Rev. 2018; 1: CD006649