Cochrane risk of bias tool was used inadequately in the majority of non-Cochrane systematic reviews



      To analyze how many non-Cochrane systematic reviews (NCSRs) used Cochrane's risk of bias (RoB) tool, domains they used, and whether judgments and comments about RoB were in line with Cochrane Handbook.


      This was a methodological (research-on-research) study. We retrieved NCSRs from PubMed, extracted information about methods used for RoB assessment, and if they used 2011 Cochrane RoB tool, we analyzed their RoB methods and compared them with Cochrane Handbook guidance.


      We included 508 NCSRs; 431 (85%) reported they analyzed RoB, and 269 (53%) used Cochrane RoB tool. Only 16 of those 269 (5.9%) reported both a judgment and a supporting comment in the Cochrane RoB table in the manuscript (N = 4) or in a supplementary file (N = 12). Fifteen reviews, with 158 included trials, used judgments low/high/unclear; 41% of analyzed available judgments were inadequate, either because judgment was not in line with comment or comment was missing.


      Most NCSRs use Cochrane RoB tool to assess RoB, but most of them reported it incompletely, with high prevalence of inadequate judgments. Authors, editors, and peer-reviewers should make an effort to improve completeness and adequacy of Cochrane RoB assessment in non-Cochrane reviews.


      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to Journal of Clinical Epidemiology
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


      1. Higgins J.P.T. Green S. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0. The Cochrane Collaboration, London, UK2011 (Available at)
        Date accessed: April 27, 2020
        • Babic A.
        • Pijuk A.
        • Brazdilova L.
        • Georgieva Y.
        • Raposo Pereira M.A.
        • Poklepovic Pericic T.
        • et al.
        The judgement of biases included in the category "other bias" in Cochrane systematic reviews of interventions: a systematic survey.
        BMC Med Res Methodol. 2019; 19: 77
        • Babic A.
        • Tokalic R.
        • Amilcar Silva Cunha J.
        • Novak I.
        • Suto J.
        • Vidak M.
        • et al.
        Assessments of attrition bias in Cochrane systematic reviews are highly inconsistent and thus hindering trial comparability.
        BMC Med Res Methodol. 2019; 19: 76
        • Barcot O.
        • Boric M.
        • Dosenovic S.
        • Poklepovic Pericic T.
        • Cavar M.
        • Puljak L.
        Risk of bias assessments for blinding of participants and personnel in Cochrane reviews were frequently inadequate.
        J Clin Epidemiol. 2019; 113: 104-113
        • Propadalo I.
        • Tranfic M.
        • Vuka I.
        • Barcot O.
        • Pericic T.P.
        • Puljak L.
        Cochrane reviews, risk of bias assessments for allocation concealment were frequently not in line with Cochrane's Handbook guidance.
        J Clin Epidemiol. 2019; 106: 10-17
        • Saric F.
        • Barcot O.
        • Puljak L.
        Risk of bias assessments for selective reporting were inadequate in the majority of Cochrane reviews.
        J Clin Epidemiol. 2019; 112: 53-58
        • Barcot O.
        • Boric M.
        • Poklepovic Pericic T.
        • Cavar M.
        • Dosenovic S.
        • Vuka I.
        • et al.
        Risk of bias judgments for random sequence generation in Cochrane systematic reviews were frequently not in line with Cochrane Handbook.
        BMC Med Res Methodol. 2019; 19: 170
        • Higgins J.P.
        • Altman D.G.
        • Gotzsche P.C.
        • Juni P.
        • Moher D.
        • Oxman A.D.
        • et al.
        The Cochrane Collaboration's tool for assessing risk of bias in randomised trials.
        BMJ. 2011; 343: d5928
        • Schmidt C.B.
        • van Loon B.J.P.
        • Vergouwen A.C.M.
        • Snoek F.J.
        • Honig A.
        Systematic review and meta-analysis of psychological interventions in people with diabetes and elevated diabetes-distress.
        Diabet Med. 2018; 35: 1157-1172
        • Puljak L.
        Adequate and complete reporting of Cochrane risk of bias tool.
        Pain. 2019; 160: 984
        • Moher D.
        • Liberati A.
        • Tetzlaff J.
        • Altman D.G.
        Preferred reporting items for systematic reviews and meta-analyses: the PRISMA Statement.
        Open Med. 2009; 3: e123-e130
        • Shea B.J.
        • Reeves B.C.
        • Wells G.
        • Thuku M.
        • Hamel C.
        • Moran J.
        • et al.
        Amstar 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both.
        BMJ. 2017; 358: j4008
        • Sterne J.A.C.
        • Savovic J.
        • Page M.J.
        • Elbers R.G.
        • Blencowe N.S.
        • Boutron I.
        • et al.
        RoB 2: a revised tool for assessing risk of bias in randomised trials.
        BMJ. 2019; 366: l4898