GRADE guidelines: 18. How ROBINS-I and other tools to assess risk of bias in nonrandomized studies should be used to rate the certainty of a body of evidence

  • Holger J. Schünemann
    Correspondence
    Corresponding author. Chair and Professor, Department of Health Research Methods, Evidence and Impact, McMaster University Health Sciences Centre, Room 2C16, 1280 Main Street West, Hamilton, Ontario L8N 4K1, Canada. Tel.: +1 905 525 9140 x 24931; fax: +1 905 522 9507 .
    Affiliations
    Department of Health Research Methods, Evidence, and Impact and McGRADE Center, McMaster University, 1280 Main Street West, Hamilton, Ontario, L8S4K1, Canada

    Department of Medicine, McMaster University, 1280 Main Street West, Hamilton, Ontario, L8S4K1, Canada
    Search for articles by this author
  • Carlos Cuello
    Affiliations
    Department of Health Research Methods, Evidence, and Impact and McGRADE Center, McMaster University, 1280 Main Street West, Hamilton, Ontario, L8S4K1, Canada
    Search for articles by this author
  • Elie A. Akl
    Affiliations
    Department of Health Research Methods, Evidence, and Impact and McGRADE Center, McMaster University, 1280 Main Street West, Hamilton, Ontario, L8S4K1, Canada

    AUB GRADE Center, Clinical Research Institute, American University of Beirut, PO Box 11-0236, Riad El Solh, Beirut, 1107 2020, Lebanon
    Search for articles by this author
  • Reem A. Mustafa
    Affiliations
    Department of Health Research Methods, Evidence, and Impact and McGRADE Center, McMaster University, 1280 Main Street West, Hamilton, Ontario, L8S4K1, Canada

    Department of Medicine, University of Kansas Medical Center, 3901 Rainbow Blvd, MS3002, Kansas City, KS, 66160, USA
    Search for articles by this author
  • Jörg J. Meerpohl
    Affiliations
    Cochrane Germany, Medical Center University of Freiburg, Breisacher Strasse 153, Freiburg, 79110, Germany
    Search for articles by this author
  • Kris Thayer
    Affiliations
    Integrated Risk Information System (IRIS) Division, National Center for Environmental Assessment, Environmental Protection Agency, 1200 Pennsylvania Avenue, N.W.Washington, DC 20460, USA
    Search for articles by this author
  • Rebecca L. Morgan
    Affiliations
    Department of Health Research Methods, Evidence, and Impact and McGRADE Center, McMaster University, 1280 Main Street West, Hamilton, Ontario, L8S4K1, Canada
    Search for articles by this author
  • Gerald Gartlehner
    Affiliations
    Department for Evidence-Based Medicine and Clinical Epidemiology, Danube University Krems, Dr Karl Dorrek Straße 30, Krems, 3500, Austria
    Search for articles by this author
  • Regina Kunz
    Affiliations
    Basel Institute of Clinical Epidemiology, University Hospital Basel, Hebelstrasse 10, Basel, 4031, Switzerland
    Search for articles by this author
  • S Vittal Katikireddi
    Affiliations
    MRC/CSO Social and Public Health Sciences Unit, University of Glasgow, Top Floor, 200 Renfield Street, Glasgow, G2 3QB, Scotland
    Search for articles by this author
  • Jonathan Sterne
    Affiliations
    Population Health Sciences, Bristol Medical School, University of Bristol, Canynge Hall, 39 Whatley Road, Bristol, BS8 2PS, UK
    Search for articles by this author
  • Julian PT Higgins
    Affiliations
    Population Health Sciences, Bristol Medical School, University of Bristol, Canynge Hall, 39 Whatley Road, Bristol, BS8 2PS, UK
    Search for articles by this author
  • Gordon Guyatt
    Affiliations
    Department of Health Research Methods, Evidence, and Impact and McGRADE Center, McMaster University, 1280 Main Street West, Hamilton, Ontario, L8S4K1, Canada

    Department of Medicine, McMaster University, 1280 Main Street West, Hamilton, Ontario, L8S4K1, Canada
    Search for articles by this author
  • GRADE Working Group
Published:February 09, 2018DOI:https://doi.org/10.1016/j.jclinepi.2018.01.012

      Abstract

      Objective

      To provide guidance on how systematic review authors, guideline developers, and health technology assessment practitioners should approach the use of the risk of bias in nonrandomized studies of interventions (ROBINS-I) tool as a part of GRADE's certainty rating process.

      Study Design and Setting

      The study design and setting comprised iterative discussions, testing in systematic reviews, and presentation at GRADE working group meetings with feedback from the GRADE working group.

      Results

      We describe where to start the initial assessment of a body of evidence with the use of ROBINS-I and where one would anticipate the final rating would end up. The GRADE accounted for issues that mitigate concerns about confounding and selection bias by introducing the upgrading domains: large effects, dose-effect relations, and when plausible residual confounders or other biases increase certainty. They will need to be considered in an assessment of a body of evidence when using ROBINS-I.

      Conclusions

      The use of ROBINS-I in GRADE assessments may allow for a better comparison of evidence from randomized controlled trials (RCTs) and nonrandomized studies (NRSs) because they are placed on a common metric for risk of bias. Challenges remain, including appropriate presentation of evidence from RCTs and NRSs for decision-making and how to optimally integrate RCTs and NRSs in an evidence assessment.

      Keywords

      To read this article in full you will need to make a payment

      References

        • Guyatt G.
        • Oxman A.D.
        • Akl E.A.
        • Kunz R.
        • Vist G.
        • Brozek J.
        • et al.
        GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables.
        J Clin Epidemiol. 2011; 64: 383-394
        • Guyatt G.H.
        • Oxman A.D.
        • Schunemann H.J.
        • Tugwell P.
        • Knotterus A.
        GRADE guidelines: a new series of articles in the Journal of Clinical Epidemiology.
        J Clin Epidemiol. 2011; 64: 380-382
        • Puhan M.A.
        • Schunemann H.J.
        • Murad M.H.
        • Li T.
        • Brignardello-Petersen R.
        • Singh J.A.
        • et al.
        A GRADE Working Group approach for rating the quality of treatment effect estimates from network meta-analysis.
        BMJ. 2014; 349: g5630
        • Schunemann H.J.
        • Best D.
        • Vist G.
        • Oxman A.D.
        • Group G.W.
        Letters, numbers, symbols and words: how to communicate grades of evidence and recommendations.
        CMAJ. 2003; 169: 677-680
        • Schunemann H.J.
        • Oxman A.D.
        • Brozek J.
        • Glasziou P.
        • Bossuyt P.
        • Chang S.
        • et al.
        GRADE: assessing the quality of evidence for diagnostic recommendations.
        ACP J Club. 2008; 149: 2
        • Spencer F.A.
        • Iorio A.
        • You J.
        • Murad M.H.
        • Schunemann H.J.
        • Vandvik P.O.
        • et al.
        Uncertainties in baseline risk estimates and confidence in treatment effects.
        BMJ. 2012; 345: e7401
        • Schunemann H.J.
        • Oxman A.D.
        • Brozek J.
        • Glasziou P.
        • Jaeschke R.
        • Vist G.E.
        • et al.
        Grading quality of evidence and strength of recommendations for diagnostic tests and strategies.
        BMJ. 2008; 336: 1106-1110
        • Guyatt G.
        • Oxman A.D.
        • Sultan S.
        • Brozek J.
        • Glasziou P.
        • Alonso-Coello P.
        • et al.
        GRADE guidelines: 11. Making an overall rating of confidence in effect estimates for a single outcome and for all outcomes.
        J Clin Epidemiol. 2013; 66: 151-157
        • Alonso-Coello P.
        • Oxman A.D.
        • Moberg J.
        • Brignardello-Petersen R.
        • Akl E.A.
        • Davoli M.
        • et al.
        GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 2: clinical practice guidelines.
        BMJ. 2016; 353: i2089
        • Alonso-Coello P.
        • Schunemann H.J.
        • Moberg J.
        • Brignardello-Petersen R.
        • Akl E.A.
        • Davoli M.
        • et al.
        GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 1: Introduction.
        BMJ. 2016; 353: i2016
        • Schunemann H.J.
        • Mustafa R.
        • Brozek J.
        • Santesso N.
        • Alonso-Coello P.
        • Guyatt G.
        • et al.
        GRADE Guidelines: 16. GRADE evidence to decision frameworks for tests in clinical practice and public health.
        J Clin Epidemiol. 2016; 76: 89-98
        • Burford B.J.
        • Rehfuess E.
        • Schunemann H.J.
        • Akl E.A.
        • Waters E.
        • Armstrong R.
        • et al.
        Assessing evidence in public health: the added value of GRADE.
        J Public Health (Oxf). 2012; 34: 631-635
        • Thayer K.A.
        • Schunemann H.J.
        Using GRADE to respond to health questions with different levels of urgency.
        Environ Int. 2016; 92-93: 585-589
        • Schunemann H.J.
        • Hill S.R.
        • Kakad M.
        • Vist G.E.
        • Bellamy R.
        • Stockman L.
        • et al.
        Transparent development of the WHO rapid advice guidelines.
        PLoS Med. 2007; 4: e119
        • Iorio A.
        • Spencer F.A.
        • Falavigna M.
        • Alba C.
        • Lang E.
        • Burnand B.
        • et al.
        Use of GRADE for assessment of evidence about prognosis: rating confidence in estimates of event rates in broad categories of patients.
        BMJ. 2015; 350: h870
        • Guyatt G.H.
        • Ebrahim S.
        • Alonso-Coello P.
        • Johnston B.C.
        • Mathioudakis A.G.
        • Briel M.
        • et al.
        GRADE guidelines 17: assessing the risk of bias associated with missing participant outcome data in a body of evidence.
        J Clin Epidemiol. 2017; 87: 14-22
        • Morgan R.L.
        • Thayer K.A.
        • Bero L.
        • Bruce N.
        • Falck-Ytter Y.
        • Ghersi D.
        • et al.
        GRADE: assessing the quality of evidence in environmental and occupational health.
        Environ Int. 2016; 92-93: 611-616
        • Anderson S.T.I.
        • Higgins J.P.T.
        Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography.
        Int J Epidemiol. 2007; 36: 666-676
        • Sterne J.A.
        • Hernan M.A.
        • Reeves B.C.
        • Savovic J.
        • Berkman N.D.
        • Viswanathan M.
        • et al.
        ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions.
        BMJ. 2016; 355: i4919
        • Katikireddi S.V.
        • Egan M.
        • Petticrew M.
        How do systematic reviews incorporate risk of bias assessments into the synthesis of evidence? A methodological study.
        J Epidemiol Community Health. 2015; 69: 189-195
        • Atkins D.
        • Best D.
        • Briss P.A.
        • Eccles M.
        • Falck-Ytter Y.
        • Flottorp S.
        • et al.
        Grading quality of evidence and strength of recommendations.
        BMJ. 2004; 328: 1490
        • Craig P.
        • Katikireddi S.V.
        • Leyland A.
        • Popham F.
        Natural experiments: an overview of methods, approaches, and contributions to public health intervention research.
        Annu Rev Public Health. 2017; 38: 39-56
        • Guyatt G.H.
        • Oxman A.D.
        • Sultan S.
        • Glasziou P.
        • Akl E.A.
        • Alonso-Coello P.
        • et al.
        GRADE guidelines: 9. Rating up the quality of evidence.
        J Clin Epidemiol. 2011; 64: 1311-1316
        • Hultcrantz M.
        • Rind D.
        • Akl E.A.
        • Treweek S.
        • Mustafa R.A.
        • Iorio A.
        • et al.
        The GRADE Working Group clarifies the construct of certainty of evidence.
        J Clin Epidemiol. 2017; 87: 4-13
        • Devereaux P.J.
        • Choi P.T.
        • Lacchetti C.
        • Weaver B.
        • Schunemann H.J.
        • Haines T.
        • et al.
        A systematic review and meta-analysis of studies comparing mortality rates of private for-profit and private not-for-profit hospitals.
        CMAJ. 2002; 166: 1399-1406