GRADE Guidelines: 19. Assessing the certainty of evidence in the importance of outcomes or values and preferences—Risk of bias and indirectness

  • Yuan Zhang
    Affiliations
    Department of Health Research Methods, Evidence, and Impact & McMaster GRADE Centre, McMaster University, 1280 Main Street West, Hamilton, Ontario L8N 4K1, Canada
    Search for articles by this author
  • Pablo Alonso-Coello
    Correspondence
    Corresponding authors. Department of Health Research Methods, Evidence, and Impact and McMaster GRADE Centre, McMaster University Health Sciences Centre, Room 2C16, 1280 Main Street West, Hamilton, Ontario L8N 4K1, Canada. Tel.: +1-905-525-9140x24931; fax: +1-905-522-9507.
    Affiliations
    Department of Health Research Methods, Evidence, and Impact & McMaster GRADE Centre, McMaster University, 1280 Main Street West, Hamilton, Ontario L8N 4K1, Canada

    Centro Cochrane Iberoamericano, Instituto de Investigacion Biomedica (IIB Sant Pau-CIBERESP), Sant Antoni Maria Claret 167, 08025 Barcelona, Spain
    Search for articles by this author
  • Gordon H. Guyatt
    Affiliations
    Department of Health Research Methods, Evidence, and Impact & McMaster GRADE Centre, McMaster University, 1280 Main Street West, Hamilton, Ontario L8N 4K1, Canada
    Search for articles by this author
  • Juan José Yepes-Nuñez
    Affiliations
    Department of Health Research Methods, Evidence, and Impact & McMaster GRADE Centre, McMaster University, 1280 Main Street West, Hamilton, Ontario L8N 4K1, Canada
    Search for articles by this author
  • Elie A. Akl
    Affiliations
    Department of Health Research Methods, Evidence, and Impact & McMaster GRADE Centre, McMaster University, 1280 Main Street West, Hamilton, Ontario L8N 4K1, Canada

    Department of Internal Medicine, Faculty of Medicine, American University of Beirut, Beirut, Lebanon
    Search for articles by this author
  • Glen Hazlewood
    Affiliations
    Department of Medicine and Department of Community Health Sciences, University of Calgary, Calgary, Alberta, Canada
    Search for articles by this author
  • Hector Pardo-Hernandez
    Affiliations
    Centro Cochrane Iberoamericano, Instituto de Investigacion Biomedica (IIB Sant Pau-CIBERESP), Sant Antoni Maria Claret 167, 08025 Barcelona, Spain
    Search for articles by this author
  • Itziar Etxeandia-Ikobaltzeta
    Affiliations
    Department of Health Research Methods, Evidence, and Impact & McMaster GRADE Centre, McMaster University, 1280 Main Street West, Hamilton, Ontario L8N 4K1, Canada
    Search for articles by this author
  • Amir Qaseem
    Affiliations
    Department of Clinical Policy, American College of Physicians, 190 N. Independence Mall West, Philadelphia, PA 19106, USA
    Search for articles by this author
  • John W. Williams Jr.
    Affiliations
    Center of Innovation for Health Services Research in Primary Care, Durham Veterans Affairs Medical Center and Duke University, Durham, NC 27701, USA
    Search for articles by this author
  • Peter Tugwell
    Affiliations
    Department of Medicine, University of Ottawa, Ottawa, Ontario, Canada
    Search for articles by this author
  • Signe Flottorp
    Affiliations
    Division for Health Services, Norwegian Institute of Public Health, Oslo, Norway

    Department of Health Management and Health Economics, Institute of Health and Society, University of Oslo, Oslo, Norway
    Search for articles by this author
  • Yaping Chang
    Affiliations
    Department of Health Research Methods, Evidence, and Impact & McMaster GRADE Centre, McMaster University, 1280 Main Street West, Hamilton, Ontario L8N 4K1, Canada
    Search for articles by this author
  • Yuqing Zhang
    Affiliations
    Department of Health Research Methods, Evidence, and Impact & McMaster GRADE Centre, McMaster University, 1280 Main Street West, Hamilton, Ontario L8N 4K1, Canada
    Search for articles by this author
  • Reem A. Mustafa
    Affiliations
    Department of Health Research Methods, Evidence, and Impact & McMaster GRADE Centre, McMaster University, 1280 Main Street West, Hamilton, Ontario L8N 4K1, Canada

    Department of Internal Medicine, Division of Nephrology and Hypertension, University of Kansas Medical Center, Kansas City, KS, USA
    Search for articles by this author
  • María Ximena Rojas
    Affiliations
    Department of Clinical Epidemiology and Biostatistics, Pontificia Universidad Javeriana, Bogotá, Colombia
    Search for articles by this author
  • Holger J. Schünemann
    Correspondence
    Corresponding authors. Department of Health Research Methods, Evidence, and Impact and McMaster GRADE Centre, McMaster University Health Sciences Centre, Room 2C16, 1280 Main Street West, Hamilton, Ontario L8N 4K1, Canada. Tel.: +1-905-525-9140x24931; fax: +1-905-522-9507.
    Affiliations
    Department of Health Research Methods, Evidence, and Impact & McMaster GRADE Centre, McMaster University, 1280 Main Street West, Hamilton, Ontario L8N 4K1, Canada

    Department of Medicine, McMaster University, Hamilton, Ontario, Canada
    Search for articles by this author
Published:February 13, 2018DOI:https://doi.org/10.1016/j.jclinepi.2018.01.013

      Abstract

      Objectives

      The Grading of Recommendations Assessment, Development, and Evaluation (GRADE) working group defines patient values and preferences as the relative importance patients place on the main health outcomes. We provide GRADE guidance for assessing the risk of bias and indirectness domains for certainty of evidence about the relative importance of outcomes.

      Study Design and Setting

      We applied the GRADE domains to rate the certainty of evidence in the importance of outcomes to several systematic reviews, iteratively reviewed draft guidance and consulted GRADE members and other stakeholders for feedback.

      Results

      This is the first of two articles. A body of evidence addressing the importance of outcomes starts at “high certainty”; concerns with risk of bias, indirectness, inconsistency, imprecision, and publication bias lead to downgrading to moderate, low, or very low certainty. We propose subdomains of risk of bias as selection of the study population, missing data, the type of measurement instrument, and confounding; we have developed items for each subdomain. The population, intervention, comparison, and outcome elements associated with the evidence determine the degree of indirectness.

      Conclusion

      This article provides guidance and examples for rating the risk of bias and indirectness for a body of evidence summarizing the importance of outcomes.

      Keywords

      To read this article in full you will need to make a payment

      References

        • Schunemann H.J.
        • Fretheim A.
        • Oxman A.D.
        Improving the use of research evidence in guideline development: 10. Integrating values and consumer involvement.
        Health Res Policy Syst. 2006; 4: 22
        • Murad M.H.
        • Montori V.M.
        • Guyatt G.H.
        Incorporating patient preferences in evidence-based medicine.
        JAMA. 2008; 300 (author reply-4): 2483
        • Schunemann H.J.
        • Wiercioch W.
        • Etxeandia I.
        • Falavigna M.
        • Santesso N.
        • Mustafa R.
        • et al.
        Guidelines 2.0: systematic development of a comprehensive checklist for a successful guideline enterprise.
        CMAJ. 2014; 186: E123-E142
        • MacLean S.
        • Mulla S.
        • Akl E.A.
        • Jankowski M.
        • Vandvik P.O.
        • Ebrahim S.
        • et al.
        Patient values and preferences in decision making for antithrombotic therapy: a systematic review: antithrombotic therapy and prevention of thrombosis, 9th ed.: American College of Chest Physicians Evidence-Based Clinical Practice Guidelines.
        Chest. 2012; 141: e1S-e23S
        • Andrews J.C.
        • Schunemann H.J.
        • Oxman A.D.
        • Pottie K.
        • Meerpohl J.J.
        • Coello P.A.
        • et al.
        GRADE guidelines: 15. Going from evidence to recommendation-determinants of a recommendation's direction and strength.
        J Clin Epidemiol. 2013; 66: 726-735
        • Krahn M.
        • Naglie G.
        The next step in guideline development: incorporating patient preferences.
        JAMA. 2008; 300: 436-438
        • van der Weijden T.
        • Legare F.
        • Boivin A.
        • Burgers J.S.
        • van Veenendaal H.
        • Stiggelbout A.M.
        • et al.
        How to integrate individual patient values and preferences in clinical practice guidelines? A research protocol.
        Implement Sci. 2010; 5: 10
        • Gafni A.
        The standard gamble method: what is being measured and how it is interpreted.
        Health Serv Res. 1994; 29: 207-224
        • Torrance G.W.
        Measurement of health state utilities for economic appraisal.
        J Health Econ. 1986; 5: 1-30
        • Torrance G.W.
        Utility measurement in healthcare: the things I never got to.
        PharmacoEconomics. 2006; 24: 1069-1078
        • Churchill D.N.
        • Torrance G.W.
        • Taylor D.W.
        • Barnes C.C.
        • Ludwin D.
        • Shimizu A.
        • et al.
        Measurement of quality of life in end-stage renal disease: the time trade-off approach.
        Clin Invest Med. 1987; 10: 14-20
        • Dolan P.
        • Gudex C.
        • Kind P.
        • Williams A.
        The time trade-off method: results from a general population study.
        Health Econ. 1996; 5: 141-154
        • Torrance G.W.
        • Feeny D.
        • Furlong W.
        Visual analog scales: do they have a role in the measurement of preferences for health states?.
        Med Decis Making. 2001; 21: 329-334
        • Morimoto T.
        • Fukui T.
        Utilities measured by rating scale, time trade-off, and standard gamble: review and reference for health care professionals.
        J Epidemiol. 2002; 12: 160-178
        • Ryan M.
        • Gerard K.
        Using discrete choice experiments to value health care programmes: current practice and future research reflections.
        Appl Health Econ Health Policy. 2003; 2: 55-64
        • Ryan M.
        Discrete choice experiments in health care.
        BMJ. 2004; 328: 360-361
        • Stevens T.H.
        • Belkner R.
        • Dennis D.
        • Kittredge D.
        • Willis C.
        Comparison of contingent valuation and conjoint analysis in ecosystem management.
        Ecol Econ. 2000; 32: 63-74
        • Alonso-Coello P.
        • Montori V.M.
        • Diaz M.G.
        • Devereaux P.J.
        • Mas G.
        • Diez A.I.
        • et al.
        Values and preferences for oral antithrombotic therapy in patients with atrial fibrillation: physician and patient perspectives.
        Health Expect. 2014; 18: 2318-2327
        • Devereaux P.J.
        • Anderson D.R.
        • Gardner M.J.
        • Putnam W.
        • Flowerdew G.J.
        • Brownell B.F.
        • et al.
        Differences between perspectives of physicians and patients on anticoagulation in patients with atrial fibrillation: observational study.
        BMJ. 2001; 323: 1218-1222
        • Craig B.M.
        • Busschbach J.J.
        • Salomon J.A.
        Modeling ranking, time trade-off, and visual analog scale values for EQ-5D health states: a review and comparison of methods.
        Med Care. 2009; 47: 634-641
        • Rabin R.
        • de Charro F.
        EQ-5D: a measure of health status from the EuroQol Group.
        Ann Med. 2001; 33: 337-343
        • Sepucha K.
        • Ozanne E.M.
        How to define and measure concordance between patients' preferences and medical treatments: a systematic review of approaches and recommendations for standardization.
        Patient Educ Couns. 2010; 78: 12-23
        • King M.
        • Nazareth I.
        • Lampe F.
        • Bower P.
        • Chandler M.
        • Morou M.
        • et al.
        Conceptual framework and systematic review of the effects of participants' and professionals' preferences in randomised controlled trials.
        Health Technol Assess. 2005; 9: 1-186
        • Cronin M.
        • Meaney S.
        • Jepson N.J.
        • Allen P.F.
        A qualitative study of trends in patient preferences for the management of the partially dentate state.
        Gerodontology. 2009; 26: 137-142
        • DeJean D.
        • Giacomini M.
        • Vanstone M.
        • Brundisini F.
        Patient experiences of depression and anxiety with chronic disease: a systematic review and qualitative meta-synthesis.
        Ont Health Technol Assess Ser. 2013; 13: 1-33
        • Atkins D.
        • Best D.
        • Briss P.A.
        • Eccles M.
        • Falck-Ytter Y.
        • Flottorp S.
        • et al.
        Grading quality of evidence and strength of recommendations.
        BMJ. 2004; 328
        • Guyatt G.H.
        • Oxman A.D.
        • Vist G.E.
        • Kunz R.
        • Falck-Ytter Y.
        • Alonso-Coello P.
        • et al.
        GRADE: an emerging consensus on rating quality of evidence and strength of recommendations.
        BMJ. 2008; 336: 924-926
        • Schunemann H.J.
        • Oxman A.D.
        • Brozek J.
        • Glasziou P.
        • Jaeschke R.
        • Vist G.E.
        • et al.
        Grading quality of evidence and strength of recommendations for diagnostic tests and strategies.
        BMJ. 2008; 336: 1106-1110
        • Brunetti M.
        • Shemilt I.
        • Pregno S.
        • Vale L.
        • Oxman A.D.
        • Lord J.
        • et al.
        GRADE guidelines: 10. Considering resource use and rating the quality of economic evidence.
        J Clin Epidemiol. 2013; 66: 140-150
        • Iorio A.
        • Spencer F.A.
        • Falavigna M.
        • Alba C.
        • Lang E.
        • Burnand B.
        • et al.
        Use of GRADE for assessment of evidence about prognosis: rating confidence in estimates of event rates in broad categories of patients.
        BMJ. 2015; 350: h870
        • Lewin S.
        • Glenton C.
        • Munthe-Kaas H.
        • Carlsen B.
        • Colvin C.J.
        • Gulmezoglu M.
        • et al.
        Using qualitative evidence in decision making for health and social interventions: an approach to assess confidence in findings from qualitative evidence syntheses (GRADE-CERQual).
        PLoS Med. 2015; 12
        • Akl E.A.
        • Grant B.J.
        • Guyatt G.H.
        • Montori V.M.
        • Schunemann H.J.
        A decision aid for COPD patients considering inhaled steroid therapy: development and before and after pilot testing.
        BMC Med Inform Decis Mak. 2007; 7: 12
        • Guyatt G.H.
        • Oxman A.D.
        • Kunz R.
        • Falck-Ytter Y.
        • Vist G.E.
        • Liberati A.
        • et al.
        Going from evidence to recommendations.
        BMJ. 2008; 336: 1049-1051
        • Alonso-Coello P.
        • Schünemann H.J.
        • Moberg J.
        • Brignardello-Petersen R.
        • Akl E.A.
        • Davoli M.
        • et al.
        GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 1: Introduction.
        BMJ. 2016; 353: i2016
        • Schunemann H.J.
        • Mustafa R.
        • Brozek J.
        • Santesso N.
        • Alonso-Coello P.
        • Guyatt G.
        • et al.
        GRADE Guidelines: 16. GRADE evidence to decision frameworks for tests in clinical practice and public health.
        J Clin Epidemiol. 2016; 76: 89-98
        • Schunemann H.J.
        • Hill S.R.
        • Kakad M.
        • Vist G.E.
        • Bellamy R.
        • Stockman L.
        • et al.
        Transparent development of the WHO rapid advice guidelines.
        PLoS Med. 2007; 4: e119
        • Kelson M.
        • Akl E.A.
        • Bastian H.
        • Cluzeau F.
        • Curtis J.R.
        • Guyatt G.
        • et al.
        Integrating values and consumer involvement in guidelines with the patient at the center: article 8 in Integrating and coordinating efforts in COPD guideline development. An official ATS/ERS workshop report.
        Proc Am Thorac Soc. 2012; 9: 262-268
        • Balshem H.
        • Helfand M.
        • Schunemann H.J.
        • Oxman A.D.
        • Kunz R.
        • Brozek J.
        • et al.
        GRADE guidelines: 3. Rating the quality of evidence.
        J Clin Epidemiol. 2011; 64: 401-406
        • Yepes-Nuñez J.J.
        • Zhang Y.
        • Xie F.
        • Alonso-Coello P.
        • Selva A.
        • Schünemann H.
        • et al.
        Forty-two systematic reviews generated 23 items for assessing the risk of bias in values and preferences' studies.
        J Clin Epidemiol. 2017; 85: 21-31
        • Guyatt G.H.
        • Oxman A.D.
        • Sultan S.
        • Glasziou P.
        • Akl E.A.
        • Alonso-Coello P.
        • et al.
        GRADE guidelines: 9. Rating up the quality of evidence.
        J Clin Epidemiol. 2011; 64: 1311-1316
        • Guyatt G.
        • Oxman A.D.
        • Akl E.A.
        • Kunz R.
        • Vist G.
        • Brozek J.
        • et al.
        GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables.
        J Clin Epidemiol. 2011; 64: 383-394
        • Karanicolas P.J.
        • Montori V.M.
        • Devereaux P.J.
        • Schunemann H.
        • Guyatt G.H.
        A new “mechanistic-practical” framework for designing and interpreting randomized trials.
        J Clin Epidemiol. 2009; 62: 479-484
        • Levin K.A.
        Study design III: cross-sectional studies.
        Evid Based Dent. 2006; 7: 24-25
        • Fincham J.E.
        Response rates and responsiveness for surveys, standards, and the Journal.
        Am J Pharm Educ. 2008; 72: 43
        • Guyatt G.H.
        • Oxman A.D.
        • Kunz R.
        • Atkins D.
        • Brozek J.
        • Vist G.
        • et al.
        GRADE guidelines: 2. Framing the question and deciding on important outcomes.
        J Clin Epidemiol. 2011; 64: 395-400
        • Guyatt G.H.
        • Oxman A.D.
        • Vist G.
        • Kunz R.
        • Brozek J.
        • Alonso-Coello P.
        • et al.
        GRADE guidelines: 4. Rating the quality of evidence–study limitations (risk of bias).
        J Clin Epidemiol. 2011; 64: 407-415
        • Joy S.M.
        • Little E.
        • Maruthur N.M.
        • Purnell T.S.
        • Bridges J.F.
        Patient preferences for the treatment of type 2 diabetes: a scoping review.
        PharmacoEconomics. 2013; 31: 877-892
        • Guyatt G.H.
        • Oxman A.D.
        • Kunz R.
        • Woodcock J.
        • Brozek J.
        • Helfand M.
        • et al.
        GRADE guidelines: 8. Rating the quality of evidence–indirectness.
        J Clin Epidemiol. 2011; 64: 1303-1310
        • Holbrook A.
        • Labiris R.
        • Goldsmith C.H.
        • Ota K.
        • Harb S.
        • Sebaldt R.J.
        Influence of decision aids on patient preferences for anticoagulant therapy: a randomized trial.
        CMAJ. 2007; 176: 1583-1587
        • Emberton M.
        Medical treatment of benign prostatic hyperplasia: physician and patient preferences and satisfaction.
        Int J Clin Pract. 2010; 64: 1425-1435
        • Brazier J.E.
        • Yang Y.
        • Tsuchiya A.
        • Rowen D.L.
        A review of studies mapping (or cross walking) non-preference based measures of health to generic preference-based measures.
        Eur J Health Econ. 2010; 11: 215-225