Advertisement

GRADE guidelines: 20. Assessing the certainty of evidence in the importance of outcomes or values and preferences—inconsistency, imprecision, and other domains

      Abstract

      Objective

      To provide Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) guidance for assessing inconsistency, imprecision, and other domains for the certainty of evidence about the relative importance of outcomes.

      Study Design and Setting

      We applied the GRADE domains to rate the certainty of evidence in the importance of outcomes to several systematic reviews, iteratively reviewed draft guidance, and consulted GRADE members and other stakeholders for feedback.

      Results

      We describe the rationale for considering the remaining GRADE domains when rating the certainty in a body of evidence for the relative importance of outcomes. As meta-analyses are not common in this context, inconsistency and imprecision assessments are challenging. Furthermore, confusion exists about inconsistency, imprecision, and true variability in the relative importance of outcomes. To clarify this issue, we suggest that the true variability is neither equivalent to inconsistency nor imprecision. Specifically, inconsistency arises from population, intervention, comparison and outcome and methodological elements that should be explored and, if possible, explained. The width of the confidence interval and sample size inform judgments about imprecision. We also provide suggestions on how to detect publication bias and discuss the domains to rate up the certainty.

      Conclusion

      We provide guidance and examples for rating inconsistency, imprecision, and other domains for a body of evidence describing the relative importance of outcomes.

      Keywords

      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'

      Subscribe:

      Subscribe to Journal of Clinical Epidemiology
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect

      References

        • Guyatt G.
        • Oxman A.D.
        • Akl E.A.
        • Kunz R.
        • Vist G.
        • Brozek J.
        • et al.
        GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables.
        J Clin Epidemiol. 2011; 64: 383-394
        • Guyatt G.H.
        • Oxman A.D.
        • Schunemann H.J.
        • Tugwell P.
        • Knottnerus A.
        GRADE guidelines: a new series of articles in the Journal of Clinical Epidemiology.
        J Clin Epidemiol. 2011; 64: 380-382
        • Schunemann H.J.
        • Best D.
        • Vist G.
        • Oxman A.D.
        Letters, numbers, symbols and words: how to communicate grades of evidence and recommendations.
        CMAJ. 2003; 169: 677-680
        • Atkins D.
        • Best D.
        • Briss P.A.
        • Eccles M.
        • Falck-Ytter Y.
        • Flottorp S.
        • et al.
        Grading quality of evidence and strength of recommendations.
        BMJ. 2004; : 328
        • Balshem H.
        • Helfand M.
        • Schunemann H.J.
        • Oxman A.D.
        • Kunz R.
        • Brozek J.
        • et al.
        GRADE guidelines: 3. Rating the quality of evidence.
        J Clin Epidemiol. 2011; 64: 401-406
        • Guyatt G.H.
        • Oxman A.D.
        • Santesso N.
        • Helfand M.
        • Vist G.
        • Kunz R.
        • et al.
        GRADE guidelines: 12. Preparing summary of findings tables-binary outcomes.
        J Clin Epidemiol. 2013; 66: 158-172
        • Guyatt G.H.
        • Thorlund K.
        • Oxman A.D.
        • Walter S.D.
        • Patrick D.
        • Furukawa T.A.
        • et al.
        GRADE guidelines: 13. Preparing summary of findings tables and evidence profiles-continuous outcomes.
        J Clin Epidemiol. 2013; 66: 173-183
        • Langendam M.
        • Carrasco-Labra A.
        • Santesso N.
        • Mustafa R.A.
        • Brignardello-Petersen R.
        • Ventresca M.
        • et al.
        Improving GRADE evidence tables part 2: a systematic survey of explanatory notes shows more guidance is needed.
        J Clin Epidemiol. 2016; 74: 19-27
        • Santesso N.
        • Carrasco-Labra A.
        • Langendam M.
        • Brignardello-Petersen R.
        • Mustafa R.A.
        • Heus P.
        • et al.
        Improving GRADE evidence tables part 3: detailed guidance for explanatory footnotes supports creating and understanding GRADE certainty in the evidence judgments.
        J Clin Epidemiol. 2016; 74: 28-39
        • Alonso-Coello P.
        • Schünemann H.J.
        • Moberg J.
        • Brignardello-Petersen R.
        • Akl E.A.
        • Davoli M.
        • et al.
        GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 1: Introduction.
        BMJ. 2016; 353: i2016
        • Alonso-Coello P.
        • Oxman A.D.
        • Moberg J.
        • Brignardello-Petersen R.
        • Akl E.A.
        • Davoli M.
        • et al.
        GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 2: clinical practice guidelines.
        BMJ. 2016; 353: i2089
        • Andrews J.
        • Guyatt G.
        • Oxman A.D.
        • Alderson P.
        • Dahm P.
        • Falck-Ytter Y.
        • et al.
        GRADE guidelines: 14. Going from evidence to recommendations: the significance and presentation of recommendations.
        J Clin Epidemiol. 2013; 66: 719-725
        • Andrews J.C.
        • Schunemann H.J.
        • Oxman A.D.
        • Pottie K.
        • Meerpohl J.J.
        • Coello P.A.
        • et al.
        GRADE guidelines: 15. Going from evidence to recommendation-determinants of a recommendation's direction and strength.
        J Clin Epidemiol. 2013; 66: 726-735
        • Schunemann H.J.
        • Mustafa R.
        • Brozek J.
        • Santesso N.
        • Alonso-Coello P.
        • Guyatt G.
        • et al.
        GRADE Guidelines: 16. GRADE evidence to decision frameworks for tests in clinical practice and public health.
        J Clin Epidemiol. 2016; 76: 89-98
        • Parmelli E.
        • Amato L.
        • Oxman A.D.
        • Alonso-Coello P.
        • Brunetti M.
        • Moberg J.
        • et al.
        GRADE evidence to decision (EtD) framework for coverage decisions.
        Int J Technol Assess Health Care. 2017; 33: 176-182
        • Burford B.J.
        • Rehfuess E.
        • Schunemann H.J.
        • Akl E.A.
        • Waters E.
        • Armstrong R.
        • et al.
        Assessing evidence in public health: the added value of GRADE.
        J Public Health (Oxford, England). 2012; 34: 631-635
        • Puhan M.A.
        • Schunemann H.J.
        • Murad M.H.
        • Li T.
        • Brignardello-Petersen R.
        • Singh J.A.
        • et al.
        A GRADE Working Group approach for rating the quality of treatment effect estimates from network meta-analysis.
        BMJ. 2014; 349: g5630
        • Morgan R.L.
        • Thayer K.A.
        • Bero L.
        • Bruce N.
        • Falck-Ytter Y.
        • Ghersi D.
        • et al.
        GRADE: assessing the quality of evidence in environmental and occupational health.
        Environ Int. 2016; 92-93: 611-616
        • Schunemann H.J.
        • Oxman A.D.
        • Brozek J.
        • Glasziou P.
        • Jaeschke R.
        • Vist G.E.
        • et al.
        Grading quality of evidence and strength of recommendations for diagnostic tests and strategies.
        BMJ. 2008; 336: 1106-1110
        • Iorio A.
        • Spencer F.A.
        • Falavigna M.
        • Alba C.
        • Lang E.
        • Burnand B.
        • et al.
        Use of GRADE for assessment of evidence about prognosis: rating confidence in estimates of event rates in broad categories of patients.
        BMJ. 2015; 350: h870
        • Thayer K.A.
        • Schunemann H.J.
        Using GRADE to respond to health questions with different levels of urgency.
        Environ Int. 2016; 92-93: 585-589
        • Schunemann H.J.
        • Hill S.R.
        • Kakad M.
        • Vist G.E.
        • Bellamy R.
        • Stockman L.
        • et al.
        Transparent development of the WHO rapid advice guidelines.
        PLoS Med. 2007; 4: e119
        • Schunemann H.
        • Cuello-Garcia C.
        • Akl E.A.
        • Mustafa R.
        • Meerpohl J.
        • Thayer K.
        • et al.
        GRADE Guidelines: 18. How ROBINS-I and other tools to assess risk of bias in non-randomized studies should be used to rate the certainty of a body of evidence.
        J Clin Epidemiol. 2018; ([Epub ahead of print])https://doi.org/10.1016/j.jclinepi.2018.01.012
        • Guyatt G.H.
        • Ebrahim S.
        • Alonso-Coello P.
        • Johnston B.C.
        • Mathioudakis A.G.
        • Briel M.
        • et al.
        GRADE guidelines 17: assessing the risk of bias associated with missing participant outcome data in a body of evidence.
        J Clin Epidemiol. 2017; 87: 14-22
        • Kawata A.K.
        • Kleinman L.
        • Harding G.
        • Ramachandran S.
        Evaluation of patient preference and willingness to pay for attributes of maintenance medication for chronic obstructive pulmonary disease (COPD).
        Patient. 2014; 7: 413-426
        • Zhang Y.
        • P A.-C.
        • Guyatt G.
        • Yepes-Nuñez J.J.
        • Akl E.
        • Hazlewood G.
        • et al.
        GRADE guidance for rating the certainty of a body of evidence describing the importance of outcomes or values and preferences: 1. Risk of bias and indirectness.
        J Clin Epidemiol. 2018; ([Epub ahead of print])https://doi.org/10.1016/j.jclinepi.2018.01.013
        • Joy S.M.
        • Little E.
        • Maruthur N.M.
        • Purnell T.S.
        • Bridges J.F.
        Patient preferences for the treatment of type 2 diabetes: a scoping review.
        PharmacoEconomics. 2013; 31: 877-892
        • Torrance G.W.
        Preferences for health states: a review of measurement methods.
        Mead Johnson Symp Perinat Dev Med. 1982; : 37-45
        • Ryan M.
        • Scott D.A.
        • Reeves C.
        • Bate A.
        • Van Teijlingen E.R.
        • Russell E.M.
        • et al.
        Eliciting public preferences for healthcare: a systematic review of techniques.
        Health Technology Assess. 2001; 5: 1-186
        • Sepucha K.
        • Ozanne E.M.
        How to define and measure concordance between patients' preferences and medical treatments: a systematic review of approaches and recommendations for standardization.
        Patient Educ Couns. 2010; 78: 12-23
        • Guyatt G.H.
        • Oxman A.D.
        • Kunz R.
        • Woodcock J.
        • Brozek J.
        • Helfand M.
        • et al.
        GRADE guidelines: 7. Rating the quality of evidence–inconsistency.
        J Clin Epidemiol. 2011; 64: 1294-1302
        • Rucker G.
        • Schwarzer G.
        • Carpenter J.R.
        • Schumacher M.
        Undue reliance on I(2) in assessing heterogeneity may mislead.
        BMC Med Res Methodol. 2008; 8: 79
        • Badia X.
        • Herdman M.
        • Kind P.
        The influence of ill-health experience on the valuation of health.
        Pharmacoeconomics. 1998; 13: 687-696
        • Brazier J.
        • Rowen D.
        • Karimi M.
        • Peasgood T.
        • Tsuchiya A.
        • Ratcliffe J.
        Experience-based utility and own health state valuation for a health state classification system: why and how to do it.
        Eur J Health Econ. 2018; 19: 881-891
        • Sun X.
        • Briel M.
        • Walter S.D.
        • Guyatt G.H.
        Is a subgroup effect believable? Updating criteria to evaluate the credibility of subgroup analyses.
        BMJ. 2010; 340: c117
        • Umar N.
        • Yamamoto S.
        • Loerbroks A.
        • Terris D.
        Elicitation and use of patients' preferences in the treatment of psoriasis: a systematic review.
        Acta Derm Venereol. 2012; 92: 341-346
        • Guyatt G.H.
        • Oxman A.D.
        • Kunz R.
        • Brozek J.
        • Alonso-Coello P.
        • Rind D.
        • et al.
        GRADE guidelines 6. Rating the quality of evidence–imprecision.
        J Clin Epidemiol. 2011; 64: 1283-1293
        • Schunemann H.J.
        Interpreting GRADE's levels of certainty or quality of the evidence: GRADE for statisticians, considering review information size or less emphasis on imprecision?.
        J Clin Epidemiol. 2016; 75: 6-15
        • Jaeschke R.
        • Guyatt G.H.
        • Dellinger P.
        • Schunemann H.
        • Levy M.M.
        • Kunz R.
        • et al.
        Use of GRADE grid to reach decisions on clinical practice guidelines when consensus is elusive.
        BMJ. 2008; 337: a744
        • Guyatt G.H.
        • Norris S.L.
        • Schulman S.
        • Hirsh J.
        • Eckman M.H.
        • Akl E.A.
        • et al.
        Methodology for the development of antithrombotic therapy and prevention of thrombosis guidelines: antithrombotic therapy and prevention of Thrombosis, 9th ed.: American College of chest Physicians evidence-based clinical practice guidelines.
        Chest. 2012; 141: 53s-70s
        • Zhang Y.
        • Morgan R.
        • Alonso-Coello P.
        • Wiercioch W.
        • Bała M.
        • Jaeschke R.
        • et al.
        A systematic review on how patients value chronic obstructive pulmonary disease outcomes.
        Eur Respir J. 2018; 52
        • Guyatt G.H.
        • Oxman A.D.
        • Montori V.
        • Vist G.
        • Kunz R.
        • Brozek J.
        • et al.
        GRADE guidelines: 5. Rating the quality of evidence–publication bias.
        J Clin Epidemiol. 2011; 64: 1277-1282
        • Guyatt G.H.
        • Oxman A.D.
        • Sultan S.
        • Glasziou P.
        • Akl E.A.
        • Alonso-Coello P.
        • et al.
        GRADE guidelines: 9. Rating up the quality of evidence.
        J Clin Epidemiol. 2011; 64: 1311-1316
        • Schunemann H.J.
        • Griffith L.
        • Jaeschke R.
        • Goldstein R.
        • Stubbing D.
        • Guyatt G.H.
        Evaluation of the minimal important difference for the feeling thermometer and the St. George's Respiratory Questionnaire in patients with chronic airflow obstruction.
        J Clin Epidemiol. 2003; 56: 1170-1176
        • Alonso-Coello P.
        • Montori V.M.
        • Diaz M.G.
        • Devereaux P.J.
        • Mas G.
        • Diez A.I.
        • et al.
        Values and preferences for oral antithrombotic therapy in patients with atrial fibrillation: physician and patient perspectives.
        Health Expect. 2015; 18: 2318-2327
        • Gage B.F.
        • Cardinalli A.B.
        • Albers G.W.
        • Owens D.K.
        Cost-effectiveness of warfarin and aspirin for prophylaxis of stroke in patients with nonvalvular atrial fibrillation.
        JAMA. 1995; 274: 1839-1845
        • Gage B.F.
        • Cardinalli A.B.
        • Owens D.K.
        The effect of stroke and stroke prophylaxis with aspirin or warfarin on quality of life.
        Arch Intern Med. 1996; 156: 1829-1836
        • Man-Son-Hing M.
        • Laupacis A.
        • O'Connor A.M.
        • Coyle D.
        • Berquist R.
        • McAlister F.
        Patient preference-based treatment thresholds and recommendations: a comparison of decision-analytic modeling with the probability-tradeoff technique.
        Med Decis Making. 2000; 20: 394-403
        • Protheroe J.
        • Fahey T.
        • Montgomery A.A.
        • Peters T.J.
        The impact of patients' preferences on the treatment of atrial fibrillation: observational study of patient based decision analysis.
        BMJ. 2000; 320: 1380-1384
        • Slot K.B.
        • Berge E.
        Thrombolytic treatment for stroke: patient preferences for treatment, information, and involvement.
        J Stroke Cerebrovasc Dis. 2009; 18: 17-22
        • Thomson R.
        • Parkin D.
        • Eccles M.
        • Sudlow M.
        • Robinson A.
        Decision analysis and guidelines for anticoagulant therapy to prevent stroke in patients with atrial fibrillation.
        Lancet. 2000; 355: 956-962