Abstract
Objective
To provide Grading of Recommendations, Assessment, Development, and Evaluation (GRADE)
guidance for assessing inconsistency, imprecision, and other domains for the certainty
of evidence about the relative importance of outcomes.
Study Design and Setting
We applied the GRADE domains to rate the certainty of evidence in the importance of
outcomes to several systematic reviews, iteratively reviewed draft guidance, and consulted
GRADE members and other stakeholders for feedback.
Results
We describe the rationale for considering the remaining GRADE domains when rating
the certainty in a body of evidence for the relative importance of outcomes. As meta-analyses
are not common in this context, inconsistency and imprecision assessments are challenging.
Furthermore, confusion exists about inconsistency, imprecision, and true variability
in the relative importance of outcomes. To clarify this issue, we suggest that the
true variability is neither equivalent to inconsistency nor imprecision. Specifically,
inconsistency arises from population, intervention, comparison and outcome and methodological
elements that should be explored and, if possible, explained. The width of the confidence
interval and sample size inform judgments about imprecision. We also provide suggestions
on how to detect publication bias and discuss the domains to rate up the certainty.
Conclusion
We provide guidance and examples for rating inconsistency, imprecision, and other
domains for a body of evidence describing the relative importance of outcomes.
Keywords
To read this article in full you will need to make a payment
Purchase one-time access:
Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online accessOne-time access price info
- For academic or personal research use, select 'Academic and Personal'
- For corporate R&D use, select 'Corporate R&D Professionals'
Subscribe:
Subscribe to Journal of Clinical EpidemiologyAlready a print subscriber? Claim online access
Already an online subscriber? Sign in
Register: Create an account
Institutional Access: Sign in to ScienceDirect
References
- GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables.J Clin Epidemiol. 2011; 64: 383-394
- GRADE guidelines: a new series of articles in the Journal of Clinical Epidemiology.J Clin Epidemiol. 2011; 64: 380-382
- Letters, numbers, symbols and words: how to communicate grades of evidence and recommendations.CMAJ. 2003; 169: 677-680
- Grading quality of evidence and strength of recommendations.BMJ. 2004; : 328
- GRADE guidelines: 3. Rating the quality of evidence.J Clin Epidemiol. 2011; 64: 401-406
- GRADE guidelines: 12. Preparing summary of findings tables-binary outcomes.J Clin Epidemiol. 2013; 66: 158-172
- GRADE guidelines: 13. Preparing summary of findings tables and evidence profiles-continuous outcomes.J Clin Epidemiol. 2013; 66: 173-183
- Improving GRADE evidence tables part 2: a systematic survey of explanatory notes shows more guidance is needed.J Clin Epidemiol. 2016; 74: 19-27
- Improving GRADE evidence tables part 3: detailed guidance for explanatory footnotes supports creating and understanding GRADE certainty in the evidence judgments.J Clin Epidemiol. 2016; 74: 28-39
- GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 1: Introduction.BMJ. 2016; 353: i2016
- GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 2: clinical practice guidelines.BMJ. 2016; 353: i2089
- GRADE guidelines: 14. Going from evidence to recommendations: the significance and presentation of recommendations.J Clin Epidemiol. 2013; 66: 719-725
- GRADE guidelines: 15. Going from evidence to recommendation-determinants of a recommendation's direction and strength.J Clin Epidemiol. 2013; 66: 726-735
- GRADE Guidelines: 16. GRADE evidence to decision frameworks for tests in clinical practice and public health.J Clin Epidemiol. 2016; 76: 89-98
- GRADE evidence to decision (EtD) framework for coverage decisions.Int J Technol Assess Health Care. 2017; 33: 176-182
- Assessing evidence in public health: the added value of GRADE.J Public Health (Oxford, England). 2012; 34: 631-635
- A GRADE Working Group approach for rating the quality of treatment effect estimates from network meta-analysis.BMJ. 2014; 349: g5630
- GRADE: assessing the quality of evidence in environmental and occupational health.Environ Int. 2016; 92-93: 611-616
- Grading quality of evidence and strength of recommendations for diagnostic tests and strategies.BMJ. 2008; 336: 1106-1110
- Use of GRADE for assessment of evidence about prognosis: rating confidence in estimates of event rates in broad categories of patients.BMJ. 2015; 350: h870
- Using GRADE to respond to health questions with different levels of urgency.Environ Int. 2016; 92-93: 585-589
- Transparent development of the WHO rapid advice guidelines.PLoS Med. 2007; 4: e119
- GRADE Guidelines: 18. How ROBINS-I and other tools to assess risk of bias in non-randomized studies should be used to rate the certainty of a body of evidence.J Clin Epidemiol. 2018; ([Epub ahead of print])https://doi.org/10.1016/j.jclinepi.2018.01.012
- GRADE guidelines 17: assessing the risk of bias associated with missing participant outcome data in a body of evidence.J Clin Epidemiol. 2017; 87: 14-22
- Evaluation of patient preference and willingness to pay for attributes of maintenance medication for chronic obstructive pulmonary disease (COPD).Patient. 2014; 7: 413-426
- GRADE guidance for rating the certainty of a body of evidence describing the importance of outcomes or values and preferences: 1. Risk of bias and indirectness.J Clin Epidemiol. 2018; ([Epub ahead of print])https://doi.org/10.1016/j.jclinepi.2018.01.013
- Patient preferences for the treatment of type 2 diabetes: a scoping review.PharmacoEconomics. 2013; 31: 877-892
- Preferences for health states: a review of measurement methods.Mead Johnson Symp Perinat Dev Med. 1982; : 37-45
- Eliciting public preferences for healthcare: a systematic review of techniques.Health Technology Assess. 2001; 5: 1-186
- How to define and measure concordance between patients' preferences and medical treatments: a systematic review of approaches and recommendations for standardization.Patient Educ Couns. 2010; 78: 12-23
- GRADE guidelines: 7. Rating the quality of evidence–inconsistency.J Clin Epidemiol. 2011; 64: 1294-1302
- Undue reliance on I(2) in assessing heterogeneity may mislead.BMC Med Res Methodol. 2008; 8: 79
- The influence of ill-health experience on the valuation of health.Pharmacoeconomics. 1998; 13: 687-696
- Experience-based utility and own health state valuation for a health state classification system: why and how to do it.Eur J Health Econ. 2018; 19: 881-891
- Is a subgroup effect believable? Updating criteria to evaluate the credibility of subgroup analyses.BMJ. 2010; 340: c117
- Elicitation and use of patients' preferences in the treatment of psoriasis: a systematic review.Acta Derm Venereol. 2012; 92: 341-346
- GRADE guidelines 6. Rating the quality of evidence–imprecision.J Clin Epidemiol. 2011; 64: 1283-1293
- Interpreting GRADE's levels of certainty or quality of the evidence: GRADE for statisticians, considering review information size or less emphasis on imprecision?.J Clin Epidemiol. 2016; 75: 6-15
- Use of GRADE grid to reach decisions on clinical practice guidelines when consensus is elusive.BMJ. 2008; 337: a744
- Methodology for the development of antithrombotic therapy and prevention of thrombosis guidelines: antithrombotic therapy and prevention of Thrombosis, 9th ed.: American College of chest Physicians evidence-based clinical practice guidelines.Chest. 2012; 141: 53s-70s
- A systematic review on how patients value chronic obstructive pulmonary disease outcomes.Eur Respir J. 2018; 52
- GRADE guidelines: 5. Rating the quality of evidence–publication bias.J Clin Epidemiol. 2011; 64: 1277-1282
- GRADE guidelines: 9. Rating up the quality of evidence.J Clin Epidemiol. 2011; 64: 1311-1316
- Evaluation of the minimal important difference for the feeling thermometer and the St. George's Respiratory Questionnaire in patients with chronic airflow obstruction.J Clin Epidemiol. 2003; 56: 1170-1176
- Values and preferences for oral antithrombotic therapy in patients with atrial fibrillation: physician and patient perspectives.Health Expect. 2015; 18: 2318-2327
- Cost-effectiveness of warfarin and aspirin for prophylaxis of stroke in patients with nonvalvular atrial fibrillation.JAMA. 1995; 274: 1839-1845
- The effect of stroke and stroke prophylaxis with aspirin or warfarin on quality of life.Arch Intern Med. 1996; 156: 1829-1836
- Patient preference-based treatment thresholds and recommendations: a comparison of decision-analytic modeling with the probability-tradeoff technique.Med Decis Making. 2000; 20: 394-403
- The impact of patients' preferences on the treatment of atrial fibrillation: observational study of patient based decision analysis.BMJ. 2000; 320: 1380-1384
- Thrombolytic treatment for stroke: patient preferences for treatment, information, and involvement.J Stroke Cerebrovasc Dis. 2009; 18: 17-22
- Decision analysis and guidelines for anticoagulant therapy to prevent stroke in patients with atrial fibrillation.Lancet. 2000; 355: 956-962
Article info
Publication history
Published online: May 22, 2018
Accepted:
May 3,
2018
Footnotes
Ethics approval and consent to participate: Not required. This study does not involve de novo patient data collection. No patient informed consent and institutional review board approval have been sought.
Consent for publication: Not applicable.
Availability of data and materials: The data sets supporting the conclusions of this article are included within the article and its additional file.
Funding: This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors. It was funded through internal research funds at McMaster University available to HJS. GH is supported by a CIHR New Investigator Salary Award and a The Arthritis Society Young Investigator Salary Award, neither of which is directly related to this research project.
Conflict of interest: All authors have completed the ICMJE uniform disclosure form at http://www.icmje.org/conflicts-of-interest/); no financial relationships with any organizations that might have an interest in the submitted work in the previous 3 years, no other relationships or activities that could appear to have influenced the submitted work. Authors are members of the GRADE Working Group.
Identification
Copyright
© 2018 Published by Elsevier Inc.
