Study Design and Setting
- •In systematic reviews and guidelines The Grading of Recommendations Assessment, Development and Evaluation (GRADE) now suggests aligning approach that relies on thresholds and CIs of the absolute effect (i.e., CI approach) as a primary criterion for imprecision rating.
- •Using the CI approach, when the CI is wide and considerably cross the threshold(s) of interests (i.e., one or both boundaries of CIs suggest inferences appreciably different from point estimate), one should consider rating down two levels for imprecision, and when the CI is very wide that the two boundaries of CI suggest very different inferences, one should consider rating down three levels for imprecision.
- •Using the OIS approach, for dichotomous outcomes, one should consider rating down two levels for imprecision, when the ratio of the upper to the lower boundary of the CI is more than 2.5 for odds ratio or three for risk ratio; for continuous outcomes, when the sample size is smaller than 30–50% of the OIS.
- •When the baseline risk is very low, GRADE suggests being more restrained in rating down for imprecision.
- •Building on prior GRADE guidance on imprecision rating, this article provides specific guidance on circumstances when, using a minimally contextualized approach, one should consider rating down more than one level for imprecision based on the CI approach and the OIS approach.
What this adds to what is known?
- •The article alerts GRADE users to the merits of using the same CI approach to imprecision in both systematic reviews and guidelines.
- •GRADE users should seriously consider rating down imprecision by more than one level when the CI appreciably crosses the threshold(s) of interest; or when the CI does not cross the threshold(s) and the relative effect is large, the sample size of meta-analysis is far less than the OIS.
What is the implication, what should change now?
- 1.Systematic reviews are most useful for target audiences when authors make judgements in relation to possible thresholds. Thus, authors of systematic reviews are much more likely to use the approach that relies on thresholds and CIs around the absolute effect (i.e., CI approach) than OIS to judge imprecision (i.e., OIS approach).
- 2.When the CI appreciably crosses the threshold(s) of interest, authors of systematic reviews and guidelines should be more inclined to consider rating down more than one level (i.e., two or three levels) for imprecision.
- Using a minimally contextualized approach (typically in systematic reviews), authors consider only one outcome at a time. Authors rate their certainty in relation to the null—rating their certainty that an effect is truly present—or in relation to a minimally important difference (MID)—rating their certainty that an important effect is truly present.
- Using a partially contextualized approach, authors rate their certainty that the true effect falls in a range representing a trivial, small, moderate, or large effects for one outcome at a time.
- In a fully contextualized approach (typically used in guidelines), authors simultaneously consider multiple outcomes (i.e., trading off desirable vs. undesirable health effects of an intervention) and set a decision threshold above which they would recommend in favor of an intervention and below which they would recommend against it.
2. GRADE now suggests aligning imprecision criteria for systematic reviews and guidelines using confidence interval approach as a primary criterion for imprecision rating
- Consider a systematic review of corticosteroids vs. no corticosteroids for patients with sepsis []. A meta-analysis of randomized controlled trials reports that corticosteroids yielded 2.2 fewer deaths per 100 patients with a confidence interval (CI) from 4.1 fewer to 0 fewer (Fig. 1) []. Considering the importance of the outcome and using a minimally contextualized approach, the authors could have set an minimally important difference (MID; i.e., the threshold of interest) at a reduction of 0.5 deaths per 100 patients (i.e., 5 deaths per 1,000 patients).
- Because the point estimate falls above the MID, authors would rate their certainty that corticosteroids result in an important reduction in death (i.e., the target of certainty rating) []. Because the CI crosses the MID of 0.5% (i.e., the effect of corticosteroids might be trivial), authors would rate down at least one level for imprecision. Should the authors rate down two levels for imprecision? The answer is probably not. First, the extent to which the CI crosses the threshold is relatively modest—0.5 per 100. Second, the CI does not include an increase in deaths with corticosteroids. A misguided conclusion of benefit would not, therefore, put patients and clinicians at risk of administration of a lethal intervention. The authors would therefore conclude that corticosteroids probably result in an important reduction in death for patients with sepsis.
3. Based on confidence interval approach GRADE now suggests more frequent rating down two levels for imprecision
- When high certainty evidence exists, authors of systematic reviews and guidelines can summarize, “effects present”; when moderate certainty evidence exists, authors can conclude, “effects probably or likely present”; and when low quality evidence exists, the plain language summary is “effects possibly present.” When the certainty of evidence is very low, authors can make a statement indicating that the evidence is very uncertain [].
- Authors may be appropriately uncomfortable with a plain language summary resulting from rating down only one level for imprecision (e.g., an intervention “probably” has an important effect) when, for instance, a CI includes appreciable harm. In such instances, they may be more comfortable with a summary resulting from rating down two levels for imprecision (i.e., an intervention “may” have an important effect) or a summary resulting from rating down three levels for imprecision (i.e., the evidence is very uncertain about the effect of the intervention).
3.1 Example 1 [an effect of important benefit, with possibility of important harm]
- 1.When rating the certainty that there is a true important benefit, the point estimate reflects an important benefit, and the boundary of the CI least favorable to the intervention includes the possibility of harm, particularly important harm (Example 1).
- 2.When rating the certainty that there is a true important harm, the point estimate reflects an important harm, and the boundary of the CI most favorable to the intervention includes the possibility of benefit, particularly important benefit (Example 2).
Rating certainty in relation to an minimally important difference threshold when point estimate suggests an important effect (circumstance 1–2)
- 3.When rating the certainty that the true effect is a trivial or no effect, the point estimate is consistent with a trivial effect, and the CI includes the possibility of both important benefit and important harm (Example 3).
- 4.When rating the certainty that the true effect is a trivial or no effect, the point estimate is consistent with a trivial effect, and the CI includes the possibility of substantial (possibly large) important harm (Example 4).
- 5.When rating the certainty that the true effect is a trivial or no effect, the point estimate is consistent with a trivial effect, and the CI includes the possibility of substantial (possibly large) important benefit. (Example 5).
Rating certainty in relation to minimally important difference thresholds when point estimate suggests a trivial effect (circumstance 3–5)
- 6.When rating the certainty of nonzero benefit, the point estimate suggests benefit, and the CI includes the possibility of important harm (Example 6).
- 7.When rating the certainty of nonzero harm, the point estimate suggests harm, and the CI includes the possibility of important benefit (Example 7).
Rating certainty in relation to the null effect threshold (circumstance 6–7)
3.2 Example 2 [an effect of important harm, with possibility of important benefit]
3.3 Example 3 [trivial effect, with possibility of important benefit and important harm]
3.4 Example 4 [trivial effect, with possibility of substantial important harm]
3.5 Example 5 [trivial effect, with possibility of substantial important benefit]
- Lozano L.E.C.
- Nampo F.K.
- Agarwal A.
- Desai P.
- Litzow M.
- Sekeres M.A.
- et al.
3.6 Example 6 [an effect of benefit, with possibility of important harm]
3.7 Example 7 [an effect of harm, with possibility of important benefit]
4. When the confidence interval does not cross threshold(s) of interest and the relative effect is large, GRADE suggests considering whether the optimal information size is met
4.1 When systematic review and guideline authors should check optimal information size
4.2 Based on optimal information size calculation when GRADE suggests rating down two levels for imprecision
- 1.For dichotomous outcomes, when the ratio of the upper to the lower boundary of the CI is more than 2.5 for odds ratio or three for risk ratioR (Appendix A, Example 1).
- 2.For continuous outcomes, when the total sample size of a meta-analysis is smaller than 30–50% of the OIS (Appendix A, Example 2).
4.3 When the baseline risk is very low, GRADE suggests being more restrained in rating down for imprecision
5. Based on confidence interval approach when GRADE suggests rating down three levels for imprecision
Appendix A. Supplementary Data
- Appendix Table
- Appendix Figure
- GRADE guidelines 6. Rating the quality of evidence—imprecision.J Clin Epidemiol. 2011; 64: 1283-1293
- The GRADE Working Group clarifies the construct of certainty of evidence.J Clin Epidemiol. 2017; 87: 4-13
- Corticosteroids in sepsis: an updated systematic review and meta-analysis.Crit Care Med. 2018; 46: 1411-1420
- GRADE guidelines 32: GRADE offers guidance on choosing targets of GRADE certainty of evidence ratings.J Clin Epidemiol. 2021; 137: 163-175
- Improving GRADE evidence tables part 1: a randomized trial shows improved understanding of content in summary of findings tables with a new format.J Clin Epidemiol. 2016; 74: 7-18
- GRADE guidelines 26: informative statements to communicate the findings of systematic reviews of interventions.J Clin Epidemiol. 2020; 119: 126-135
- Transcatheter versus surgical aortic valve replacement in patients with severe aortic stenosis at low and intermediate risk: systematic review and meta-analysis.BMJ. 2016; 354: i5130
- Association of thyroid hormone therapy with quality of life and thyroid-related symptoms in patients with subclinical hypothyroidism: a systematic review and meta-analysis.JAMA. 2018; 320: 1349-1359
- Less intensive antileukemic therapies (monotherapy and/or combination) for older adults with acute myeloid leukemia who are not candidates for intensive antileukemic therapy: a systematic review and meta-analysis.PLoS One. 2011; 17: e0263240
- The comparative efficacy and safety of alternative glucocorticoids regimens in patients with ANCA-associated vasculitis: a systematic review.BMJ Open. 2011; 12: e050507
- Low intensity pulsed ultrasound for bone healing: systematic review of randomized controlled trials.BMJ. 2017; 356: j656
- GRADE guidelines 33: addressing imprecision in a network meta-analysis.J Clin Epidemiol. 2021; 139: 49-56
- A power primer.Psychol Bull. 1992; 112: 155-159
- GRADE Guidance article: rating down for imprecision in the partially or fully contextualized approaches to assessing certainty of evidence.J Clin Epidemiol. 2022; (In press)
Publication stageIn Press Journal Pre-Proof
Funding: L.Z. is funded by the Science and Technology Plan Project of Sichuan Province ( 2020YFS0035 ).
Conflict of interest: All authors of this paper are GRADE Working Group members. Gordon H. Guyatt and Holger J. Schünemann are the co-founders and co-chairs of GRADE Working Group.
Author Contributions: G.H.G., R.B.P., L.Z., M.H., and H.J.S. conceptualized this work and collected examples. L.Z., R.B.P., and G.H.G. drafted the manuscript. R.A.M., M.H.M., A.I., G.T., E.A.A., and M.M. provided input that resulted in important modifications. All authors approved the final manuscript.
User LicenseCreative Commons Attribution (CC BY 4.0) |
- Read, print & download
- Redistribute or republish the final article
- Text & data mine
- Translate the article
- Reuse portions or extracts from the article in other works
- Sell or re-use for commercial purposes
Elsevier's open access license policy