Original Article| Volume 108, P77-85, April 2019

GRADE approach to rate the certainty from a network meta-analysis: addressing incoherence

Published:December 05, 2018DOI:


      This article presents official guidance from the Grading of Recommendations Assessments, Development, and Evaluation (GRADE) working group on how to address incoherence when assessing the certainty in the evidence from network meta-analysis. Incoherence represents important differences between direct and indirect estimates that contribute to a network estimate. Bias due to limitations in study design or publication bias, indirectness, and intransitivity may be responsible for incoherence. Addressing incoherence requires a judgment regarding the importance of the impact on the network estimate. Reviewers need to be alert to the possibility of misguidedly arriving at excessively low ratings of certainty by rating down for both incoherence and other closely related GRADE domains. This article describes and illustrates each of these issues and provides explicit guidance on how to deal with them.


      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to Journal of Clinical Epidemiology
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Brignardello-Petersen R.
        • Bonner A.
        • Alexander P.E.
        • Siemieniuk R.A.
        • Furukawa T.A.
        • Rochwerg B.
        • et al.
        Advances in the GRADE approach to rate the certainty in estimates from a network meta-analysis.
        J Clin Epidemiol. 2017; 93: 36-44
        • Puhan M.A.
        • Schunemann H.J.
        • Murad M.H.
        • Li T.
        • Brignardello-Petersen R.
        • Singh J.A.
        • et al.
        A GRADE Working Group approach for rating the quality of treatment effect estimates from network meta-analysis.
        BMJ. 2014; 349: g5630
        • Lu G.
        • Ades A.
        Assessing evidence inconsistency in mixed treatment comparisons.
        J Am Stat Soc. 2006; 101: 447-459
        • Guyatt G.H.
        • Oxman A.D.
        • Kunz R.
        • Woodcock J.
        • Brozek J.
        • Helfand M.
        • et al.
        GRADE guidelines: 7. Rating the quality of evidence--inconsistency.
        J Clin Epidemiol. 2011; 64: 1294-1302
        • Foote C.J.
        • Guyatt G.H.
        • Vignesh K.N.
        • Mundi R.
        • Chaudhry H.
        • Heels-Ansdell D.
        • et al.
        Which surgical treatment for open tibial shaft fractures results in the fewest reoperations? A network meta-analysis.
        Clin Orthop Relat Res. 2015; 473: 2179-2192
        • Hultcrantz M.
        • Rind D.
        • Akl E.A.
        • Treweek S.
        • Mustafa R.A.
        • Iorio A.
        • et al.
        The GRADE Working Group clarifies the construct of certainty of evidence.
        J Clin Epidemiol. 2017; 87: 4-13
        • Guyatt G.H.
        • Oxman A.D.
        • Kunz R.
        • Brozek J.
        • Alonso-Coello P.
        • Rind D.
        • et al.
        GRADE guidelines 6. Rating the quality of evidence--imprecision.
        J Clin Epidemiol. 2011; 64: 1283-1293
        • Guyatt G.H.
        • Oxman A.D.
        • Sultan S.
        • Glasziou P.
        • Akl E.A.
        • Alonso-Coello P.
        • et al.
        GRADE guidelines: 9. Rating up the quality of evidence.
        J Clin Epidemiol. 2011; 64: 1311-1316
        • Brown T.J.
        • Hooper L.
        • Elliott R.A.
        • Payne K.
        • Webb R.
        • Roberts C.
        • et al.
        A comparison of the cost-effectiveness of five strategies for the prevention of non-steroidal anti-inflammatory drug-induced gastrointestinal toxicity: a systematic review with economic modelling.
        Health Technol Assess. 2006; 10 (xi–xiii, 1–183): iii-iv
        • Efthimiou O.
        • Debray T.P.
        • van Valkenhoef G.
        • Trelle S.
        • Panayidou K.
        • Moons K.G.
        • et al.
        GetReal in network meta-analysis: a review of the methodology.
        Res Synth Methods. 2016; 7: 236-263
        • Salanti G.
        Indirect and mixed-treatment comparison, network, or multiple-treatments meta-analysis: many names, many benefits, many concerns for the next generation evidence synthesis tool.
        Res Synth Methods. 2012; 3: 80-97
        • Cipriani A.
        • Furukawa T.A.
        • Salanti G.
        • Geddes J.R.
        • Higgins J.P.
        • Churchill R.
        • et al.
        Comparative efficacy and acceptability of 12 new-generation antidepressants: a multiple-treatments meta-analysis.
        Lancet. 2009; 373: 746-758