GRADE Series - Sharon Straus, Rachel Churchill and Sasha Shepperd, Guest Editors| Volume 64, ISSUE 12, P1294-1302, December 01, 2011

GRADE guidelines: 7. Rating the quality of evidence—inconsistency


      This article deals with inconsistency of relative (rather than absolute) treatment effects in binary/dichotomous outcomes. A body of evidence is not rated up in quality if studies yield consistent results, but may be rated down in quality if inconsistent. Criteria for evaluating consistency include similarity of point estimates, extent of overlap of confidence intervals, and statistical criteria including tests of heterogeneity and I2. To explore heterogeneity, systematic review authors should generate and test a small number of a priori hypotheses related to patients, interventions, outcomes, and methodology. When inconsistency is large and unexplained, rating down quality for inconsistency is appropriate, particularly if some studies suggest substantial benefit, and others no effect or harm (rather than only large vs. small effects).
      Apparent subgroup effects may be spurious. Credibility is increased if subgroup effects are based on a small number of a priori hypotheses with a specified direction; subgroup comparisons come from within rather than between studies; tests of interaction generate low P-values; and have a biological rationale.


      To read this article in full you will need to make a payment


        • Furukawa T.A.
        • Guyatt G.H.
        • Griffith L.E.
        Can we individualize the “number needed to treat”? An empirical study of summary effect measures in meta-analyses.
        Int J Epidemiol. 2002; 31: 72-76
        • Deeks J.J.
        Issues in the selection of a summary statistic for meta-analysis of clinical trials with binary outcomes.
        Stat Med. 2002; 21: 1575-1600
        • Schmid C.H.
        • Lau J.
        • McIntosh M.W.
        • Cappelleri J.C.
        An empirical study of the effect of the control rate as a predictor of treatment efficacy in meta-analysis of clinical trials.
        Stat Med. 1998; 17: 1923-1942
        • Guyatt G.
        • Oxman A.D.
        • Kunz R.
        • Atkins D.
        • Brozek J.
        • Vist G.
        • et al.
        Grade guidelines: 2. Framing the question.
        J Clin Epidemiol. 2011; 64: 395-400
      1. Guyatt G, Oxman A, Vist G, Santesso N, Kunz R, et al. Grade guidelines: 12. Preparing summary of findings tables. J Clin Epidemiol [in press].

        • Balshem H.
        • Helfand M.
        • Schünemann H.J.
        • Oxman A.D.
        • Kunz R.
        • Brozek J.
        • et al.
        Grade guidelines: 3 Rating the quality of evidence—introduction.
        J Clin Epidemiol. 2011; 64: 401-406
        • Guyatt G.
        • Jaeschke R.
        • Prasad K.
        • Cook D.
        Summarizing the evidence.
        in: Guyatt G. Rennie D. Meade M. Cook D. The users' guides to the medical literature: a manual for evidence-based clinical practice. McGraw-Hill, New York, NY2008
        • Deeks J.
        • Higgins J.
        • Altman D.
        Analyzing data and undertaking meta-analyses.
        in: Higgins J. Green S. Cochrane handbook for systematic reviews of interventions version 5.0.0. Wiley, Chichester2008
        • Rucker G.
        • Schwarzer G.
        • Carpenter J.R.
        • Schumacher M.
        Undue reliance on I(2) in assessing heterogeneity may mislead.
        BMC Med Res Methodol. 2008; 8: 79
        • Juni P.
        • Nartey L.
        • Reichenbach S.
        • Sterchi R.
        • Dieppe P.A.
        • Egger M.
        Risk of cardiovascular events and rofecoxib: cumulative meta-analysis.
        Lancet. 2004; 364: 2021-2029
        • Guyatt G.
        • Devereaux P.J.
        • Lexchin J.
        • Stone S.B.
        • Yalnizyan A.
        • Himmelstein D.
        • et al.
        A systematic review of studies comparing health outcomes in Canada and the United States.
        Open Med. 2007; 1: e27-e36
        • Alonso-Coello P.
        • Zhou Q.
        • Martinez-Zapata M.J.
        • Mills E.
        • Heels-Ansdell D.
        • Johanson J.F.
        • et al.
        Meta-analysis of flavonoids for the treatment of haemorrhoids.
        Br J Surg. 2006; 93: 909-920
        • Tang B.M.
        • et al.
        Use of calcium or calcium in combination with vitamin D supplementation to prevent fractures and bone loss in people aged 50 years and older: a meta-analysis.
        Lancet. 2007; 370: 657-666
        • Guyatt G.
        • Wyer P.
        • Ioannidis J.
        When to believe a subgroup analysis.
        in: Guyatt G. The users' guides to the medical literature: a manual for evidence-based clinical practice. McGraw-Hill, New York, NY2008
        • Oxman A.D.
        • Cook D.J.
        • Guyatt G.H.
        Users' guides to the medical literature. VI. How to use an overview. Evidence-Based Medicine Working Group.
        JAMA. 1994; 272: 1367-1371
        • Sun X.
        • Briel M.
        • Walter S.D.
        • Guyatt G.H.
        Is a subgroup effect believable? Updating criteria to evaluate the credibility of subgroup analyses.
        BMJ. 2010; 340: c117
        • Briel M.
        • Meade M.
        • Mercat A.
        • Brower R.G.
        • Talmor D.
        • Walter S.D.
        • et al.
        Higher vs lower positive end-expiratory pressure in patients with acute lung injury and acute respiratory distress syndrome: systematic review and meta-analysis.
        JAMA. 2010; 303: 865-873
        • Wyer P.C.
        • Perera P.
        • Jin Z.
        • Zhou Q.
        • Cook D.J.
        • Walter S.D.
        • et al.
        Vasopressin or epinephrine for out-of-hospital cardiac arrest.
        Ann Emerg Med. 2006; 48: 86-97
        • Wenzel V.
        • Krismer A.C.
        • Arntz H.R.
        • Sitter H.
        • Stadlbauer K.H.
        • Lindner K.H.
        • et al.
        A comparison of vasopressin and epinephrine for out-of-hospital cardiopulmonary resuscitation.
        N Engl J Med. 2004; 350: 105-113