World Health Organization strong recommendations based on low-quality evidence (study quality) are frequent and often inconsistent with GRADE guidance

Published:December 19, 2014DOI:



      In 2007 the World Health Organization (WHO) adopted the GRADE system for development of public health guidelines. Previously we found that many strong recommendations issued by WHO are based on evidence for which there is only low or very low confidence in the estimates of effect (discordant recommendations). GRADE guidance indicates that such discordant recommendations are rarely appropriate but suggests five paradigmatic situations in which discordant recommendations may be warranted. We sought to provide insight into the many discordant recommendations in WHO guidelines.

      Study Design and Setting

      We examined all guidelines that used the GRADE method and were approved by the WHO Guideline Review Committee between 2007 and 2012. Teams of reviewers independently abstracted data from eligible guidelines and classified recommendations either into one of the five paradigms for appropriately-formulated discordant recommendations or into three additional categories in which discordant recommendations were inconsistent with GRADE guidance: 1) the evidence warranted moderate or high confidence (a misclassification of evidence) rather than low or very low confidence; 2) good practice statements; or 3) uncertainty in the estimates of effect would best lead to a conditional (weak) recommendation.


      The 33 eligible guidelines included 160 discordant recommendations, of which 98 (61.3%) addressed drug interventions and 132 (82.5%) provided some rationale (though not entirely explicit at times) for the strong recommendation. Of 160 discordant recommendations, 25 (15.6%) were judged consistent with one of the five paradigms for appropriate recommendations; 33 (21%) were based on evidence warranting moderate or high confidence in the estimates of effect; 29 (18%) were good practice statements; and 73 (46%) warranted a conditional, rather than a strong recommendation.


      WHO discordant recommendations are often inconsistent with GRADE guidance, possibly threatening the integrity of the process. Further training in GRADE methods for WHO guideline development group members may be necessary, along with further research on what motivates the formulation of such recommendations.


      To read this article in full you will need to make a payment


      1. World Health Organization (WHO). WHO guidelines approved by the Guidelines Review Committee. Available at Accessed May 27, 2012.

        • Graham R.
        • Mancher M.
        • Wolman D.M.
        • Greenfield S.
        • Steingerg E.
        Committee on Standards for Developing Trustworthy Clinical Practice Guidelines. Institute of Medicine. Clinical Practice Guidelines We Can Trust.
        1st ed. National Academies Press, Washington, DC2011
      2. Institute of Medicine (IOM). Standards for Developing Trustworthy Clinical Practice Guidelines. Standard 2. Management of Conflict of Interest. Available at Accessed January 3, 2015.

        • Oxman A.D.
        • Lavis J.N.
        • Fretheim A.
        Use of evidence in WHO recommendations.
        Lancet. 2007; 369: 1883-1889
      3. WHO Handbook for Guideline Development. 2012. Available at Accessed April 25, 2013.

        • Guyatt G.H.
        • Oxman A.D.
        • Schunemann H.J.
        • Tugwell P.
        • Knottnerus A.
        GRADE guidelines: a new series of articles in the Journal of Clinical Epidemiology.
        J Clin Epidemiol. 2011; 64: 380-382
        • Balshem H.
        • Helfand M.
        • Schünemann H.J.
        • Oxman A.D.
        • Kunz R.
        • Brozek J.
        • et al.
        GRADE guidelines: 3. Rating the quality of evidence.
        J Clin Epidemiol. 2011; 64: 401-406
        • Guyatt G.H.
        • Oxman A.D.
        • Vist G.E.
        • Kunz R.
        • Falck-Ytter Y.
        • Alonso-Coello P.
        • et al.
        • GRADE Working Group
        GRADE: an emerging consensus on rating quality of evidence and strength of recommendations.
        BMJ. 2008; 336: 924-926
        • Guyatt G.
        • Oxman A.D.
        • Akl E.A.
        • Kunz R.
        • Vist G.
        • Brozek J.
        • et al.
        GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables.
        J Clin Epidemiol. 2011; 64: 383-394
        • Andrews J.C.
        • Schünemann H.J.
        • Oxman A.D.
        • Pottie K.
        • Meerpohl J.J.
        • Coello P.A.
        • et al.
        GRADE guidelines: 15. Going from evidence to recommendation-determinants of a recommendation's direction and strength.
        J Clin Epidemiol. 2013; 66: 726-735
        • Brozek J.L.
        • Akl E.A.
        • Alonso-Coello P.
        • Lang D.
        • Jaeschke R.
        • Williams J.W.
        • et al.
        • GRADEWorking Group
        Grading quality of evidence and strength of recommendations in clinical practice guidelines. Part 1 of 3. An overview of the GRADE approach and grading quality of evidence about interventions.
        Allergy. 2009; 64: 669-677
        • WHO
        Guideline: guidelines for the programmatic management of drug-resistant tuberculosis.
        World Health Organization (WHO), Geneva, Switzerland2011 (Available at) (Accessed January 2012)
        • Bhasin S.
        • Cunningham G.R.
        • Hayes F.J.
        • Matsumoto A.M.
        • Snyder P.J.
        • Swerdloff R.S.
        • et al.
        Testosterone therapy in men with androgen deficiency syndromes: an Endocrine Society clinical practice guideline.
        J Clin Endocrinol Metab. 2010; 95: 2536-2559
        • WHO
        Guideline: WHO guidelines for the management of postpartum haemorrhage and retained placenta.
        World Health Organization (WHO), Geneva, Switzerland2009 (Available at) (Accessed January 2012)
        • Bates S.M.
        • Greer I.A.
        • Middeldorp S.
        • Veenstra D.L.
        • Prabulos A.M.
        • Vandvik P.O.
        • et al.
        VTE, thrombophilia, antithrombotic therapy, and pregnancy: antithrombotic therapy and prevention of thrombosis, 9th ed: American College of Chest Physicians evidence-based clinical practice guidelines.
        Chest. 2012; 141: e691S-e736S
        • WHO
        Guideline: rapid advice. Treatment of tuberculosis in children.
        World Health Organization (WHO), Geneva, Switzerland2010 (Available at) (Accessed January 2012)
        • Alexander P.E.
        • Bero L.
        • Montori V.M.
        • Brito J.P.
        • Stoltzfus R.
        • Djulbegovic B.
        • et al.
        World Health Organization recommendations are often strong based on low confidence in effect estimates.
        J Clin Epidemiol. 2014; 67: 629-634
        • Guyatt G.H.
        • Schünemann H.J.
        • Djulbegovic B.
        • Akl E.A.
        Guideline panels should not GRADE good practice statements.
        J Clin Epidemiol. 2014; (in press)
        • Landis J.R.
        • Koch G.G.
        The measurement of observer agreement for categorical data.
        Biometrics. 1977; 33: 159-174
        • Brito J.P.
        • Domecq J.P.
        • Murad M.H.
        • Guyatt G.H.
        • Montori V.M.
        The endocrine society guidelines: when the confidence cart goes before the evidence horse.
        J Clin Endocrinol Metab. 2013; 98: 3246-3252