AHRQ Series Paper 2: Principles for developing guidance: AHRQ and the Effective Health-Care Program

  • Mark Helfand
    Oregon Evidence-based Practice Center, Department of Medical Informatics and Clinical Epidemiology, Oregon Health & Science University, 3181 SW Sam Jackson Park Road, Portland, OR 97239, USA

    Department of Hospital and Specialty Medicine, Portland VA Medical Center, 3710 SW U.S. Veterans Hospital Road, Portland, OR 97239, USA
    Search for articles by this author
  • Howard Balshem
    Corresponding author. Tel.: +1-503-418-1964; fax: +1-503-418-3332.
    Oregon Evidence-based Practice Center, Department of Medical Informatics and Clinical Epidemiology, Oregon Health & Science University, 3181 SW Sam Jackson Park Road, Portland, OR 97239, USA
    Search for articles by this author



      This article describes some of the fundamental principles that have been developed to guide the work of producing comparative effectiveness reviews (CERs).

      Study Design and Setting

      We briefly describe the role stakeholders play in providing important insights that inform the evidence-gathering process, and discuss the critical role of analytic frameworks in illuminating the relationship between surrogate measures and health outcomes, providing an understanding of the context in which clinical decisions are made and the uncertainties that underlie clinical controversies.


      We describe the Effective Health Care program conceptual model for considering different types of evidence that emphasizes minimizing the risk of bias, but places high-quality, highly applicable evidence about effectiveness at the top of the hierarchy. Finally, we briefly describe areas of future methodological research.


      CERs have become a foundation for decision-making in clinical practice and health policy. To be useful, CERs must approach the evidence from a patient-centered perspective; explore the clinical logic underlying the rationale for a service; cast a broad net with respect to types of evidence, placing a high value on effectiveness and applicability, in addition to internal validity; and, present benefits and harms for treatments and tests in a consistent way.


      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to Journal of Clinical Epidemiology
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Woolf S.H.
        Manual for conducting systematic reviews.
        Agency for Health Care Policy and Research: AHRQ, 1996
        • AHRQ
        Suggesting a topic for effective health care research.
        (Updated 2009. Available at) (Accessed April 27, 2009)
        • Aschengrau A.
        • Seage G.R.
        Essentials of epidemiology in public health.
        Bartlett & Jones, Boston, MA2003
        • Mrkobrada M.
        • Thiessen-Philbrook H.
        • Haynes R.B.
        • Iansavichus A.V.
        • Rehman F.
        • Garg A.X.
        Need for quality improvement in renal systematic reviews.
        Clin J Am Soc Nephrol. 2008; 3: 1102-1114
        • Shrier I.
        • Boivin J.F.
        • Platt R.W.
        • Steele R.J.
        • Brophy J.M.
        • Carnevale F.
        • et al.
        The interpretation of systematic reviews with meta-analyses: an objective or subjective process?.
        BMC Med Inform Decis Mak. 2008; 8: 19
        • Egger M.
        • Smith G.D.
        Principles of and procedures for systematic reviews.
        in: Egger M. Smith G.D. Altman D.G. Systematic review in health care: meta-analysis in context. 2nd edition. BMJ Publishing Group, London, England2001: 23-42
        • Moher D.
        • Soeken K.
        • Sampson M.
        • Ben-Porat L.
        • Berman B.
        Assessing the quality of reports of systematic reviews in pediatric complementary and alternative medicine.
        BMC Pediatr. 2002; 2: 3
        • Santaguida P.L.
        • Helfand M.
        • Raina P.
        Challenges in systematic reviews that evaluate drug efficacy or effectiveness.
        Ann Intern Med. 2005; 142 ([review]): 1066-1072
        • Chou R.
        • Helfand M.
        Challenges in systematic reviews that assess treatment harms.
        Ann Intern Med. 2005; 142 ([review]): 1090-1099
        • Norris S.L.
        • Atkins D.
        Challenges in using nonrandomized studies in systematic reviews of treatment interventions.
        Ann Intern Med. 2005; 142 ([review]): 1112-1119
        • Tatsioni A.
        • Zarin D.A.
        • Aronson N.
        • Samson D.J.
        • Flamm C.R.
        • Schmid C.
        • et al.
        Challenges in systematic reviews of diagnostic technologies.
        Ann Intern Med. 2005; 142 ([review]): 1048-1055
        • Hartling L.
        • McAlister F.A.
        • Rowe B.H.
        • Ezekowitz J.
        • Friesen C.
        • Klassen T.P.
        Challenges in systematic reviews of therapeutic devices and procedures.
        Ann Intern Med. 2005; 142: 1100-1111
        • Helfand M.
        • Morton S.
        • Guallar E.
        • Mulrow C.
        A guide to this supplement.
        Ann Intern Med. 2005; 142: 1033-1034
      1. Higgins J.P.T. Green S. Cochrane Handbook for Systematic Reviews of Interventions 4.2.6 [updated September 2006]. In: The Cochrane Library, Issue 4. John Wiley & Sons, Ltd., Chichester, UK2006
        • National Health Service Centre for Reviews and Dissemination
        Undertaking systematic reviews of research on effectiveness.
        (CRD Report 4)2nd edition. NHS Centre for Reviews and Dissemination, The University of York, York, UK2001 (Report No.: 4)
        • National Health Service Centre for Reviews and Dissemination
        Review methods and resources.
        NHS Centre for Reviews and Dissemination, The University of York, York, UK2007
        • Egger M.
        • Zellweger-Zahner T.
        • Schneider M.
        • Junker C.
        • Lengeler C.
        • Antes G.
        Language bias in randomised controlled trials published in English and German.
        Lancet. 1997; 350: 326-329
        • Moher D.
        • Fortin P.
        • Jadad A.R.
        • Juni P.
        • Klassen T.
        • Le Lorier J.
        • et al.
        Completeness of reporting of trials published in languages other than English: implications for conduct and reporting of systematic reviews.
        Lancet. 1996; 347: 363-366
        • Scherer R.W.
        • Dickersin K.
        • Langenberg P.
        Full publication of results initially presented in abstracts. A meta-analysis.
        JAMA. 1994; 272: 158-162
        • Slutsky J.
        • Atkins D.
        • Chang S.
        • Collins S.B.
        AHRQ Series Paper 1: Comparing medical interventions: AHRQ and the effective health-care program.
        J Clin Epidemiol. 2010; 63 (In this issue): 481-483
        • Whitlock E.P.
        • Lopez S.A.
        • Chang S.
        • Helfand M.
        • Eder M.
        • Floyd N
        AHRQ Series Paper 3: Identifying, selecting, and refining topics for comparative effectiveness systematic reviews: AHRQ and the Effective Health-Care Program.
        J Clin Epidemiol. 2010; 63 (In this issue): 491-501
        • Helfand M.
        Using evidence reports: progress and challenges in evidence-based decision making.
        Health Aff (Millwood). 2005; 24: 123-127
        • Drummond M.F.
        • Schwartz J.S.
        • Jönsson B.
        • Luce B.R.
        • Neumann P.J.
        • Siebert U.
        • et al.
        Key principles for the improved conduct of health technology assessments for resource allocation decisions.
        Int J Technol Assess Health Care. 2008; 24: 244-258
        • Black D.
        J Med Ethics. 2000; 26 ([editorial]): 229-230
        • Guyatt G.H.
        • Montori V.M.
        • Devereaux P.J.
        • Schunemann H.
        • Bhandari M.
        Patients at the centre: in our practice, and in our use of language.
        Evid Based Med. 2004; 9 ([editorial]): 6-7
        • Guyatt G.H.
        • Cook D.J.
        • Haynes B.
        Evidence based medicine has come a long way.
        BMJ. 2004; 329 ([editorial]): 990-991
        • Bravata D.M.
        • McDonald K.M.
        • Shojania K.G.
        • Sundaram V.
        • Owens D.K.
        Challenges in systematic reviews: synthesis of topics related to the delivery, organization, and financing of health care.
        Ann Intern Med. 2005; 142: 1056-1065
        • Harris R.P.
        • Helfand M.
        • Woolf S.H.
        • Lohr K.N.
        • Mulrow C.D.
        • Teutsch S.M.
        • et al.
        Current methods of the US Preventive Services Task Force: a review of the process.
        Am J Prev Med. 2001; 20: 21-35
        • Whitlock E.P.
        • Orleans C.T.
        • Pender N.
        • Allan J.
        Evaluating primary care behavioral counseling interventions: an evidence-based approach.
        Am J Prev Med. 2002; 22 ([review]): 267-284
        • Woolf S.H.
        • DiGuiseppi C.G.
        • Atkins D.
        • Kamerow D.B.
        Developing evidence-based clinical practice guidelines: lessons learned by the US Preventive Services Task Force.
        Annu Rev Public Health. 1996; 17 ([review]): 511-538
        • Mulrow C.
        • Langhorne P.
        • Grimshaw J.
        Integrating heterogeneous pieces of evidence in systematic reviews.
        Ann Intern Med. 1997; 127: 989-995
        • Bigby M.
        Challenges to the hierarchy of evidence: does the emperor have no clothes?.
        Arch Dermatol. 2001; 137 ([article criticism]): 345-346
        • Devereaux P.J.
        • Yusuf S.
        The evolution of the randomized controlled trial and its role in evidence-based decision making.
        J Intern Med. 2003; 254: 105-113
        • Shrier I.
        • Boivin J.-F.
        • Steele R.J.
        • Platt R.W.
        • Furlan A.
        • Kakuma R.
        • et al.
        Should meta-analyses of interventions include observational studies in addition to randomized controlled trials? A critical examination of underlying principles.
        Am J Epidemiol. 2007; 166: 1203-1209
        • Walach H.
        • Falkenberg T.
        • Fonnebo V.
        • Lewith G.
        • Jonas W.B.
        Circular instead of hierarchical: methodological principles for the evaluation of complex interventions.
        BMC Med Res Methodol. 2006; 6: 29
        • Tucker J.A.
        • Roth D.L.
        Extending the evidence hierarchy to enhance evidence-based practice for substance use disorders.
        Addiction. 2006; 101: 918-932
        • Atkins D.
        • Fink K.
        • Slutsky J.
        Better information for better health care: the evidence-based practice center program and the Agency for Healthcare Research and Quality.
        Ann Intern Med. 2005; 142: 1035-1041
        • Shekelle P.G.
        • Morton S.C.
        • Suttorp M.J.
        • Buscemi N.
        • Friesen C.
        Challenges in systematic reviews of complementary and alternative medicine topics.
        Ann Intern Med. 2005; 142: 1042-1047
        • Pignone M.
        • Saha S.
        • Hoerger T.
        • Lohr K.N.
        • Teutsch S.
        • Mandelblatt J.
        Challenges in systematic reviews of economic analyses.
        Ann Intern Med. 2005; 142: 1073-1079
        • Godwin M.
        • Ruhland L.
        • Casson I.
        • MacDonald S.
        • Delva D.
        • Birtwhistle R.
        • et al.
        Pragmatic controlled clinical trials in primary care: the struggle between external and internal validity.
        BMC Med Res Methodol. 2003; 3: 28
        • Fullerton D.S.P.
        • Atherly D.S.
        Formularies, therapeutics, and outcomes: new opportunities.
        Med Care. 2004; 42: III39-III44
        • Glasgow R.E.
        • Magid D.J.
        • Beck A.
        • Ritzwoller D.
        • Estabrooks P.A.
        Practical clinical trials for translating research to practice: design and measurement recommendations.
        Med Care. 2005; 43: 551-557
        • Kotaska A.
        Inappropriate use of randomised trials to evaluate complex phenomena: case study of vaginal breech delivery.
        BMJ. 2004; 329 ([review]): 1039-1042
        • Tunis S.R.
        • Stryer D.B.
        • Clancy C.M.
        Practical clinical trials: increasing the value of clinical research for decision making in clinical and health policy.
        JAMA. 2003; 290: 1624-1632
        • Medical Research Council
        A framework for development and evaluation of RCTs for complex interventions to improve health.
        Medical Research Council, London, England2000 (Contract No.: Document Number)
        • McAlister F.A.
        • Straus S.E.
        • Sackett D.L.
        Why we need large, simple studies of the clinical examination: the problem and a proposed solution. CARE-COAD1 group. Clinical Assessment of the Reliability of the Examination-Chronic Obstructive Airways Disease Group.
        Lancet. 1999; 354: 1721-1724
        • Mosteller F.
        The promise of risk-based allocation trials in assessing new treatments.
        Am J Public Health. 1996; 86 ([editorial]): 622-623