AHRQ Series Part II: Methods Guide for Comparative Effectiveness - Guest Editor, Mark Helfand| Volume 64, ISSUE 11, P1198-1207, November 2011

Download started.


Assessing applicability when comparing medical interventions: AHRQ and the Effective Health Care Program



      To describe a systematic approach for identifying, reporting, and synthesizing information to allow consistent and transparent consideration of the applicability of the evidence in a systematic review according to the Population, Intervention, Comparator, Outcome, Setting domains.

      Study Design and Setting

      Comparative effectiveness reviews need to consider whether available evidence is applicable to specific clinical or policy questions to be useful to decision makers. Authors reviewed the literature and developed guidance for the Effective Health Care program.


      Because applicability depends on the specific questions and needs of the users, it is difficult to devise a valid uniform scale for rating the overall applicability of individual studies or body of evidence. We recommend consulting stakeholders to identify the factors most relevant to applicability for their decisions. Applicability should be considered separately for benefits and harms. Observational studies can help determine whether trial populations and interventions are representative of “real world” practice. Reviewers should describe differences between available evidence and the ideally applicable evidence for the question being asked and offer a qualitative judgment about the importance and potential effect of those differences.


      Careful consideration of applicability may improve the usefulness of systematic reviews in informing practice and policy.


      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to Journal of Clinical Epidemiology
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Federal Coordinating Council for Comparative Effectiveness Research
        Report to the president and the congress on comparative effectiveness research.
        (Available at) (Accessed June 30, 2009)
      1. Higgins J.P.T. Green S. Cochrane handbook for systematic reviews of interventions version 5.0.2. The Cochrane Collaboration, 2009 (Available at) (Accessed September 2009)
        • Shadish W.
        • Cook T.
        • Campbell D.
        Experimental and quasi-experimental designs for generalized causal inference.
        Houghton Mifflin, Boston, MA2002
        • Guyatt G.H.
        • Oxman A.D.
        • Kunz R.
        • Vist G.E.
        • Falck-Ytter Y.
        • Schünemann H.J.
        • GRADE Working Group
        What is “quality of evidence” and why is it important to clinicians?.
        BMJ. 2008; 336: 995-998
        • Rothwell P.M.
        External validity of randomised controlled trials: “to whom do the results of this trial apply?”.
        Lancet. 2005; 365: 82-93
        • Chou R.
        • Aronson N.
        • Atkins D.
        • Ismaila A.S.
        • Santaguida P.
        • Smith D.H.
        • et al.
        AHRQ series paper 4: assessing harms when comparing medical interventions: AHRQ and the effective health-care program.
        J Clin Epidemiol. 2010; 63: 502-512
        • Bornhöft G.
        • Maxion-Bergemann S.
        • Wolf U.
        • Kienle G.S.
        • Michalsen A.
        • Vollmar H.C.
        • et al.
        Checklist for the qualitative evaluation of clinical studies with particular focus on external validity and model validity.
        BMC Med Res Methodol. 2006; 6: 56
        • Green L.W.
        • Glasgow R.E.
        Evaluating the relevance, generalization, and applicability of research: issues in external validation and translation methodology.
        Eval Health Prof. 2006; 29: 126-153
        • Pibouleau L.
        • Boutron I.
        • Reeves B.C.
        • Nizard R.
        • Ravaud P.
        Applicability and generalisability of published results of randomised controlled trials and non-randomised studies evaluating four orthopaedic procedures: methodological systematic review.
        BMJ. 2009; 339: b4538
        • Owens D.K.
        • Lohr K.N.
        • Atkins D.
        • Treadwell J.R.
        • Reston J.T.
        • Bass E.B.
        • et al.
        AHRQ series paper 5: grading the strength of a body of evidence when comparing medical interventions—agency for healthcare research and quality and the effective health-care program.
        J Clin Epidemiol. 2010; 63: 513-523
        • Falck-Ytter Y.
        • Schünemann H.
        • Guyatt G.
        AHRQ series commentary 1: rating the evidence in comparative effectiveness reviews.
        J Clin Epidemiol. 2010; 63: 474-475
        • Guirguis-Blake J.
        • Calonge N.
        • Miller T.
        • Siu A.
        • Teutsch S.
        • Whitlock E.
        • U.S. Preventive Services Task Force
        Current processes of the U.S. Preventive Services Task Force: refining evidence-based recommendation development.
        Ann Intern Med. 2007; 147: 117-122
        • Cummings S.R.
        • Black D.M.
        • Thompson D.E.
        • Applegate W.B.
        • Barrett-Connor E.
        • Musliner T.A.
        • et al.
        • for the Fracture Intervention Trial Research Group
        Effect of alendronate on risk of fracture in women with low bone density but without vertebral fractures: results from the Fracture Intervention Trial.
        JAMA. 1998; 280: 2077-2082
        • Dhruva S.S.
        • Redberg R.F.
        Variations between clinical trial participants and Medicare beneficiaries in evidence used for Medicare national coverage decisions.
        Arch Intern Med. 2008; 168: 136-140
        • Bravata D.M.
        • McDonald K.M.
        • Gienger A.L.
        • Sundaram V.
        • Perez M.V.
        • Varghese R.
        • et al.
        Comparative effectiveness of percutaneous coronary interventions and coronary artery bypass grafting for coronary artery disease. Comparative Effectiveness Review No. 9. (Prepared by Stanford-UCSF Evidence-based Practice Center under Contract No. 290-02-0017.).
        Agency for Healthcare Research and Quality, Rockville, MD2007 (Available at) (Accessed February 23, 2011)
        • Anderson G.L.
        • Limacher M.
        • Assaf A.R.
        • Bassford T.
        • Beresford S.A.
        • Black H.
        • et al.
        • Women’s Health Initiative Steering Committee
        Effects of conjugated equine estrogen in postmenopausal women with hysterectomy: the Women’s Health Initiative randomized controlled trial.
        JAMA. 2004; 291: 1701-1712
        • Gartlehner G.
        • Hansen R.A.
        • Thieda P.
        • DeVeaugh-Geiss A.M.
        • Gaynes B.N.
        • Krebs E.E.
        • et al.
        Comparative effectiveness of second-generation antidepressants in the pharmacologic treatment of adult depression. Comparative Effectiveness Review No. 7. (Prepared by RTI International-University of North Carolina Evidence-based Practice Center under Contract No. 290-02-0016.).
        Agency for Healthcare Research and Quality, Rockville, MD2007 (Available at) (Accessed February 23, 2011)
        • Whitlock E.P.
        • O’Connor E.A.
        • Williams S.B.
        • Beil T.L.
        • Lutz K.W.
        Effectiveness of weight management programs in children and adolescents. Evidence Report/Technology Assessment No. 170 (Prepared by the Oregon Evidence-based Practice Center under Contract No. 290-02-0024). AHRQ Publication No. 08-E014.
        Agency for Healthcare Research and Quality, Rockville, MD2008
        • Fletcher C.V.
        Translating efficacy into effectiveness in antiretroviral therapy: beyond the pill count.
        Drugs. 2007; 67: 1969-1979
        • Walker C.F.
        • Kordas K.
        • Stoltzfus R.J.
        • Black R.E.
        Interactive effects of iron and zinc on biochemical and functional outcomes in supplementation trials.
        Am J Clin Nutr. 2005; 82: 5-12
        • Wennberg D.
        • Lucas F.
        • Birkmeyer J.
        • Bredenberg C.E.
        • Fisher E.S.
        Variation in carotid endarterectomy mortality in the Medicare population.
        JAMA. 1998; 279: 1278-1281
        • Detke M.J.
        • Wiltse C.G.
        • Mallinckrodt C.H.
        • McNamara R.K.
        • Demitrack M.A.
        • Bitter I.
        Duloxetine in the acute and long-term treatment of major depressive disorder: a placebo- and paroxetine-controlled trial.
        Eur Neuropsychopharmacol. 2004; 14: 457-470
        • Li J.
        • Zhang Q.
        • Zhang M.
        • Egger M.
        Intravenous magnesium for acute myocardial infarction.
        Cochrane Database Syst Rev. 2007; (Art. No.: CD002755)
        • Ferreira-González I.
        • Permanyer-Miralda G.
        • Domingo-Salvany A.
        • Busse J.W.
        • Heels-Ansdell D.
        • Montori V.M.
        • et al.
        Problems with use of composite end points in cardiovascular trials: systematic review of randomised controlled trials.
        BMJ. 2007; 334: 786
        • Ioannidis J.P.
        • Lau J.
        The impact of high-risk patients on the results of clinical trials.
        J Clin Epidemiol. 1997; 50: 1089-1098
        • Hansen R.A.
        • Gartlehner G.
        • Kaufer D.
        • Lohr K.
        • Carey T.
        Drug class review of Alzheimer’s drugs.
        Final Report. 2006; (Available at) (Accessed February 23, 2011)
      2. Humphrey L, Chan BKS, Detlefsen S, Helfand M, Screening for breast cancer. Prepared by Oregon Health Sciences University under Contract No. 290-97-0018. Rockville, MD. Agency for Healthcare Research and Quality. Available at: Accessed August 2002

        • Wilt T.J.
        • Lederle F.A.
        • MacDonald R.
        • Jonk Y.C.
        • Rector T.S.
        • Kane R.L.
        Comparison of endovascular and open surgical repairs for abdominal aortic aneurysm. Evidence Report/Technology Assessment No. 144. (Prepared by the University of Minnesota Evidence-based Practice Center under Contract No. 290-02-0009.) AHRQ Publication No. 06-E017.
        Agency for Healthcare Research and Quality, Rockville, MDAugust 2006
        • Kravitz R.L.
        • Duan N.
        • Braslow J.
        Evidence-based medicine, heterogeneity of treatment effects, and the trouble with averages.
        Milbank Q. 2004; 82: 661-687
        • Rothwell P.M.
        Factors that can affect the external validity of randomised controlled trials.
        PLoS Clin Trials. 2006; 1: e9
        • National Institute for Health and Clinical Excellence
        Lipid modification: cardiovascular risk assessment and the modification of blood lipids for the primary and secondary prevention of cardiovascular disease.
        NICE, London, UK2008 (Available at)
        • Tunis S.R.
        • Stryer D.B.
        • Clancy C.M.
        Practical clinical trials: increasing the value of clinical research for decision making in clinical and health policy.
        JAMA. 2003; 290: 1624-1632
        • Godwin M.
        • Ruhland L.
        • Casson I.
        • MacDonald S.
        • Delva D.
        • Birtwhistle R.
        • et al.
        Pragmatic controlled clinical trials in primary care: the struggle between external and internal validity.
        BMC Med Res Methodol. 2003; 3: 28
        • Atkins D.
        Creating and synthesizing evidence with decision makers in mind: integrating evidence from clinical trials and other study designs.
        Med Care. 2007; 45: S16-S22
        • Gartlehner G.
        • Hansen R.A.
        • Nissman D.
        • Lohr K.N.
        • Carey T.S.
        A simple and valid tool distinguished efficacy from effectiveness studies.
        J Clin Epidemiol. 2006; 59: 1040-1048
        • Thorpe K.E.
        • Zwarenstein M.
        • Oxman A.D.
        • Treweek S.
        • Furberg C.D.
        • Altman D.G.
        • et al.
        A pragmatic-explanatory continuum indicator summary (PRECIS): a tool to help trial designers.
        J Clin Epidemiol. 2009; 62: 464-475
        • Riley R.D.
        • Lambert P.C.
        • Abo-Zaid G.
        Meta-analysis of individual participant data: rationale, conduct, and reporting.
        BMJ. 2010; 340: c221
        • Chambers B.R.
        • Donnan G.
        Carotid endarterectomy for asymptomatic carotid stenosis.
        Cochrane Database Syst Rev. 2005; (Art. No.: CD001923)
        • Sackett D.L.
        • Richardson W.S.
        • Rosenberg W.
        • Haynes R.B.
        Evidence-based medicine—how to practice and teach EBM.
        Churchill Livingstone, New York, NY1997
        • Wilt T.J.
        • Shamliyan T.
        • Taylor B.
        • MacDonald R.
        • Tacklind J.
        • Rutks I.
        • et al.
        Comparative effectiveness of therapies for clinically localized prostate cancer. Comparative Effectiveness Review No. 13. (Prepared by Minnesota Evidence-based Practice Center under Contract No. 290-02-0009.).
        Agency for Healthcare Research and Quality, Rockville, MDFebruary 2008 (Available at) (Accessed February 23, 2011)
        • Bill-Axelson A.
        • Holmberg L.
        • Ruutu M.
        • Häggman M.
        • Andersson S.O.
        • Bratell S.
        • et al.
        • Scandinavian Prostate Cancer Screening Group Study No. 4
        Radical prostatectomy versus watchful waiting in early prostate cancer.
        N Engl J Med. 2005; 12: 1977-1984
        • Iversen P.
        • Madsen P.O.
        • Corle D.K.
        Radical prostatectomy versus expectant treatment for early carcinoma of the prostate. Twenty-three year followup of a prospective randomized study.
        Scan J Urol Nephrol Suppl. 1995; 172: 65-72