AHRQ Series Part II: Methods Guide for Comparative Effectiveness - Guest Editor, Mark Helfand| Volume 64, ISSUE 11, P1178-1186, November 2011

Observational studies in systemic reviews of comparative effectiveness: AHRQ and the Effective Health Care Program



      Systematic reviewers disagree about the ability of observational studies to answer questions about the benefits or intended effects of pharmacotherapeutic, device, or procedural interventions. This study provides a framework for decision making on the inclusion of observational studies to assess benefits and intended effects in comparative effectiveness reviews (CERs).

      Study Design and Setting

      The conceptual model and recommendations were developed using a consensus process by members of the methods workgroup of the Effective Health Care Program of the Agency for Healthcare Research and Quality.


      In considering whether to use observational studies in CERs for addressing beneficial effects, reviewers should answer two questions: (1) Are there gaps in the evidence from randomized controlled trials (RCTs)? (2) Will observational studies provide valid and useful information? The latter question involves the following: (a) refocusing the study questions on gaps in the evidence from RCTs, (b) assessing the risk of bias of the body of evidence of observational studies, and (c) assessing whether available observational studies address the gap review questions.


      Because it is unusual to find sufficient evidence from RCTs to answer all key questions concerning benefit or the balance of benefits and harms, comparative effectiveness reviewers should routinely assess the appropriateness of inclusion of observational studies for questions of benefit. Furthermore, reviewers should explicitly state the rationale for inclusion or exclusion of observational studies when conducting CERs.


      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to Journal of Clinical Epidemiology
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Laupacis A.
        • Paterson J.M.
        • Mamdani M.
        • Rostom A.
        • Anderson G.M.
        Gaps in the evaluation and monitoring of new pharmaceuticals: proposal for a different approach.
        Can Med Assoc J. 2003; 169: 1167-1170
        • Etminan M.
        • Gill S.
        • Fitzgerald M.
        • Samii A.
        Challenges and opportunities for pharmacoepidemiology in drug-therapy decision making.
        J Clin Pharmacol. 2006; 46: 6-9
        • Chou R.
        • Aronson N.
        • Atkins D.
        • Ismaila A.S.
        • Santaguida P.
        • Smith D.H.
        • Whitlock E.
        • Wilt T.J.
        • Moher D.
        Assessing harms when comparing medical interventions: AHRQ and the Effective Health-Care Program.
        J Clin Epidemiol. 2010; 63: 502-512
        • Moja L.P.
        • Telaro E.
        • D’Amico R.
        • Moschetti I.
        • Coe L.
        • Liberati A.
        Assessment of methodological quality of primary studies by systematic reviews: results of the metaquality cross sectional study.
        BMJ. 2005; 330: 1053-1057
        • Norris S.L.
        • Atkins D.
        Challenges in using nonrandomized studies in systematic reviews of treatment interventions.
        Ann Intern Med. 2005; 142: 1112-1119
        • Schneeweiss S.
        • Avorn J.
        A review of uses of health care utilization databases for epidemiologic research on therapeutics.
        J Clin Epidemiol. 2005; 58: 323-337
        • Vandenbroucke J.P.
        When are observational studies as credible as randomised trials?.
        Lancet. 2004; 363: 1728-1731
      1. Higgins J.P.T. Green S. Cochrane handbook for systematic reviews of interventions. John Wiley & Sons, Ltd., Chichester, UK2008
        • Deeks J.J.
        • Dinnes J.
        • D’Amico R.
        • Sowden A.J.
        • Sakarovitch C.
        • Song F.
        • Petticrew M.
        • Altman D.G.
        • International Stroke Trial Collaborative Group; European Carotid Surgery Trial Collaborative Group
        Evaluating non-randomised intervention studies.
        Health Technol Assess. 2003; 7 (1–173): iii-x
        • West S.
        • King V.
        • Carey T.S.
        • Lohr K.N.
        • McKoy N.
        • Sutton S.F.
        • Lux L.
        Systems to rate the strength of scientific evidence. Evidence report/technology assessment no. 47.
        Agency for Healthcare Research & Quality, Rockville, MD2002
        • von Elm E.
        • Altman D.G.
        • Egger M.
        • Pocock S.J.
        • Gotzsche P.C.
        • Vandenbroucke J.P.
        The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies.
        Lancet. 2007; 370: 1453-1457
        • Helfand M.
        • Balshem H.
        AHRQ series, paper 2: principles for developing guidance: AHRQ and the Effective Health-care Program.
        J Clin Epidemiol. 2010; 63: 484-490
        • Black N.
        Why we need observational studies to evaluate the effectiveness of health care.
        BMJ. 1996; 312: 1215-1218
        • GRADE Working Group
        Grading quality of evidence and strength of recommendations.
        BMJ. 2004; 328: 1490-1498
        • Owens D.K.
        • Lohr K.N.
        • Atkins D.
        • Treadwell J.R.
        • Reston J.T.
        • Bass E.B.
        • Chang S.
        • Helfand M.
        Grading the strength of a body of evidence when comparing medical interventions: Agency for Healthcare Research and Quality and the Effective Health Care Program.
        J Clin Epidemiol. 2010; 63: 513-523
      2. McDonagh M, Peterson K, Carson S, Chan B, Thakurta, S. 2008. Drug Effectiveness Review Project, Drug class review: atypical antipsychotic drugs, final report update 2. Portland, OR: Oregon Evidence-based Practice Center.

        • Gartlehner G.
        • Hansen R.A.
        • Nissman D.
        • Lohr K.N.
        • Carey T.S.
        A simple and valid tool distinguished efficacy from effectiveness studies.
        J Clin Epidemiol. 2006; 59: 1040-1048
        • Bravata D.M.
        • McDonald K.M.
        • Gienger A.L.
        • Sundaram V.
        • Perez M.V.
        • Varghese R.
        • Kapoor J.R.
        • Ardehali R.
        • McKinnon M.C.
        • Stave C.D.
        • et al.
        Comparative effectiveness of percutaneous coronary interventions and coronary artery bypass grafting for coronary artery disease.
        Agency for Healthcare Research and Quality, Rockville, MD2007
        • Rothwell P.M.
        External validity of randomised controlled trials: “to whom do the results of this trial apply?”.
        Lancet. 2005; 365: 82-93
      3. Atkins D, Chang S, Gartlehner G, Buckley DI, Whitlock E, Berliner E, Matchar D. Assessing the applicability of studies when comparing medical interventions. Agency for Healthcare Research and Quality; December 2010. Methods Guide for Comparative Effectiveness Reviews. AHRQ Publication No. 11-EHC019-EF. Available at

        • Heart Protection Study Collaborative Group
        MRC/BHF Heart Protection Study of cholesterol lowering with simvastatin in 20,536 high-risk individuals: a randomised placebo-controlled trial.
        Lancet. 2002; 360: 7-22
        • Oremus M.
        • Hanson M.
        • Whitlock R.
        • Young E.
        • Gupta A.
        • Dal Cin A.
        • Archer C.
        • Raina P.
        The uses of heparin to treat burn injury. Evidence report/technology assessment no. 148. (Prepared by the McMaster University Evidence-based Practice Center, under Contract N. 290-02-0020). AHRQ Publication No. 07-E004.
        Agency for Healthcare Research and Quality, Rockville, MD2006
        • Helfand M.
        • Peterson K.
        Drug class review on the triptans.
        Drug Effectiveness Review Project, Oregon Evidence-based Practice Center, Portland, OR2003
        • Glasziou P.
        • Chalmers I.
        • Rawlins M.
        • McCulloch P.
        When are randomised trials unnecessary? Picking signal from noise.
        BMJ. 2007; 334: 349-351
        • Reeves B.C.
        • Deeks J.J.
        • Higgins J.P.T.
        • Wells G.A.
        Chapter 13: Including nonrandomized studies.
        in: Higgins J.P.T. Green S. Cochrane handbook for systematic reviews. John Wiley & Sons, Ltd., Chichester, UK2008
        • Go A.S.
        • Yang J.
        • Gurwitz J.H.
        • Hsu J.
        • Lane K.
        • Platt R.
        Comparative effectiveness of different beta-adrenergic antagonists on mortality among adults with heart failure in clinical practice.
        Arch Intern Med. 2008; 168: 2415-2421
        • Ray W.A.
        Evaluating medication effects outside of clinical trials: new-user designs.
        Am J Epidemiol. 2003; 158: 915-920
        • Hutten J.L.
        • Williamson P.R.
        Bias in meta-analysis due to outcome variable selection within studies.
        J Royal Stat Soc C. 2000; 49: 359-370
        • Chan A.W.
        • Hrobjartsson A.
        • Haahr M.T.
        • Gotzsche P.C.
        • Altman D.G.
        Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles.
        JAMA. 2004; 291: 2457-2465
        • Chan A.W.
        • Krleza-Jeric K.
        • Schmid I.
        • Altman D.G.
        Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research.
        Can Med Assoc J. 2004; 171: 735-740
        • Furukawa T.A.
        • Watanabe N.
        • Omori I.M.
        • Montori V.M.
        • Guyatt G.H.
        Association between unreported outcomes and effect size estimates in Cochrane meta-analyses.
        JAMA. 2007; 297: 468-470
        • Peters J.
        • Mengersen K.
        • Peters J.
        • Mengersen K.
        Selective reporting of adjusted estimates in observational epidemiology studies: reasons and implications for meta-analyses.
        Eval Health Prof. 2008; 31: 370-389
        • Guyatt G.H.
        • Oxman A.D.
        • Vist G.E.
        • Kunz R.
        • Falck-Ytter Y.
        • Alonso-Coello P.
        • Schunemann H.J.
        • GRADE Working Group
        GRADE: an emerging consensus on rating quality of evidence and strength of recommendations.
        BMJ. 2008; 336: 924-926

      Linked Article