Abstract
Objective
Systematic reviewers disagree about the ability of observational studies to answer
questions about the benefits or intended effects of pharmacotherapeutic, device, or
procedural interventions. This study provides a framework for decision making on the
inclusion of observational studies to assess benefits and intended effects in comparative
effectiveness reviews (CERs).
Study Design and Setting
The conceptual model and recommendations were developed using a consensus process
by members of the methods workgroup of the Effective Health Care Program of the Agency
for Healthcare Research and Quality.
Results
In considering whether to use observational studies in CERs for addressing beneficial
effects, reviewers should answer two questions: (1) Are there gaps in the evidence
from randomized controlled trials (RCTs)? (2) Will observational studies provide valid
and useful information? The latter question involves the following: (a) refocusing
the study questions on gaps in the evidence from RCTs, (b) assessing the risk of bias
of the body of evidence of observational studies, and (c) assessing whether available
observational studies address the gap review questions.
Conclusions
Because it is unusual to find sufficient evidence from RCTs to answer all key questions
concerning benefit or the balance of benefits and harms, comparative effectiveness
reviewers should routinely assess the appropriateness of inclusion of observational
studies for questions of benefit. Furthermore, reviewers should explicitly state the
rationale for inclusion or exclusion of observational studies when conducting CERs.
Keywords
To read this article in full you will need to make a payment
Purchase one-time access:
Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online accessOne-time access price info
- For academic or personal research use, select 'Academic and Personal'
- For corporate R&D use, select 'Corporate R&D Professionals'
Subscribe:
Subscribe to Journal of Clinical EpidemiologyAlready a print subscriber? Claim online access
Already an online subscriber? Sign in
Register: Create an account
Institutional Access: Sign in to ScienceDirect
References
- Gaps in the evaluation and monitoring of new pharmaceuticals: proposal for a different approach.Can Med Assoc J. 2003; 169: 1167-1170
- Challenges and opportunities for pharmacoepidemiology in drug-therapy decision making.J Clin Pharmacol. 2006; 46: 6-9
- Assessing harms when comparing medical interventions: AHRQ and the Effective Health-Care Program.J Clin Epidemiol. 2010; 63: 502-512
- Assessment of methodological quality of primary studies by systematic reviews: results of the metaquality cross sectional study.BMJ. 2005; 330: 1053-1057
- Challenges in using nonrandomized studies in systematic reviews of treatment interventions.Ann Intern Med. 2005; 142: 1112-1119
- A review of uses of health care utilization databases for epidemiologic research on therapeutics.J Clin Epidemiol. 2005; 58: 323-337
- When are observational studies as credible as randomised trials?.Lancet. 2004; 363: 1728-1731
- Higgins J.P.T. Green S. Cochrane handbook for systematic reviews of interventions. John Wiley & Sons, Ltd., Chichester, UK2008
- Evaluating non-randomised intervention studies.Health Technol Assess. 2003; 7 (1–173): iii-x
- Systems to rate the strength of scientific evidence. Evidence report/technology assessment no. 47.Agency for Healthcare Research & Quality, Rockville, MD2002
- The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies.Lancet. 2007; 370: 1453-1457
- AHRQ series, paper 2: principles for developing guidance: AHRQ and the Effective Health-care Program.J Clin Epidemiol. 2010; 63: 484-490
- Why we need observational studies to evaluate the effectiveness of health care.BMJ. 1996; 312: 1215-1218
- Grading quality of evidence and strength of recommendations.BMJ. 2004; 328: 1490-1498
- Grading the strength of a body of evidence when comparing medical interventions: Agency for Healthcare Research and Quality and the Effective Health Care Program.J Clin Epidemiol. 2010; 63: 513-523
McDonagh M, Peterson K, Carson S, Chan B, Thakurta, S. 2008. Drug Effectiveness Review Project, Drug class review: atypical antipsychotic drugs, final report update 2. Portland, OR: Oregon Evidence-based Practice Center.
- A simple and valid tool distinguished efficacy from effectiveness studies.J Clin Epidemiol. 2006; 59: 1040-1048
- Comparative effectiveness of percutaneous coronary interventions and coronary artery bypass grafting for coronary artery disease.Agency for Healthcare Research and Quality, Rockville, MD2007
- External validity of randomised controlled trials: “to whom do the results of this trial apply?”.Lancet. 2005; 365: 82-93
Atkins D, Chang S, Gartlehner G, Buckley DI, Whitlock E, Berliner E, Matchar D. Assessing the applicability of studies when comparing medical interventions. Agency for Healthcare Research and Quality; December 2010. Methods Guide for Comparative Effectiveness Reviews. AHRQ Publication No. 11-EHC019-EF. Available at http://effectivehealthcare.ahrq.gov/.
- MRC/BHF Heart Protection Study of cholesterol lowering with simvastatin in 20,536 high-risk individuals: a randomised placebo-controlled trial.Lancet. 2002; 360: 7-22
- The uses of heparin to treat burn injury. Evidence report/technology assessment no. 148. (Prepared by the McMaster University Evidence-based Practice Center, under Contract N. 290-02-0020). AHRQ Publication No. 07-E004.Agency for Healthcare Research and Quality, Rockville, MD2006
- Drug class review on the triptans.Drug Effectiveness Review Project, Oregon Evidence-based Practice Center, Portland, OR2003
- When are randomised trials unnecessary? Picking signal from noise.BMJ. 2007; 334: 349-351
- Chapter 13: Including nonrandomized studies.in: Higgins J.P.T. Green S. Cochrane handbook for systematic reviews. John Wiley & Sons, Ltd., Chichester, UK2008
- Comparative effectiveness of different beta-adrenergic antagonists on mortality among adults with heart failure in clinical practice.Arch Intern Med. 2008; 168: 2415-2421
- Evaluating medication effects outside of clinical trials: new-user designs.Am J Epidemiol. 2003; 158: 915-920
- Bias in meta-analysis due to outcome variable selection within studies.J Royal Stat Soc C. 2000; 49: 359-370
- Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles.JAMA. 2004; 291: 2457-2465
- Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research.Can Med Assoc J. 2004; 171: 735-740
- Association between unreported outcomes and effect size estimates in Cochrane meta-analyses.JAMA. 2007; 297: 468-470
- Selective reporting of adjusted estimates in observational epidemiology studies: reasons and implications for meta-analyses.Eval Health Prof. 2008; 31: 370-389
- GRADE: an emerging consensus on rating quality of evidence and strength of recommendations.BMJ. 2008; 336: 924-926
Article info
Publication history
Published online: June 02, 2011
Accepted:
April 21,
2010
Footnotes
The views expressed in this article are those of the authors and do not represent the official policies of the Agency for Healthcare Research and Quality, the Department of Health and Human Services, the Department of Veterans Affairs, the Veterans Health Administration, or the Health Services Research and Development Service.
Identification
Copyright
© 2011 Elsevier Inc. Published by Elsevier Inc. All rights reserved.
ScienceDirect
Access this article on ScienceDirectLinked Article
- Erratum to “Observational studies in systemic reviews of comparative effectiveness: AHRQ and the Effective Health Care Program” [J Clin Epidemiol 2011;64:1178–1186]Journal of Clinical EpidemiologyVol. 66Issue 10
