Advertisement

AHRQ Series Paper 4: Assessing harms when comparing medical interventions: AHRQ and the Effective Health-Care Program

Published:September 29, 2008DOI:https://doi.org/10.1016/j.jclinepi.2008.06.007

      Abstract

      Comparative effectiveness reviews (CERs) are systematic reviews that evaluate evidence on alternative interventions to help clinicians, policy makers, and patients make informed treatment choices. Reviews should assess harms and benefits to provide balanced assessments of alternative interventions. Identifying important harms of treatment and quantifying the magnitude of any risks require CER authors to consider a broad range of data sources, including randomized controlled trials (RCTs) and observational studies. This may require evaluation of unpublished data in addition to published reports. Appropriate synthesis of harms data must also consider issues related to evaluation of rare or uncommon events, assessments of equivalence or noninferiority, and use of indirect comparisons. This article presents guidance for evaluating harms when conducting and reporting CERs. We include suggestions for prioritizing harms to be evaluated, use of terminology related to reporting of harms, selection of sources of evidence on harms, assessment of risk of bias (quality) of harms reporting, synthesis of evidence on harms, and reporting of evidence on harms.

      Keywords

      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'

      Subscribe:

      Subscribe to Journal of Clinical Epidemiology
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect

      References

        • Lohr K.N.
        Emerging methods in comparative effectiveness and safety: symposium overview and summary.
        Med Care. 2007; 45: S5-S8
        • GRADE Working Group
        Grading quality of evidence and strength of recommendations.
        Br Med J. 2004; 328: 1490
        • McIntosh H.M.
        • Woolacott N.F.
        • Bagnall A.-M.
        Assessing harmful effects in systematic reviews.
        BMC Med Res Methodol. 2004; 4: 19
        • Loke Y.K.
        • Price D.
        • Herxheimer A.
        Systematic reviews of adverse effects: framework for a structured approach.
        BMC Med Res Methodol. 2007; 7: 32
        • Ioannidis J.P.A.
        • Evans S.J.W.
        • Gotzsche P.C.
        • O'Neill R.T.
        • Altman D.G.
        • Schulz K.
        • et al.
        Better reporting of harms in randomized trials: an extension of the CONSORT statement.
        Ann Intern Med. 2004; 141: 781-788
        • Edwards I.R.
        • Aronson J.K.
        Adverse drug reactions: definitions, diagnosis, and management.
        Lancet. 2000; 356: 1255-1259
        • Ioannidis J.P.A.
        • Lau J.
        Completeness of safety reporting in randomized trials: an evaluation of 7 medical areas.
        J Am Med Assoc. 2001; 285: 437-443
        • Loke Y.
        • Derry S.
        Reporting of adverse drug reactions in randomised controlled trials—a systematic survey.
        BMC Clin Pharmacol. 2001; 1: 3
        • Vandenbroucke J.P.
        Benefits and harms of drug treatments.
        Br Med J. 2004; 329: 2-3
        • Rothwell P.M.
        External validity of randomised controlled trials: “to whom do the results of this trial apply?”.
        Lancet. 2005; 365: 82-93
        • Chou R.
        • Fu R.
        • Huffman L.H.
        • Korthuis P.T.
        Initial highly-active antiretroviral therapy with a protease inhibitor versus a non-nucleoside reverse transcriptase inhibitor: discrepancies between direct and indirect meta-analyses.
        Lancet. 2006; 368: 1503-1515
        • Song F.
        • Altman D.G.
        • Glenny A.M.
        • Deeks J.J.
        Validity of indirect comparison for estimating efficacy of competing interventions: empirical evidence from published meta-analyses.
        Br Med J. 2003; 326: 472
        • Chan A.
        • Hrobjartsson A.
        • Haahr M.
        • Gotzsche P.C.
        • Altman D.G.
        Empirical evidence for selective reporting of outcomes in randomized trials.
        J Am Med Assoc. 2004; 291: 2457-2465
        • Easterbrook P.J.
        • Berlin J.A.
        • Gopalan R.
        • Matthews D.R.
        Publication bias in clinical research.
        Lancet. 1991; 337: 867-872
        • McDonagh M.
        • Helfand M.
        • Carson S.
        • Russman B.S.
        Hyperbaric oxygen therapy for traumatic brain injury: a systematic review of the evidence.
        Arch Phys Med Rehabil. 2004; 85: 1198-1204
        • Kearney P.M.
        • Baigent C.
        • Godwin J.
        • Halls H.
        • Emberson J.R.
        • Patrono C.
        Do selective cyclo-oxygenase-2 inhibitors and traditional non-steroidal anti-inflammatory drugs increase the risk of atherothrombosis? Meta-analysis of randomized trials.
        Br Med J. 2006; 332: 1302-1308
        • Egger M.
        • Juni P.
        • Bartlett C.
        • Holenstein F.
        • Sterne J.
        How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study.
        Health Technol Assess. 2003; 7: 1-76
        • Turner E.H.
        • Matthews A.M.
        • Linardatos E.
        • Tell R.A.
        • Rosenthal R.
        Selective publication of antidepressant trials and its influence on apparent efficacy.
        N Engl J Med. 2008; 358: 252-260
        • Whittington C.J.
        • Kendall T.
        • Fonagy P.
        • Cottrell D.
        • Cotgrove A.
        • Boddington E.
        Selective serotonin reuptake inhibitors in childhood depression: systematic review of published versus unpublished data.
        Lancet. 2004; 363: 1341-1345
        • Laine C.
        • Goodman S.N.
        • Griswold M.E.
        • Sox H.C.
        Reproducible research: moving toward research the public can really trust.
        Ann Intern Med. 2007; 146: 450-453
        • Toma M.
        • McAlister F.A.
        • Bialy L.
        • Adams D.
        • Vandermeer B.
        • Armstrong P.W.
        Transition from meeting abstract to full-length journal article for randomized controlled trials.
        J Am Med Assoc. 2006; 295: 1281-1287
        • Ridker P.M.
        • Torres J.
        Reported outcomes in major cardiovascular clinical trials funded by For-profit and Not-For-Profit organizations: 2000–2005.
        J Am Med Assoc. 2006; 295: 2270-2274
        • Sterne J.A.
        • Egger M.
        • Smith G.D.
        Investigating and dealing with publication and other biases in meta-analysis.
        Br Med J. 2001; 323: 101-105
        • Bombardier C.
        • Laine L.
        • Reicin A.
        • Shapiro D.
        • Burgos-Vargas R.
        • Davis B.
        • et al.
        Comparison of upper gastrointestinal toxicity of rofecoxib and naproxen in patients with rheumatoid arthritis.
        N Engl J Med. 2000; 343 ([see comment]): 1520-1528
        • Silverstein F.E.
        • Faich G.
        • Goldstein J.L.
        • Simon L.S.
        • Pincus T.
        • Whelton A.
        • et al.
        Gastrointestinal toxicity with celecoxib vs nonsteroidal anti-inflammatory drugs for osteoarthritis and rheumatoid arthritis: the CLASS study: a randomized controlled trial. Celecoxib Long-term Arthritis Safety Study.
        J Am Med Assoc. 2000; 284 ([see comment]): 1247-1255
        • Hrachovec J.B.
        • Mora M.
        Reporting of 6-month vs 12-month data in a clinical trial of celecoxib.
        J Am Med Assoc. 2001; 286: 2398
        • Witter J.
        Medical review part 1.
        (Center for Drug Evaluation and Research. Available at) (Accessed on April 3, 2008)
        • Furukawa T.A.
        • Watanabe N.
        • Montori V.M.
        • Guyatt G.H.
        Association between unreported outcomes and effect size estimates in Cochrane meta-analyses.
        J Am Med Assoc. 2007; 297 ([letter]): 468-470
        • Vandenbroucke J.P.
        When are observational studies as credible as randomised trials?.
        Lancet. 2004; 363: 1728-1731
        • von Elm E.
        • Altman D.G.
        • Egger M.
        • Pocock S.J.
        • Gøtzsche P.C.
        • Vandenbroucke J.P.
        The strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies.
        Ann Intern Med. 2007; 147: 573-577
        • Kleinbaum D.G.
        • Kupper L.L.
        • Morgenstern H.
        Epidemiologic research. Principles and quantitative methods.
        Wadsworth, Belmont, CA1982
        • Vandenbroucke J.P.
        • von Elm E.
        • Altman D.G.
        • Gotzsche P.C.
        • Mulrow C.D.
        • Pocock S.J.
        • et al.
        Strengthening the reporting of observational studies in epidemiology (STROBE): explanation and elaboration.
        Ann Intern Med. 2007; 147: W163-W194
        • Psaty B.M.
        • Koepsell T.
        • Lin D.
        • Weiss N.S.
        • Siscovick D.S.
        • Rosendaal F.R.
        • et al.
        Assessment and control for confounding by indication in observational studies.
        J Am Geriatr Soc. 1999; 47: 749-754
        • Rothman K.J.
        • Greenland S.
        Modern epidemiology.
        2nd edition. Lippincott-Raven, Philadelphia, PA1998
        • Papanikolaou P.N.
        • Christidi G.D.
        • Ioannidis J.
        Comparison of evidence on harms of medical interventions in randomized and nonrandomized studies.
        Can Med Assoc J. 2006; 174: 635-641
      1. (AHRQ Publication NO. 07-EHC001-1)Gliklich R. Dreyer N.A. Registries for evaluating patient outcomes: a user's guide. Agency for Healthcare Research and Quality, Rockville, MD2007
        • Schneeweiss S.
        • Avorn J.
        A review of uses of health care utilization databases for epidemiologic research on therapeutics.
        J Clin Epidemiol. 2005; 58: 323-337
        • Sturmer T.
        • Schneeweiss S.
        • Rothman K.J.
        • Avorn J.
        • Glynn R.J.
        Performance of propensity score calibration—a simulation study.
        Am J Epidemiol. 2007; 165: 1110-1118
        • Aronson J.K.
        • Derry S.
        • Loke Y.K.
        Adverse drug reactions: keeping up to date.
        Fund Clin Pharmacol. 2002; 16: 49-56
        • Stricker B.H.
        • Psaty B.M.
        Detection, verification, and quantification of adverse drug reactions.
        Br Med J. 2004; 329: 44-47
        • Loke Y.K.
        • Derry S.
        • Aronson J.K.
        A comparison of three different sources of data in assessing the frequencies if adverse reactions to amiodarone.
        Br J Clin Pharmacol. 2004; 57: 616-621
        • Strom B.L.
        Potential for conflict of interest in the evaluation of suspected adverse drug reactions: a counterpoint.
        J Am Med Assoc. 2004; 292: 2643-2646
        • Psaty B.M.
        • Furberg C.D.
        • Ray W.A.
        • Weiss N.S.
        Potential for conflict of interest in the evaluation of suspected adverse drug reactions: use of cerivastatin and risk of rhabdomyolysis.
        J Am Med Assoc. 2004; 292: 2622-2631
        • Bennett C.L.
        • Nebeker J.R.
        • Lyons E.A.
        • Samore M.H.
        • Feldman M.D.
        • McKoy J.M.
        • et al.
        The research on adverse drug events and reports (RADAR) project.
        J Am Med Assoc. 2005; 293: 2131-2140
        • Oleson O.
        • The Cochrane Non-Randomised Studies Methods Group (NRSMG)
        Types of study design.
        (Available at) (Accessed on April 3, 2008)
        • Juni P.
        • Altman D.G.
        • Egger M.
        Systematic reviews in health care: assessing the quality of controlled clinical trials.
        Br Med J. 2001; 323: 42-46
        • Bent S.
        • Padula A.
        • Avins A.L.
        Brief communication: better ways to question patients about adverse medical events: a randomized, controlled trial.
        Ann Inter Med. 2006; 144: 257-261
        • NCI
        Common terminology criteria for adverse events v3.0 (CTCAE).
        (Available at Accessed on April 3, 2008)
        • NIAID
        Division of AIDS table for grading the severity of adult and pediatric adverse events.
        (Available at) (Accessed on April 3, 2008)
        • Sydes M.R.
        • Spiegelhalter D.J.
        • Altman D.G.
        • Babiker A.B.
        • Parmar M.K.B.
        Systematic qualitative review of the literature on data monitoring committees for randomized controlled trials.
        Clin Trials. 2004; 1: 60-79
        • Rubin D.B.
        Estimating causal effects of treatments in randomized and nonrandomized studies.
        J Educ Psychol. 1974; 66: 688-701
        • Rochon P.A.
        • Gurwitz J.H.
        • Sykora K.
        • Mamdani M.
        • Streiner D.L.
        • Garfinkel S.
        • et al.
        Reader's guide to critical appraisal of cohort studies: 1. Role and design.
        Br Med J. 2005; 330: 895-897
        • Deeks J.J.
        • Dinnes J.
        • D'Amico R.
        • Sowden A.J.
        • Sakarovitch C.
        • Song F.
        • et al.
        Evaluating non-randomised intervention studies.
        Health Technol Assess. 2003; 7: 1-173
        • West S.
        • King V.
        • Carey T.S.
        Systems to rate the strength of scientific evidence.
        Agency for Healthcare Research and Quality, Rockville, MD2002
        • Dalziel K.
        • Round A.
        • Stein K.
        • Garside R.
        • Castelnuovo E.
        • Payne L.
        Do the findings of case series studies vary significantly according to methodological characteristics?.
        Health Technol Assess. 2005; 9: 1-146
        • Rothwell P.M.
        • Slattery J.
        • Warlow C.P.
        A systematic review of the risks of stroke and death due to endarterectomy for symptomatic carotid stenosis.
        Stroke. 1996; 27: 260-265
        • Juni P.
        • Nartey L.
        • Reichenbach S.
        • Sterchi R.
        • Dieppe P.A.
        • Egger M.
        Risk of cardiovascular events and rofecoxib: cumulative meta-analysis.
        Lancet. 2004; 364: 2021-2029
        • Ofman J.J.
        • MacLean C.H.
        • Straus W.L.
        • Morton S.C.
        • Berger M.L.
        • Roth E.A.
        • et al.
        A metaanalysis of severe upper gastrointestinal complications of nonsteroidal antiinflammatory drugs.
        J Rheumatol. 2002; 29 ([see comment]): 804-812
        • Shah R.V.
        • Albert T.J.
        • Buegel-Sanchez V.
        • Vaccaro A.R.
        • Hilibrand A.S.
        • Gauer J.N.
        Industry support and correlation to study outcome for papers published in Spine.
        Spine. 2005; 30: 1099-1104
        • Laporte J.R.
        • Ibanez L.
        • vidal X.
        • Vendrell L.
        • Leone R.
        Upper gastrointestinal bleeding associated with the use of NSAIDs: new versus older agents.
        Drug Saf. 2004; 27: 411-420
        • Martin R.C.G.
        • Brennan M.F.
        • Jacques D.P.
        Quality of complication reporting in the surgical literature.
        Ann Surg. 2002; 235: 803-813
        • Carey T.S.
        • Boden S.D.
        A critical guide to case series reports.
        Spine. 2003; 28: 1631-1634
        • Chou R.
        • Fu R.
        • Carson S.
        • Saha S.
        • Helfand M.
        Methodological shortcomings predicted lower harm estimates in one of two sets of studies of clinical interventions.
        J Clin Epidemiol. 2006; 60: 18-28
        • Juni P.
        • Witschi A.
        • Bloch R.
        • Egger M.
        The hazards of scoring the quality of clinical trials for meta-analysis.
        J Am Med Assoc. 1999; 282: 1054-1060
        • Santaguida P.L.
        • Raina P.
        The development of the McHarm quality assessment scale for adverse events: Delphi consensus on important criteria for evaluating harms.
        (Available at) (Accessed on May 14, 2008)
        • Venning G.R.
        Validity of anecdotal reports of suspected adverse drug reactions: the problem of false alarms.
        Br Med J. 1982; 284: 249-252
        • Loke Y.K.
        • Price D.
        • Derry S.
        • Aronson J.K.
        Case reports of suspected adverse drug reactions—systematic literature survey of follow-up.
        Br Med J. 2006; 332: 335-359
        • Begaud B.
        • Moride Y.
        • Tubert-Bitter P.
        • Chaslerie A.
        • Haramburu F.
        False-positives in spontaneous reporting: should we worry about them?.
        Br J Clin Pharmacol. 1994; 38: 401-404
        • Danan G.
        • Benichou C.
        Causality assessment of adverse reactions to drugs—I. A novel method based on the conclusions of international consensus meetings: application to drug-induced liver injuries.
        J Clin Epidemiol. 1993; 46: 1323-1330
        • Michel D.J.
        • Knodel L.C.
        Comparison of three algorithms used to evaluate adverse drug reactions.
        Am J Hosp Pharm. 1986; 43: 1709-1744
        • Aronson J.K.
        Anecdotes as evidence.
        Br Med J. 2003; 326: 1346
        • Lau J.
        • Ioannidis J.P.
        • Schmid C.H.
        Quantitative synthesis in systematic reviews.
        Ann Intern Med. 1997; 127: 820-826
        • Moher D.
        • Pham B.
        • Jones A.
        • Cook D.J.
        • Jadad A.R.
        • Moher M.
        • et al.
        Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses?.
        Lancet. 1998; 352: 609-613
        • Goodman S.N.
        • Berlin J.A.
        The use of predicted confidence intervals when planning experiments and the misuses of power when interpreting results.
        Ann Intern Med. 1994; 121: 200-206
        • Jonville-Bera A.P.
        • Giraudeau B.
        • Autret-Leca E.
        Reporting of drug tolerance in randomized clinical trials: when data conflict with authors' conclusions.
        Ann Intern Med. 2006; 144: 306-307
        • de Gans J.
        • van de Beek D.
        Dexamethasone in adults with bacterial meningitis.
        N Engl J Med. 2002; 347: 1549-1556
        • Piaggio G.
        • Elbourne D.R.
        • Altman D.G.
        • Pocock S.J.
        • Evans S.J.
        • CONSORT Group
        Reporting of noninferiority and equivalence randomized trials: an extension of the CONSORT statement.
        J Am Med Assoc. 2006; 295: 1152-1160
        • Ware J.H.
        • Antman E.M.
        Equivalence trials.
        N Engl J Med. 1997; 337: 1159-1161
        • Bucher H.C.
        • Guyatt G.H.
        • Griffith L.E.
        • Walter S.D.
        The results of direct and indirect treatment comparisons in meta-analysis of randomized controlled trials.
        J Clin Epidemiol. 1997; 50: 683-691
        • Zhang J.
        • Ding E.L.
        • Song Y.
        Adverse effects of cyclooxygenase 2 inhibitors on renal and arrhythmia events: meta-analysis of randomized trials.
        J Am Med Assoc. 2006; 296: 1619-1632
        • Egger M.
        • Schneider M.
        • Davey Smith G.
        Spurious precision? Meta-analysis of observational studies.
        Br Med J. 1998; 316: 140-144