Journal of Clinical Epidemiology
Volume 63, Issue 5 , Pages 502-512 , May 2010

AHRQ Series Paper 4: Assessing harms when comparing medical interventions: AHRQ and the Effective Health-Care Program

  • Roger Chou

      Affiliations

    • Oregon Evidence-based Practice Center, Oregon Health & Science University, Department of Medicine, 3181 SW Sam Jackson Park Road, Mail Code: BICC, Portland, OR, USA
    • Corresponding Author InformationCorresponding author. Tel.: +503-494-5367; fax: +503-494-4551.
    • ,
  • Naomi Aronson

      Affiliations

    • Blue Cross Blue Shield Evidence-based Practice Center, Blue Cross Blue Shield Association, Chicago, IL, USA
    • ,
  • David Atkins

      Affiliations

    • Office of Research Development, Veteran's Affairs Administration, Washington, DC, USA
    • Previously at the Agency for Healthcare Research and Quality, Rockville, MD, USA.
    • ,
  • Afisi S. Ismaila

      Affiliations

    • McMaster Evidence-based Practice Center, McMaster University, Hamilton, Ontario, Canada
    • ,
  • Pasqualina Santaguida

      Affiliations

    • McMaster Evidence-based Practice Center, McMaster University, Hamilton, Ontario, Canada
    • ,
  • David H. Smith

      Affiliations

    • Oregon Evidence-based Practice Center, Kaiser Center for Health Research, Portland, OR, USA
    • ,
  • Evelyn Whitlock

      Affiliations

    • Oregon Evidence-based Practice Center, Kaiser Center for Health Research, Portland, OR, USA
    • ,
  • Timothy J. Wilt

      Affiliations

    • Minnesota Evidence-based Practice Center, Minneapolis VA Center for Chronic Disease Outcomes Research, MN, USA
    • ,
  • David Moher

      Affiliations

    • University of Ottawa Evidence-based Practice Center, University of Ottawa, Ottawa, Ontario, Canada

,Accepted 21 June 2008.

References 

  1. Lohr KN. Emerging methods in comparative effectiveness and safety: symposium overview and summary. Med Care. 2007;45:S5–S8
  2. GRADE Working Group. Grading quality of evidence and strength of recommendations. Br Med J. 2004;328:1490
  3. McIntosh HM, Woolacott NF, Bagnall A-M. Assessing harmful effects in systematic reviews. BMC Med Res Methodol. 2004;4:19
  4. Loke YK, Price D, Herxheimer A. Systematic reviews of adverse effects: framework for a structured approach. BMC Med Res Methodol. 2007;7:32
  5. Ioannidis JPA, Evans SJW, Gotzsche PC, O'Neill RT, Altman DG, Schulz K, et al. Better reporting of harms in randomized trials: an extension of the CONSORT statement. Ann Intern Med. 2004;141:781–788
  6. Edwards IR, Aronson JK. Adverse drug reactions: definitions, diagnosis, and management. Lancet. 2000;356:1255–1259
  7. Ioannidis JPA, Lau J. Completeness of safety reporting in randomized trials: an evaluation of 7 medical areas. J Am Med Assoc. 2001;285:437–443
  8. Loke Y, Derry S. Reporting of adverse drug reactions in randomised controlled trials—a systematic survey. BMC Clin Pharmacol. 2001;1:3
  9. Vandenbroucke JP. Benefits and harms of drug treatments. Br Med J. 2004;329:2–3
  10. Rothwell PM. External validity of randomised controlled trials: “to whom do the results of this trial apply?”. Lancet. 2005;365:82–93
  11. Chou R, Fu R, Huffman LH, Korthuis PT. Initial highly-active antiretroviral therapy with a protease inhibitor versus a non-nucleoside reverse transcriptase inhibitor: discrepancies between direct and indirect meta-analyses. Lancet. 2006;368:1503–1515
  12. Song F, Altman DG, Glenny AM, Deeks JJ. Validity of indirect comparison for estimating efficacy of competing interventions: empirical evidence from published meta-analyses. Br Med J. 2003;326:472
  13. Chan A, Hrobjartsson A, Haahr M, Gotzsche PC, Altman DG. Empirical evidence for selective reporting of outcomes in randomized trials. J Am Med Assoc. 2004;291:2457–2465
  14. Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR. Publication bias in clinical research. Lancet. 1991;337:867–872
  15. McDonagh M, Helfand M, Carson S, Russman BS. Hyperbaric oxygen therapy for traumatic brain injury: a systematic review of the evidence. Arch Phys Med Rehabil. 2004;85:1198–1204
  16. Kearney PM, Baigent C, Godwin J, Halls H, Emberson JR, Patrono C. Do selective cyclo-oxygenase-2 inhibitors and traditional non-steroidal anti-inflammatory drugs increase the risk of atherothrombosis? Meta-analysis of randomized trials. Br Med J. 2006;332:1302–1308
  17. Egger M, Juni P, Bartlett C, Holenstein F, Sterne J. How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study. Health Technol Assess. 2003;7:1–76
  18. Turner EH, Matthews AM, Linardatos E, Tell RA, Rosenthal R. Selective publication of antidepressant trials and its influence on apparent efficacy. N Engl J Med. 2008;358:252–260
  19. Whittington CJ, Kendall T, Fonagy P, Cottrell D, Cotgrove A, Boddington E. Selective serotonin reuptake inhibitors in childhood depression: systematic review of published versus unpublished data. Lancet. 2004;363:1341–1345
  20. Laine C, Goodman SN, Griswold ME, Sox HC. Reproducible research: moving toward research the public can really trust. Ann Intern Med. 2007;146:450–453
  21. Toma M, McAlister FA, Bialy L, Adams D, Vandermeer B, Armstrong PW. Transition from meeting abstract to full-length journal article for randomized controlled trials. J Am Med Assoc. 2006;295:1281–1287
  22. Ridker PM, Torres J. Reported outcomes in major cardiovascular clinical trials funded by For-profit and Not-For-Profit organizations: 2000–2005. J Am Med Assoc. 2006;295:2270–2274
  23. Sterne JA, Egger M, Smith GD. Investigating and dealing with publication and other biases in meta-analysis. Br Med J. 2001;323:101–105
  24. Bombardier C, Laine L, Reicin A, Shapiro D, Burgos-Vargas R, Davis B, et al. Comparison of upper gastrointestinal toxicity of rofecoxib and naproxen in patients with rheumatoid arthritis. [see comment] N Engl J Med. 2000;343:1520–1528
  25. Silverstein FE, Faich G, Goldstein JL, Simon LS, Pincus T, Whelton A, et al. Gastrointestinal toxicity with celecoxib vs nonsteroidal anti-inflammatory drugs for osteoarthritis and rheumatoid arthritis: the CLASS study: a randomized controlled trial. Celecoxib Long-term Arthritis Safety Study. [see comment] J Am Med Assoc. 2000;284:1247–1255
  26. Hrachovec JB, Mora M. Reporting of 6-month vs 12-month data in a clinical trial of celecoxib. J Am Med Assoc. 2001;286:2398
  27. Witter J. Medical review part 1. Center for Drug Evaluation and Research. Available at http://www.fda.gov/cder/foi/nda/2002/20-998S009_Celebrex_medr_P1.pdfAccessed on April 3, 2008
  28. Furukawa TA, Watanabe N, Montori VM, Guyatt GH. Association between unreported outcomes and effect size estimates in Cochrane meta-analyses. [letter] J Am Med Assoc. 2007;297:468–470
  29. Vandenbroucke JP. When are observational studies as credible as randomised trials?. Lancet. 2004;363:1728–1731
  30. von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP. The strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies. Ann Intern Med. 2007;147:573–577
  31. Kleinbaum DG, Kupper LL, Morgenstern H. Epidemiologic research. Principles and quantitative methods. Belmont, CA: Wadsworth; 1982;
  32. Vandenbroucke JP, von Elm E, Altman DG, Gotzsche PC, Mulrow CD, Pocock SJ, et al. Strengthening the reporting of observational studies in epidemiology (STROBE): explanation and elaboration. Ann Intern Med. 2007;147:W163–W194
  33. Psaty BM, Koepsell T, Lin D, Weiss NS, Siscovick DS, Rosendaal FR, et al. Assessment and control for confounding by indication in observational studies. J Am Geriatr Soc. 1999;47:749–754
  34. Rothman KJ, Greenland S. Modern epidemiology. 2nd edition. Philadelphia, PA: Lippincott-Raven; 1998;
  35. Papanikolaou PN, Christidi GD, Ioannidis J. Comparison of evidence on harms of medical interventions in randomized and nonrandomized studies. Can Med Assoc J. 2006;174:635–641
  36. AHRQ Publication NO. 07-EHC001-1 In:  Gliklich R,  Dreyer NA editor. Registries for evaluating patient outcomes: a user's guide. Rockville, MD: Agency for Healthcare Research and Quality; 2007;
  37. Schneeweiss S, Avorn J. A review of uses of health care utilization databases for epidemiologic research on therapeutics. J Clin Epidemiol. 2005;58:323–337
  38. Sturmer T, Schneeweiss S, Rothman KJ, Avorn J, Glynn RJ. Performance of propensity score calibration—a simulation study. Am J Epidemiol. 2007;165:1110–1118
  39. Aronson JK, Derry S, Loke YK. Adverse drug reactions: keeping up to date. Fund Clin Pharmacol. 2002;16:49–56
  40. Stricker BH, Psaty BM. Detection, verification, and quantification of adverse drug reactions. Br Med J. 2004;329:44–47
  41. Loke YK, Derry S, Aronson JK. A comparison of three different sources of data in assessing the frequencies if adverse reactions to amiodarone. Br J Clin Pharmacol. 2004;57:616–621
  42. Strom BL. Potential for conflict of interest in the evaluation of suspected adverse drug reactions: a counterpoint. J Am Med Assoc. 2004;292:2643–2646
  43. Psaty BM, Furberg CD, Ray WA, Weiss NS. Potential for conflict of interest in the evaluation of suspected adverse drug reactions: use of cerivastatin and risk of rhabdomyolysis. J Am Med Assoc. 2004;292:2622–2631
  44. Bennett CL, Nebeker JR, Lyons EA, Samore MH, Feldman MD, McKoy JM, et al. The research on adverse drug events and reports (RADAR) project. J Am Med Assoc. 2005;293:2131–2140
  45. Oleson O The Cochrane Non-Randomised Studies Methods Group (NRSMG). Types of study design. Available at http://www.cochrane.dk/nrsmg/docs/chap2.pdf1999;Accessed on April 3, 2008
  46. Juni P, Altman DG, Egger M. Systematic reviews in health care: assessing the quality of controlled clinical trials. Br Med J. 2001;323:42–46
  47. Bent S, Padula A, Avins AL. Brief communication: better ways to question patients about adverse medical events: a randomized, controlled trial. Ann Inter Med. 2006;144:257–261
  48. NCI . Common terminology criteria for adverse events v3.0 (CTCAE). http://ctep.cancer.gov/reporting/ctc_v30.html2006;Available at Accessed on April 3, 2008
  49. NIAID . Division of AIDS table for grading the severity of adult and pediatric adverse events. Available at http://www3.niaid.nih.gov/research/resources/DAIDSClinRsrch/Safety/2004;Accessed on April 3, 2008
  50. Sydes MR, Spiegelhalter DJ, Altman DG, Babiker AB, Parmar MKB. Systematic qualitative review of the literature on data monitoring committees for randomized controlled trials. Clin Trials. 2004;1:60–79
  51. Rubin DB. Estimating causal effects of treatments in randomized and nonrandomized studies. J Educ Psychol. 1974;66:688–701
  52. Rochon PA, Gurwitz JH, Sykora K, Mamdani M, Streiner DL, Garfinkel S, et al. Reader's guide to critical appraisal of cohort studies: 1. Role and design. Br Med J. 2005;330:895–897
  53. Deeks JJ, Dinnes J, D'Amico R, Sowden AJ, Sakarovitch C, Song F, et al. Evaluating non-randomised intervention studies. Health Technol Assess. 2003;7:1–173
  54. West S, King V, Carey TS. Systems to rate the strength of scientific evidence. Rockville, MD: Agency for Healthcare Research and Quality; 2002;
  55. Dalziel K, Round A, Stein K, Garside R, Castelnuovo E, Payne L. Do the findings of case series studies vary significantly according to methodological characteristics?. Health Technol Assess. 2005;9:1–146
  56. Rothwell PM, Slattery J, Warlow CP. A systematic review of the risks of stroke and death due to endarterectomy for symptomatic carotid stenosis. Stroke. 1996;27:260–265
  57. Juni P, Nartey L, Reichenbach S, Sterchi R, Dieppe PA, Egger M. Risk of cardiovascular events and rofecoxib: cumulative meta-analysis. Lancet. 2004;364:2021–2029
  58. Ofman JJ, MacLean CH, Straus WL, Morton SC, Berger ML, Roth EA, et al. A metaanalysis of severe upper gastrointestinal complications of nonsteroidal antiinflammatory drugs. [see comment] J Rheumatol. 2002;29:804–812
  59. Shah RV, Albert TJ, Buegel-Sanchez V, Vaccaro AR, Hilibrand AS, Gauer JN. Industry support and correlation to study outcome for papers published in Spine. Spine. 2005;30:1099–1104
  60. Laporte JR, Ibanez L, vidal X, Vendrell L, Leone R. Upper gastrointestinal bleeding associated with the use of NSAIDs: new versus older agents. Drug Saf. 2004;27:411–420
  61. Martin RCG, Brennan MF, Jacques DP. Quality of complication reporting in the surgical literature. Ann Surg. 2002;235:803–813
  62. Carey TS, Boden SD. A critical guide to case series reports. Spine. 2003;28:1631–1634
  63. Chou R, Fu R, Carson S, Saha S, Helfand M. Methodological shortcomings predicted lower harm estimates in one of two sets of studies of clinical interventions. J Clin Epidemiol. 2006;60:18–28
  64. Juni P, Witschi A, Bloch R, Egger M. The hazards of scoring the quality of clinical trials for meta-analysis. J Am Med Assoc. 1999;282:1054–1060
  65. Santaguida PL, Raina P. The development of the McHarm quality assessment scale for adverse events: Delphi consensus on important criteria for evaluating harms. Available at http://hiru.mcmaster.ca/epc/mcharm.pdf2008;Accessed on May 14, 2008
  66. Venning GR. Validity of anecdotal reports of suspected adverse drug reactions: the problem of false alarms. Br Med J. 1982;284:249–252
  67. Loke YK, Price D, Derry S, Aronson JK. Case reports of suspected adverse drug reactions—systematic literature survey of follow-up. Br Med J. 2006;332:335–359
  68. Begaud B, Moride Y, Tubert-Bitter P, Chaslerie A, Haramburu F. False-positives in spontaneous reporting: should we worry about them?. Br J Clin Pharmacol. 1994;38:401–404
  69. Danan G, Benichou C. Causality assessment of adverse reactions to drugs—I. A novel method based on the conclusions of international consensus meetings: application to drug-induced liver injuries. J Clin Epidemiol. 1993;46:1323–1330
  70. Michel DJ, Knodel LC. Comparison of three algorithms used to evaluate adverse drug reactions. Am J Hosp Pharm. 1986;43:1709–1744
  71. Aronson JK. Anecdotes as evidence. Br Med J. 2003;326:1346
  72. Lau J, Ioannidis JP, Schmid CH. Quantitative synthesis in systematic reviews. Ann Intern Med. 1997;127:820–826
  73. Moher D, Pham B, Jones A, Cook DJ, Jadad AR, Moher M, et al. Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses?. Lancet. 1998;352:609–613
  74. Goodman SN, Berlin JA. The use of predicted confidence intervals when planning experiments and the misuses of power when interpreting results. Ann Intern Med. 1994;121:200–206
  75. Jonville-Bera AP, Giraudeau B, Autret-Leca E. Reporting of drug tolerance in randomized clinical trials: when data conflict with authors' conclusions. Ann Intern Med. 2006;144:306–307
  76. de Gans J, van de Beek D. Dexamethasone in adults with bacterial meningitis. N Engl J Med. 2002;347:1549–1556
  77. Piaggio G, Elbourne DR, Altman DG, Pocock SJ, Evans SJ CONSORT Group. Reporting of noninferiority and equivalence randomized trials: an extension of the CONSORT statement. J Am Med Assoc. 2006;295:1152–1160
  78. Ware JH, Antman EM. Equivalence trials. N Engl J Med. 1997;337:1159–1161
  79. Bucher HC, Guyatt GH, Griffith LE, Walter SD. The results of direct and indirect treatment comparisons in meta-analysis of randomized controlled trials. J Clin Epidemiol. 1997;50:683–691
  80. Zhang J, Ding EL, Song Y. Adverse effects of cyclooxygenase 2 inhibitors on renal and arrhythmia events: meta-analysis of randomized trials. J Am Med Assoc. 2006;296:1619–1632
  81. Egger M, Schneider M, Davey Smith G. Spurious precision? Meta-analysis of observational studies. Br Med J. 1998;316:140–144

 Support: This article was written with support from the Effective Health Care Program at the US Agency for Healthcare Research and Quality. Dr. Moher is supported by a University of Ottawa Research Chair.

PII: S0895-4356(08)00161-3

doi: 10.1016/j.jclinepi.2008.06.007

Journal of Clinical Epidemiology
Volume 63, Issue 5 , Pages 502-512 , May 2010

Article Tools