Advertisement
Review Article| Volume 62, ISSUE 6, P609-616, June 2009

Download started.

Ok

Inadequate statistical power to detect clinically significant differences in adverse event rates in randomized controlled trials

  • Ruth Tsang
    Affiliations
    Vancouver General Hospital, Vancouver, British Columbia, Canada
    Search for articles by this author
  • Lindsey Colley
    Affiliations
    Centre for Health Evaluation and Outcome Sciences, Providence Health Research Institute, Vancouver, British Columbia, Canada

    Faculty of Pharmaceutical Sciences, University of British Columbia, Vancouver, British Columbia, Canada
    Search for articles by this author
  • Larry D. Lynd
    Correspondence
    Corresponding author: Faculty of Pharmaceutical Sciences, University of British Columbia, 2146, East Mall, Vancouver, BC V6T 1Z3, Canada. Tel.: +604-806-8817; fax: +604-827-4014.
    Affiliations
    Centre for Health Evaluation and Outcome Sciences, Providence Health Research Institute, Vancouver, British Columbia, Canada

    Faculty of Pharmaceutical Sciences, University of British Columbia, Vancouver, British Columbia, Canada
    Search for articles by this author
Published:November 17, 2008DOI:https://doi.org/10.1016/j.jclinepi.2008.08.005

      Abstract

      Objective

      To determine the statistical power to detect potentially clinically significant differences in serious adverse events between drug therapies reported in a sample of randomized controlled trials (RCTs).

      Study Design and Setting

      Systematic review of RCTs with positive efficacy endpoint and at least a twofold difference in the proportion of patients with serious adverse events between treatment groups from six major journals. The power of each study to detect statistically significant differences in serious adverse events was calculated.

      Results

      Of the six included trials, all performed statistical analysis on adverse events without disclosure of the statistical power for detecting the reported differences between groups. The power of each study to detect the reported differences in adverse events was calculated and yielded values ranging from 0.07 to 0.37 among trials with non–statistically significant differences.

      Conclusion

      Statistical testing for differences in the proportion of patients experiencing an adverse event is common in RCTs; non–statistically significant differences are associated with low statistical power. A high probability of type II error may lead to erroneous clinical inference resulting in harm. The statistical power for nonsignificant tests should be considered in the interpretation of results.

      Keywords

      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'

      Subscribe:

      Subscribe to Journal of Clinical Epidemiology
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect

      References

        • Freiman J.A.
        • Chalmers T.C.
        • Smith Jr., H.
        • Kuebler R.R.
        The importance of beta, the type II error and sample size in the design and interpretation of the randomized control trial. Survey of 71 “negative” trials.
        N Engl J Med. 1978; 299: 690-694
        • Chung K.C.
        • Kalliainen L.K.
        • Hayward R.A.
        Type II (beta) errors in the hand literature: the importance of power.
        J Hand Surg [Am]. 1998; 23: 20-25
        • Bailey C.S.
        • Fisher C.G.
        • Dvorak M.F.
        Type II error in the spine surgical literature.
        Spine. 2004; 29: 1146-1149
        • Brown C.G.
        • Kelen G.D.
        • Ashton J.J.
        • Werman H.A.
        The beta error and sample size determination in clinical trials in emergency medicine.
        Ann Emerg Med. 1987; 16: 183-187
        • Edlund M.J.
        • Overall J.E.
        • Rhoades H.M.
        Beta, or type II error in psychiatric controlled clinical trials.
        J Psychiatr Res. 1985; 19: 563-567
        • Hall J.C.
        The other side of statistical significance: a review of type II errors in the Australian medical literature.
        Aust N Z J Med. 1982; 12: 7-9
        • Williams J.L.
        • Hathaway C.A.
        • Kloster K.L.
        • Layne B.H.
        Low power, type II errors, and other statistical problems in recent cardiovascular research.
        Am J Physiol. 1997; 273: H487-493
        • Dimick J.B.
        • Diener-West M.
        • Lipsett P.A.
        Negative results of randomized clinical trials published in the surgical literature: equivalency or error?.
        Arch Surg. 2001; 136: 796-800
        • Fox N.
        • Mathers N.
        Empowering research: statistical power in general practice research.
        Fam Pract. 1997; 14: 324-329
        • Freedman K.B.
        • Bernstein J.
        Sample size and statistical power in clinical orthopaedic research.
        J Bone Joint Surg Am. 1999; 81: 1454-1460
        • Lochner H.V.
        • Bhandari M.
        • Tornetta 3rd, P.
        Type-II error rates (beta errors) of randomized trials in orthopaedic trauma.
        J Bone Joint Surg Am. 2001; 83: 1650-1655
        • Mittendorf R.
        • Arun V.
        • Sapugay A.M.
        The problem of the type II statistical error.
        Obstet Gynecol. 1995; 86: 857-859
        • Moher D.
        • Dulberg C.S.
        • Wells G.A.
        Statistical power, sample size, and their reporting in randomized controlled trials.
        JAMA. 1994; 272: 122-124
        • Sundaresan N.
        • Voorhies R.
        • Kwok K.L.
        • Thaler H.T.
        Hypothesis testing in neurosurgical trials.
        J Neurosurg. 1981; 54: 468-472
        • Williams H.C.
        • Seed P.
        Inadequate size of “negative” clinical trials in dermatology.
        Br J Dermatol. 1993; 128: 317-326
        • World Health Organization
        Safety of medicines: a guide to detecting and reporting adverse drug reactions.
        World Health Organization, Geneva2002
        • Connolly S.J.
        • Dorian P.
        • Roberts R.S.
        • Gent M.
        • Bailin S.
        • Fain E.S.
        • et al.
        Comparison of beta-blockers, amiodarone plus beta-blockers, or sotalol for prevention of shocks from implantable cardioverter defibrillators: the OPTIC Study: a randomized trial.
        JAMA. 2006; 295: 165-171
        • Kearon C.
        • Ginsberg J.S.
        • Julian J.A.
        • Douketis J.
        • Solymoss S.
        • Ockelford P.
        • et al.
        Comparison of fixed-dose weight-adjusted unfractionated heparin and low-molecular-weight heparin for acute treatment of venous thromboembolism.
        JAMA. 2006; 296: 935-942
        • Kremer J.M.
        • Genant H.K.
        • Moreland L.W.
        • Russell A.S.
        • Emery P.
        • Abud-Mendoza C.
        • et al.
        Effects of abatacept in patients with methotrexate-resistant active rheumatoid arthritis: a randomized trial.
        Ann Intern Med. 2006; 144: 865-876
        • Armstrong D.K.
        • Bundy B.
        • Wenzel L.
        • Huang H.Q.
        • Baergen R.
        • Lele S.
        • et al.
        Intraperitoneal cisplatin and paclitaxel in ovarian cancer.
        N Engl J Med. 2006; 354: 34-43
        • Legro R.S.
        • Barnhart H.X.
        • Schlaff W.D.
        • Carr B.R.
        • Diamond M.P.
        • Carson S.A.
        • et al.
        Clomiphene, metformin, or both for infertility in the polycystic ovary syndrome.
        N Engl J Med. 2007; 356: 551-566
        • Palumbo A.
        • Bringhen S.
        • Caravita T.
        • Merla E.
        • Capparella V.
        • Callea V.
        • et al.
        Oral melphalan and prednisone chemotherapy plus thalidomide compared with melphalan and prednisone alone in elderly patients with multiple myeloma: randomised controlled trial.
        Lancet. 2006; 367: 825-831
        • Myers E.R.
        • Silva S.G.
        • Hafley G.
        • Kunselman A.R.
        • Nestler J.E.
        • Legro R.S.
        Estimating live birth rates after ovulation induction in polycystic ovary syndrome: sample size calculations for the pregnancy in polycystic ovary syndrome trial.
        Contemp Clin Trials. 2005; 26: 271-280
        • Agresti A.
        Categorical data analysis.
        John Wiley and Sons, Inc., New York, NY2002
        • Martin A.A.
        • Herranz T.I.
        • Silva M.A.
        The Wilcoxon, Spearman, Fisher, #2-, student and Pearson tests and 2x2 tables.
        Statistician. 1995; 44: 441-450
      1. National Cancer Institute. Common toxicity criteria v.2. NCI; 1999 [updated 1999. Available at: http://ctep.cancer.gov/forms/CTCv20_4-30-992.pdf. [Accessed 2008 February 20]. April 30, 1999;

      2. Christensen R, Kristensen PK, Bartels EM, Bliddal H, Astrup A. Efficacy and safety of the weight-loss drug rimonabant: a meta-analysis of randomised trials. Lancet; 370(): 1706–1713.

        • Bongartz T.
        • Sutton A.J.
        • Sweeting M.J.
        • Buchan I.
        • Matteson E.L.
        • Montori V.
        Anti-TNF antibody therapy in rheumatoid arthritis and the risk of serious infections and malignancies: systematic review and meta-analysis of rare harmful effects in randomized controlled trials.
        JAMA. 2006; 295: 2275-2285
        • Nissen S.E.
        • Wolski K.
        Effect of rosiglitazone on the risk of myocardial infarction and death from cardiovascular causes.
        N Engl J Med. 2007; 356: 2457-2471
        • Lynd L.D.
        • O'Brien B.J.
        Advances in risk-benefit evaluation using probabilistic simulation methods: an application to the prophylaxis of deep vein thrombosis.
        J Clin Epidemiol. 2004; 57: 795-803
        • Sterne J.A.C.
        • Smith G.D.
        • Cox D.R.
        Sifting the evidence{–}what's wrong with significance tests? Another comment on the role of statistical methods.
        BMJ. 2001; 322: 226-231
        • Loke Y.K.
        • Derry S.
        Reporting of adverse drug reactions in randomised controlled trials—a systematic survey.
        BMC Clin Pharmacol. 2001; 1: 3