Advertisement

A method for calculating the fragility index of continuous outcomes

      Highlights

      • Fragility Index provides useful insight into how robust a statistically significant result really is.
      • Classic computations of Fragility Index are limited to dichotomous variables.
      • An iterative algorithm can be used to calculate Continuous Fragility Index (CFI) for continuous variables.
      • In settings where original data is not available, a multiple simulation technique can be used to estimate CFI.
      • The fragility of outcomes within the same study may vary considerably.

      Abstract

      Objective

      Clinicians’ overdependence on p-values to determine significance in clinical trials is common yet potentially misleading. The Fragility Index (FI) describes how robust a significant result is by determining the number of events the statistical significance hinges on. However, this concept cannot be applied to nondichotomous variables. We describe a method to calculate a Continuous Fragility Index (CFI) for continuous variables. We further provide a method to estimate CFI when original data is not available.

      Study Design and Setting

      An iterative substitution algorithm is described to calculate CFI prospectively from data or retrospectively from summary statistics and its response to variations in the data is reported. We then apply this method to a previously published review as a proof-of-concept.

      Results

      The CFI increases linearly with sample size, logarithmically with mean difference, and decreases exponentially with standard deviation. Forty-eight studies were included of which 30 had significant non-dichotomous outcomes. CFI and FI were uncorrelated and mean CFI was significantly higher than FI (9 vs. 2, P< 0.001).

      Conclusion

      Our algorithm extends fragility to continuous outcomes, expanding the applications of the fragility concept. The fragility of outcomes within a single study may vary based on variable type and should be evaluated independently.

      Keywords

      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'

      Subscribe:

      Subscribe to Journal of Clinical Epidemiology
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect

      References

      1. It's time to talk about ditching statistical significance.
        Nature. 2019; 567: 283https://doi.org/10.1038/d41586-019-00874-8
        • Walsh M
        • Srinathan S
        • McAuley D
        The statistical significance of randomized controlled trial results is frequently fragile: a case for a fragility index.
        J Clin Epidemiol. 2014; 67: 622-628
        • Ruxton GD.
        The unequal variance t-test is an underused alternative to Student's t-test and the Mann–Whitney U test.
        Behav Ecol. 2006; 17: 688-690https://doi.org/10.1093/beheco/ark016
        • Khan M
        • Evaniew N
        • Gichuru M
        • et al.
        The Fragility of Statistically Significant Findings From Randomized Trials in Sports Surgery: A Systematic Survey.
        Am J Sports Med. 2017; 45: 2164-2170https://doi.org/10.1177/0363546516674469
        • Team RC.
        R: A language and environment for statistical computing.
        2019
        • Windish DM
        • Huot SJ
        • Green ML.
        Medicine residents’ understanding of the biostatistics and results in the medical literature.
        JAMA. 2007; 298: 1010-1022https://doi.org/10.1001/jama.298.9.1010
        • Horton NJ
        • Switzer SS.
        Statistical Methods in the Journal.
        N Engl J Med. 2005; 353: 1977-1979https://doi.org/10.1056/NEJM200511033531823
        • Wasserstein RL
        • Lazar NA.
        The ASA's Statement on p-Values: Context, Process, and Purpose.
        Am Stat. 2016; 70: 129-133https://doi.org/10.1080/00031305.2016.1154108
        • Nuzzo R.
        Scientific method: statistical errors.
        Nature. 2014; 506: 150-152https://doi.org/10.1038/506150a
        • Woods KL
        • Fletcher S
        • Roffe C
        • Haider Y.
        Intravenous magnesium sulphate in suspected acute myocardial infarction: results of the second Leicester Intravenous Magnesium Intervention Trial (LIMIT-2).
        Lancet. 1992; 339: 1553-1558https://doi.org/10.1016/0140-6736(92)91828-V
        • Ranstam J.
        Why the P-value culture is bad and confidence intervals a better alternative.
        Osteoarthr Cartil. 2012; 20: 805-808https://doi.org/10.1016/j.joca.2012.04.001
        • Barnett AG
        • Wren JD.
        Examination of CIs in health and medical journals from 1976 to 2019: An observational study.
        BMJ Open. 2019; 9e032506https://doi.org/10.1136/bmjopen-2019-032506
        • Chavalarias D
        • Wallach JD
        • Li AHT
        • Ioannidis JPA.
        Evolution of reporting P values in the biomedical literature, 1990-2015.
        JAMA - J Am Med Assoc. 2016; 315: 1141-1148https://doi.org/10.1001/jama.2016.1952
        • Feinstein AR.
        The unit fragility index: An additional appraisal of “statistical significance” for a contrast of two proportions.
        J Clin Epidemiol. 1990; 43: 201-209https://doi.org/10.1016/0895-4356(90)90186-S
        • Robinson CM
        • Jenkins PJ
        • White TO
        • Ker A
        • Will E.
        Primary arthroscopic stabilization for a first-time anterior dislocation of the shoulder: A randomized, double-blind trial.
        J Bone Jt Surg - Ser A. 2008; 90: 708-721https://doi.org/10.2106/JBJS.G.00679
        • Zhao J
        • Huangfu X
        • He Y.
        The role of medial retinaculum plication versus medial patellofemoral ligament reconstruction in combined procedures for recurrent patellar instability in adults.
        Am J Sports Med. 2012; 40: 1355-1364https://doi.org/10.1177/0363546512439193
        • Desnoyers A
        • Nadler MB
        • Wilson BE
        • Amir E.
        A critique of the fragility index.
        Lancet Oncol. 2019; 20: e552https://doi.org/10.1016/S1470-2045(19)30583-2
        • Carter RE
        • McKie PM
        • Storlie CB.
        The Fragility Index: a P-value in sheep's clothing?.
        Eur Heart J. 2016; 38: 346-348https://doi.org/10.1093/eurheartj/ehw495
        • Acuna SA
        • Sue-Chue-Lam C
        • Dossa F.
        The fragility index - p values reimagined, flaws and all.
        JAMA Surg. 2019; 154: 674https://doi.org/10.1001/jamasurg.2019.0567
        • Machado T
        • Duarte GS
        • Gonçalves N
        • Ferreira JJ
        • Costa J.
        A critique of the fragility index.
        Lancet Oncol. 2019; 20: e553https://doi.org/10.1016/S1470-2045(19)30581-9
        • Del Paggio JC
        • Tannock IF.
        A critique of the fragility index – Authors’ reply.
        Lancet Oncol. 2019; 20: e554https://doi.org/10.1016/S1470-2045(19)30580-7