The statistical significance of meta-analyses is frequently fragile: definition of a fragility index for meta-analyses

  • Ignacio Atal
    Correspondence
    Corresponding author. Centre d’Epidémiologie Clinique, Hôpital Hôtel-Dieu, 1, place du parvis Notre Dame, 75004 Paris, France. Tel.: +33 1 42 34 87 65; Fax: +33 142348790.
    Affiliations
    Centre d’Épidémiologie Clinique, Hôpital Hôtel-Dieu, Paris, France

    Team METHODS, Centre of Research in Epidemiology and Statistics Sorbonne, Paris Cité–CRESS Inserm UMR1153, Paris, France

    Université Paris Descartes, Paris, France
    Search for articles by this author
  • Raphaël Porcher
    Affiliations
    Centre d’Épidémiologie Clinique, Hôpital Hôtel-Dieu, Paris, France

    Team METHODS, Centre of Research in Epidemiology and Statistics Sorbonne, Paris Cité–CRESS Inserm UMR1153, Paris, France

    Université Paris Descartes, Paris, France
    Search for articles by this author
  • Isabelle Boutron
    Affiliations
    Centre d’Épidémiologie Clinique, Hôpital Hôtel-Dieu, Paris, France

    Team METHODS, Centre of Research in Epidemiology and Statistics Sorbonne, Paris Cité–CRESS Inserm UMR1153, Paris, France

    Université Paris Descartes, Paris, France
    Search for articles by this author
  • Philippe Ravaud
    Affiliations
    Centre d’Épidémiologie Clinique, Hôpital Hôtel-Dieu, Paris, France

    Team METHODS, Centre of Research in Epidemiology and Statistics Sorbonne, Paris Cité–CRESS Inserm UMR1153, Paris, France

    Université Paris Descartes, Paris, France

    Epidemiology Department, Mailman School of Public Health, Columbia University, New York, NY, USA
    Search for articles by this author

      Abstract

      Objectives

      Meta-analyses inform clinical practice by summarizing treatment effect estimates based on results from several trials. However, the statistical significance of a meta-analysis (i.e., whether the pooled treatment effect is statistically significant or not) may rely on the outcome of only a few patients from specific trials in the meta-analysis. We aimed to evaluate the extent to which the statistical significance of meta-analyses can be changed (from statistically significant to nonsignificant, or vice versa) after modifying the event status of patients in specific arms of specific trials.

      Methods

      We conducted a cross-sectional analysis of meta-analyses of trials with a binary outcome from Cochrane Systematic Reviews. We defined the fragility index of meta-analyses as the minimum number of patients from one or more trials included in the meta-analysis for whom an event-status modification (i.e., changing an event to nonevent or a nonevent to event) would change the statistical significance of the pooled treatment effect. For statistically significant and nonsignificant meta-analyses, we evaluated the fragility index, the ratio between the fragility index and the total number of participants included in the trials, and the ratio between the fragility index and the total number of events.

      Results

      Our sample comprised 906 meta-analyses: 400 and 506 had statistically significant and nonsignificant pooled treatment effects, respectively. For statistically significant meta-analyses, the median fragility index was 12 (Q1–Q3: 4–33); for 29% the fragility index was 5 or less. Overall, 43% and 9% meta-analyses would have become nonsignificant if the event status was modified for less than 1% of the total participants in one or several specific trials, and for less than 1% of the total number of events, respectively. These proportions were similar for statistically nonsignificant meta-analyses. Overall, the statistical significance of 33% of all meta-analyses depended on the event status of five or fewer participants from one or more specific trials.

      Conclusion

      The statistical significance of meta-analyses often depends on the outcome of a few patients. The fragility index of meta-analyses may help in interpreting the conclusions of meta-analyses.

      Keywords

      To read this article in full you will need to make a payment

      References

        • Murad M.H.
        • Montori V.M.
        • Ioannidis J.P.A.
        • Jaeschke R.
        • Devereaux P.J.
        • Prasad K.
        • et al.
        How to read a systematic review and meta-analysis and apply the results to patient care: users’ guides to the medical literature.
        JAMA. 2014; 312: 171-179
        • Dechartres A.
        • Trinquart L.
        • Boutron I.
        • Ravaud P.
        Influence of trial sample size on treatment effect estimates: meta-epidemiological study.
        BMJ. 2013; 346: f2304
        • Dechartres A.
        • Altman D.G.
        • Trinquart L.
        • Boutron I.
        • Ravaud P.
        Association between analytic strategy and estimates of treatment outcomes in meta-analyses.
        JAMA. 2014; 312: 623-630
        • Walsh M.
        • Srinathan S.K.
        • McAuley D.F.
        • Mrkobrada M.
        • Levine O.
        • Ribic C.
        • et al.
        The statistical significance of randomized controlled trial results is frequently fragile: a case for a Fragility Index.
        J Clin Epidemiol. 2014; 67: 622-628
        • Dechartres A.
        • Trinquart L.
        • Atal I.
        • Moher D.
        • Dickersin K.
        • Boutron I.
        • et al.
        Evolution of poor reporting and inadequate methods over time in 20 920 randomised controlled trials included in Cochrane reviews: research on research study.
        BMJ. 2017; 357: j2490
        • Guido S.
        Meta: an R package for meta-analysis.
        R News. 2007; 7: 40-45
        • Young C.
        • von Dadelszen P.
        • Alfirevic Z.
        Instruments for chorionic villus sampling for prenatal diagnosis.
        Cochrane Database Syst Rev. 2013; : CD000114
        • Njei B.
        • Kongnyuy E.J.
        • Kumar S.
        • Okwen M.P.
        • Sankar M.J.
        • Mbuagbaw L.
        Optimal timing for antiretroviral therapy initiation in patients with HIV infection and concurrent cryptococcal meningitis.
        Cochrane Database Syst Rev. 2013; : CD009012
        • Dechartres A.
        • Ravaud P.
        • Atal I.
        • Riveros C.
        • Boutron I.
        Association between trial registration and treatment effect estimates: a meta-epidemiological study.
        BMC Med. 2016; 14: 100
        • George S.L.
        • Buyse M.
        Data fraud in clinical trials.
        Clin Investig. 2015; 5: 161-173
        • Shen C.
        • Shamsudeen I.
        • Farrokhyar F.
        • Sabri K.
        Fragility of results in ophthalmology randomized controlled trials: a systematic review.
        Ophthalmology. 2018; 125: 642-648
        • Chase Kruse B.
        • Matt Vassar B.
        Unbreakable? An analysis of the fragility of randomized trials that support diabetes treatment guidelines.
        Diabetes Res Clin Pract. 2017; 134: 91-105
        • Carter R.E.
        • McKie P.M.
        • Storlie C.B.
        The fragility index: a P-value in sheep’s clothing?.
        Eur Heart J. 2017; 38: 346-348
        • Sutton A.J.
        • Cooper N.J.
        • Jones D.R.
        • Lambert P.C.
        • Thompson J.R.
        • Abrams K.R.
        Evidence-based sample size calculations based upon updated meta-analysis.
        Stat Med. 2007; 26: 2479-2500
        • Guyatt G.H.
        • Oxman A.D.
        • Kunz R.
        • Brozek J.
        • Alonso-Coello P.
        • Rind D.
        • et al.
        GRADE guidelines 6. Rating the quality of evidence—imprecision.
        J Clin Epidemiol. 2011; 64: 1283-1293
        • Charles P.
        • Giraudeau B.
        • Dechartres A.
        • Baron G.
        • Ravaud P.
        Reporting of sample size calculation in randomised controlled trials: review.
        BMJ. 2009; 338: b1732
        • Garcia-Alamino J.M.
        • Bankhead C.
        • Heneghan C.
        • Pidduck N.
        • Perera R.
        Impact of heterogeneity and effect size on the estimation of the optimal information size: analysis of recently published meta-analyses.
        BMJ Open. 2017; 7: e015888
        • Sterne J.A.C.
        • Cox D.R.
        • Smith G.D.
        Sifting the evidence—what’s wrong with significance tests?Another comment on the role of statistical methods.
        BMJ. 2001; 322: 226-231
        • Boutron I.
        • Ravaud P.
        Misrepresentation and distortion of research in biomedical literature.
        Proc Natl Acad Sci U S A. 2018; 115: 2613-2619
        • Chavalarias D.
        • Wallach J.D.
        • Li A.H.T.
        • Ioannidis J.P.A.
        Evolution of reporting P values in the biomedical literature, 1990-2015.
        JAMA. 2016; 315: 1141-1148