Exclusion of studies with no events in both arms in meta-analysis impacted the conclusions

  • Chang Xu
    Chinese Evidence-Based Medicine Center & Cochrane China, West China Hospital, Sichuan University, Chengdu, China
    Search for articles by this author
  • Ling Li
    Chinese Evidence-Based Medicine Center & Cochrane China, West China Hospital, Sichuan University, Chengdu, China
    Search for articles by this author
  • Lifeng Lin
    Department of Statistics, Florida State University, Tallahassee, FL, USA
    Search for articles by this author
  • Haitao Chu
    Division of Biostatistics, School of Public Health, University of Minnesota, Minneapolis, MN, USA
    Search for articles by this author
  • Lehana Thabane
    Department of Health Research Methods, Evidence, and Impact, McMaster University, 1280 Main Street West, Hamilton, Ontario L8S 4K1, Canada

    Biostatistics Unit, Father Sean O'Sullivan Research Centre, St Joseph's Healthcare, 3rd Floor, Martha Wing, Room H-325, 50 Charlton Avenue East, Hamilton, Ontario L8N 4A6, Canada
    Search for articles by this author
  • Kang Zou
    Chinese Evidence-Based Medicine Center & Cochrane China, West China Hospital, Sichuan University, Chengdu, China
    Search for articles by this author
  • Xin Sun
    Corresponding author. Chinese Evidence-Based Medicine Center & Cochrane China, West China Hospital, Sichuan University, Chengdu, China. Tel.: + 86 189 80606047; fax: 028-85164187.
    Chinese Evidence-Based Medicine Center & Cochrane China, West China Hospital, Sichuan University, Chengdu, China
    Search for articles by this author



      Classical meta-analyses routinely treated studies with no events in both arms noninformative and excluded them from analyses. This study assessed whether such studies contain information and have an influence on the conclusions of meta-analyses.

      Study Design and Setting

      We collected meta-analyses of binary outcomes with at least one study having no events in both arms from Cochrane systematic reviews (2003–2018). We used the generalized linear mixed model to reanalyze these meta-analyses by two approaches: one including studies with no events in both arms and one excluding such studies. The magnitude and direction of odds ratio (OR), P value, and width of 95% confidence interval (CI) were compared. A simulation study was conducted to examine the robustness of results.


      We identified 442 meta-analyses. In comparing paired meta-analyses that included studies with no events in both arms vs. those not, 8 (1.80%) resulted in different directions on OR; 41 (9.28%) altered conclusions on statistical significance. Substantial changes occurred on P value (55.66% increased and 44.12% decreased) and the width of 95% CI (50.68% inflated and 49.32% declined) when excluding studies with no events. Simulation study confirmed these findings.


      Studies with no events in both arms are not necessarily noninformative. Excluding such studies may alter conclusions.


      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to Journal of Clinical Epidemiology
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Feldman B.
        • Wang E.
        • Willan A.
        • Szalai J.P.
        The randomized placebo-phase design for clinical trials.
        J Clin Epidemiol. 2001; 54: 550-557
        • Noyes J.
        • Booth A.
        • Flemming K.
        • Garside R.
        • Harden A.
        • Lewin S.
        • et al.
        Cochrane Qualitative and Implementation Methods Group guidance series-paper 3: methods for assessing methodological limitations, data extraction and synthesis, and confidence in synthesized qualitative findings.
        J Clin Epidemiol. 2018; 97: 49-58
        • Bhaumik D.K.
        • Amatya A.
        • Normand S.L.
        • Greenhouse J.
        • Kaizar E.
        • Neelon B.
        • et al.
        Meta-analysis of rare and adverse event data.
        Expert Rev Pharmacoecon Outcomes Res. 2002; 2: 367-379
        • Kuss O.
        Statistical methods for meta-analyses including information from studies without any events-add nothing to nothing and succeed nevertheless.
        Stat Med. 2015; 34: 1097-1116
        • Rücker G.
        • Schwarzer G.
        • Carpenter J.
        • Olkin I.
        Why add anything to nothing? The arcsine difference as a measure of treatment effect in meta-analysis with zero cells.
        Stat Med. 2009; 28: 721-738
      1. Higgins J.P.T. Thomas J. Chandler J. Cumpston M. Li T. Page M J. Cochrane Handbook for Systematic Reviews of Interventions. 2nd Edition. John Wiley & Sons, Chichester, UK2019
        • Marshall I.J.
        • Marshall R.
        • Wallace B.
        • Brassey J.
        • Thomas J.
        Rapid reviews may produce different results to systematic reviews: a meta-epidemiological study.
        J Clin Epidemiol. 2018; 109: 30-41
        • Page M.J.
        • McKenzie J.E.
        • Kirkham J.
        • Dwan K.
        • Kramer S.
        • Green S.
        • et al.
        Bias due to selective inclusion and reporting of outcomes and analyses in systematic reviews of randomised trials of healthcare interventions.
        Cochrane Database Syst Rev. 2014; 10: MR000035
        • Cheng J.
        • Pullenayegum E.
        • Marshall J.K.
        • Iorio A.
        • Thabane L.
        Impact of including or excluding both-armed zero-event studies on using standard meta-analysis methods for rare event outcome: a simulation study.
        BMJ Open. 2016; 6: e010983
        • Sweeting M.J.
        • Sutton A.J.
        • Lambert P.C.
        What to add to nothing? Use and avoidance of continuity corrections in meta-analysis of sparse data.
        Stat Med. 2004; 23: 1351-1375
        • Cochrane Library
        Cochrane database of systematic reviews.
        (Available at)
        • Lang D.T.
        R as a web client–the RCurl package. J Stat Softw.
        (Available at)
        • Ma X.
        • Lin L.
        • Qu Z.
        • Zhu M.
        • Chu H.
        Performance of between-study heterogeneity measurements in the Cochrane library.
        Epidemiology. 2018; 29: 821-824
        • Lin L.
        • Chu H.
        • Murad M.H.
        • Hong C.
        • Qu Z.
        • Cole S.R.
        • et al.
        Empirical comparison of publication bias tests in meta-analysis.
        J Gen Intern Med. 2018; 33: 1260-1267
        • Friedrich J.O.
        • Adhikari N.K.
        • Beyene J.
        Inclusion of zero total event trials in meta-analyses maintains analytic consistency and incorporates all available data.
        BMC Med Res Methodol. 2007; 7: 5
        • Debray T.P.A.
        • Moons K.G.M.
        • Abozaid G.M.A.
        • Koffijberg H.
        • Riley R.D.
        Individual participant data meta-analysis for a binary outcome: one-stage or two-stage?.
        PLOS One. 2013; 8: e60650
        • Burke D.L.
        • Ensor J.
        • Riley R.D.
        Meta-analysis using individual participant data: one-stage and two-stage approaches, and why they may differ.
        Stat Med. 2017; 36: 855-875
        • Altman D.G.
        • Bland J.M.
        Measurement in medicine: the analysis of method comparison studies.
        Statistician. 1983; 32: 307-317
        • Brockhaus A.C.
        • Bender R.
        • Skipka G.
        The Peto odds ratio viewed as a new effect measure.
        Stat Med. 2014; 33: 4861-4874
        • Betas D.
        • Mächler M.
        • Bolker B.
        • Walker S.
        Fitting linear mixed-effects models using lme4.
        J Stat Softw. 2015; 67: 1-42
        • Jackson D.
        • Law M.
        • Stijnen T.
        • Viechtbauer W.
        • White I.R.
        A comparison of seven random-effects models for meta-analyses that estimate the summary odds ratio.
        Stat Med. 2018; 37: 1059-1085
        • Kontopantelis E.
        A comparison of one-stage vs two-stage individual patient data meta-analysis methods: a simulation study.
        Res Synth Methods. 2018; 9: 417-430
      2. Sutton AJ, Abrams KR. Bayesian methods in meta-analysis and evidence synthesis. Stat Methods Med Res, 10(4):277-303.

        • Bai O.
        • Chen M.
        • Wang X.
        Bayesian estimation and testing in random effects meta-analysis of rare binary adverse events.
        Stat Biopharm Res. 2016; 8: 49-59
        • Chu H.
        • Nie L.
        • Chen Y.
        • Huang Y.
        • Sun W.
        Bivariate random effects models for meta-analysis of comparative studies with binary outcomes: methods for the absolute risk difference and relative risk.
        Stat Methods Med Res. 2012; 21: 621-633
        • Ren Y.
        • Lin L.
        • Lian Q.
        • Zou H.
        • Chu H.
        Real-world performance of meta-analysis methods for double-zero-event studies with dichotomous outcomes using the Cochrane database of systematic reviews.
        J Gen Intern Med. 2019; 34: 960-968
        • Liu D.
        • Liu R.Y.
        • Xie M.
        Exact meta-analysis approach for discrete data and its application to 2 × 2 tables with rare events.
        J Am Stat Assoc. 2014; 109: 1450-1465