Advertisement

A proposed framework to guide evidence synthesis practice for meta-analysis with zero-events studies

Open AccessPublished:February 12, 2021DOI:https://doi.org/10.1016/j.jclinepi.2021.02.012

      Abstract

      Objective

      In evidence synthesis practice, researchers often face the problem of how to deal with zero-events. Inappropriately dealing with zero-events studies may lead to research waste and mislead healthcare practice. We propose a framework to guide researchers to better deal with zero-events in meta-analysis.

      Study design and setting

      We used two dimensions, one with respect to the total events count across all studies in the comparative arms in a meta-analysis, and a second with respect to whether included studies have single or both arms with zero-events, to establish the framework for the classification of meta-analysis with zero-events studies. A dataset from Cochrane systematic reviews was used to evaluate the classification.

      Results

      The proposed framework classifies meta-analysis with zero-events studies into six subtypes. The classification matched well to the large real-world dataset. The applicability of existing methods for zero-events were then presented under each meta-analysis subtype based on this framework, with a 5-step principle to help researchers in evidence synthesis practice.

      Conclusions

      The proposed framework should be considered by researchers when making decisions on the selection of the synthesis methods in a meta-analysis. It also provides a reasonable basis for the development of methodological guidelines to deal with zero-events in meta-analysis.

      Keywords

      What is new?

        Key findings

      • We propose a framework to classify meta-analysis with zero-events studies, which can clearly classify meta-analysis with zero-events studies into six subtypes; the classification matched well to the large real-world dataset.
      • Based on the proposed framework, we developed a flow diagram which presents the applicability of the existing methods for each type of meta-analysis with zero-events studies to guide synthesis practice.

        What this study adds to what is known?

      • In evidence synthesis practice for meta-analysis of adverse events, researchers often face the problem of how to deal with zero-events. They usually classify studies with zero-events as single-arm-zero-events studies and double-arm-zero-events studies as an effort to deal with zero-events. While in meta-analysis of evidence synthesis practice, referring to this classification is not the best strategy to select synthesis methods to deal with the problem of zero-events, which may lead to research waste and mislead healthcare practice. In this study, we propose a new framework to classify meta-analysis with zero-events studies to guide researchers to better deal with zero-events in meta-analysis.

        What are the implications, and what should be changed?

      • The proposed framework should be routinely considered by researchers when making decisions on the selection of the synthesis methods in a meta-analysis.
      • Statisticians and methodologists may add other existing methods or novel methods in our flow diagram to enhance selection of relevant methods for evidence synthesis. They can also compare the statistical properties of the methods under each subtype to provide more concrete guidance for meta-analysis authors.

      1. Introduction

      Meta-analysis has been widely used to synthesis evidence for studies of the same topic, through either a fixed-effect or a random-effect scheme, as an effort to obtain more precise and comprehensive results for decision-making. While for meta-analysis of rare events, researchers often face the problem of how to deal with zero-events. Zero-events generally occur when the risk of events is low, the sample size is small, or the follow-up period is short - these are frequently seen with safety outcomes [
      • Zorzela L.
      • Golder S.
      • Liu Y.
      • Pilkington K.
      • Hartling L.
      • Joffe A.
      • et al.
      Quality of reporting in systematic reviews of adverse events: systematic review.
      ,
      • Chou R.
      • Aronson N.
      • Atkins D.
      • Ismaila A.S.
      • Santaguida P.
      • Smith D.H.
      • et al.
      AHRQ series paper 4: assessing harms when comparing medical interventions: AHRQ and the effective health-care program.
      ]. In a survey of a random sample of 500 Cochrane systematic reviews, 30–34.4% of the meta-analyses contained studies with zero-events [
      • Vandermeer B.
      • Bialy L.
      • Hooton N.
      • Hartling L.
      • Klassen T.P.
      • Johnston B.C.
      • et al.
      Meta-analyses of safety data: a comparison of exact versus asymptotic methods.
      ,

      Kuss O., Wandrey M., Kunze M. How frequent are meta-analyses with “double-zero” studies in systematic reviews? 2009. Available from: http://www.egms.de/static/de/meetings/gmds2009/09gmds155.shtml [Accessed at 15th-Jan, 2021]

      . These zero-events pose special challenges for making statistical inferences in single trials as well as meta-analyses.
      Risk ratio (RR), odds ratio (OR), and risk difference (RD) are the most frequently used effect measures in meta-analyses of binary outcomes, where the RR and OR are relative measures and RD is an absolute measure [
      ]. In the presence of zero-events, it is difficult to properly calculate the RR and OR directly because their denominators are zero. This is not the case for the RD because it is the difference of two risks. However, the variances of all RR, OR, and RD could not be defined using the classical delta method. With advances in methodology, the estimation of the variance of RD for zero-events has been well-addressed [
      • Klingenberg B.
      A new and improved confidence interval for the Mantel-Haenszel risk difference.
      ,
      • Sweeting M.J.
      • Sutton A.J.
      • Lambert P.C.
      What to add to nothing? Use and avoidance of continuity corrections in meta-analysis of sparse data.
      . However, a universal consensus is still not achieved to handle zero-events for the RR and OR.
      In order to develop statistical models to deal with the problems posed by zero-events, researchers usually classify studies with zero-events into two groups: (1) single-arm-zero-events studies, where zero-events only occur in one of the arms; and (2) double-arm-zero-events studies, where zero-events occur in both arms. This classification provides a foundation for statisticians to investigate methods for handling zero-events; current recommendations for meta-analysis with zero-events also rely mainly on this classification to guide evidence synthesis, for example, in the Cochrane Handbook, Peto's OR and Mantel–Haenszel (MH) methods are recommended to deal with single-arm-zero-events [
      ]. For double-arm-zero-events studies, most software automatically discard them from a meta-analysis (eg, RevMan, current version 5.4.1 [

      Cochrane Informatics and Knowledge Management Department. In: RevMan 5.4.1 2020. Available from: https://training.cochrane.org/online-learning/core-software-cochrane-reviews/revman. Accessed by 21-Oct, 2020.

      ]). Some researchers claim that such studies are noninformative because they add nothing to the pooled effects of relative measures [
      • Finkelstein M.O.
      • Levin B.
      Meta-analysis of “sparse” data: perspectives from the Avandia cases.
      ]. However, this practice could be problematic from both clinical and statistical points. First, double-arm-zero-events studies contribute an important source of evidence; discarding them is a deviation of the principle of evidence-based medicine and would lead to research waste. Second, such studies are not necessarily “noninformative,” depending on the methods used to synthesize them, for example, for studies with balanced sample sizes, zero-events in both arms indicate there are comparable treatment effects [
      • Kuss O.
      Statistical methods for meta-analyses including information from studies without any events-add nothing to nothing and succeed nevertheless.
      ]. Our previous study has shown that excluding studies with no events in both arms can result in about 10% of the meta-analyses change the statistical significance [
      • Xu C.
      • Li L.
      • Lin L.
      • Chu H.
      • Thabane L.
      • Zou K.
      • et al.
      Exclusion of studies with no events in both arms in meta-analysis impacted the conclusions.
      ].
      We recognize that the currently used classification for studies with zero-events is not primarily aimed to guide meta-analysis, but to deal with zero-events in single studies. In meta-analysis of evidence synthesis practice, referring to this classification is therefore not the best strategy to deal with the problem of zero-events. In some cases, both single-arm-zero-events studies and double-arm-zero-events studies could be included in meta-analyses. Some methods (eg, Peto's OR) may provide a good solution to single-arm-zero-events studies, but they perform poorly in double-arm-zero-events studies, and vice versa [
      • Kuss O.
      Statistical methods for meta-analyses including information from studies without any events-add nothing to nothing and succeed nevertheless.
      ,
      • Bradburn M.J.
      • Deeks J.J.
      • Berlin J.A.
      • Russell Localio A.
      Much ado about nothing: a comparison of the performance of meta-analytical methods with rare events.
      . Systematic reviewers may become confused when choosing methods to deal with zero-events studies. More importantly, inappropriate choices in evidence synthesis cause research waste (eg, discarding double-arm-zero-events) and could mislead healthcare practice.
      In this study, we propose a framework to classify meta-analysis with zero-events studies to provide a comprehensive and simple approach for meta-analysis practitioners.

      2. Classification for meta-analysis with zero-events studies

      On a meta-level basis, the proposed framework treats the whole meta-analysis as a unit, instead of each study as a unit, to classify meta-analysis with zero-events studies into different subtypes. The idea is driven from the classification method for single studies mentioned earlier. In order to better understand the framework, we first present the classification method for single studies: under single-arm-zero-events, the total events are zero in one of the arms; under double-arm-zero-events, the total events are zero in both arms. It can be seen that the classification is determined by the total events in each arm.
      Following this logic, in our meta-level framework, the total events count in each arm across studies is considered as one of the dimensions for the classification. This dimension defines two types of meta-analysis: (1) the total events count is zero in one arm or in both arms; and (2) the total events count is zero in neither arm (ie, total events >0 in both arms). The second type illustrates that even when the total events count is not zero in both arms, a meta-analysis could still contain studies with zero-events. Only using this dimension is therefore insufficient to clearly classify a meta-analysis, because for each of these two types of meta-analysis, single-arm-zero, double-arm-zero studies, or both could be involved, where different methods are required to deal with these different cases. Therefore, a second dimension is further needed to distinguish the distribution of zero-events studies in a meta-analysis: (1) meta-analysis with only single-arm-zero-events studies; (2) meta-analysis with only double-arm-zero-events studies; and (3) meta-analysis with a mixture of both single-arm-zero-events and double-arm-zero-events studies.
      These two dimensions can clearly classify meta-analysis with zero-events studies into six subtypes (2 × 3) as follows:
      • (1)
        MA-SZ: Meta-analysis contains zero-events only occurring in single arms, no double-arm-zero-events studies are included, and the total events count in neither arm is zero;
      • (2)
        MA-MZ: Meta-analysis contains zero-events occurring in both single and double arms, and the total events count in neither arm is zero;
      • (3)
        MA-DZ: Meta-analysis contains zero-events only occurring in double arms, and the total events count in neither arm is zero;
      • (4)
        MA-CSZ: Meta-analysis contains zero-events occurring in single arms, and no double-arm-zero-events studies are included, while the total events count in one of the arms is zero;
      • (5)
        MA-CMZ: Meta-analysis contains zero-events occurring in both single arm and double arms, while the total events count in one of the arms is zero;
      • (6)
        MA-CDZ: Meta-analysis only includes double-arm-zero-events studies, while the total events count in both arms are zero.
      Figure 1 illustrates the framework of the six subtypes of meta-analysis. In the above abbreviations, “MA” stands for meta-analysis, “S” for single arms, “M” for mixture of single arms and double arms, and “D” for double arms. Because subtypes 4–6 require that all included studies in a meta-analysis are zero-events studies, we define them as “completely zero-events-studies meta-analysis,” and “C” stands for “completely.” It should be noted that subtypes 1–3 may also only contain studies with zero-events, but studies without zero-events could be present.
      Fig. 1
      Fig. 1The proposed framework for meta-analysis with zero-events studies.

      3. Rationale for the classification

      Differentiating meta-analyses with zero-events studies based on the two dimensions is in light of the applicability of current methods for meta-analysis. Current methods for zero-events studies can be classified as one-stage approaches or two-stage approaches. In a two-stage approach (also known as the standard meta-analysis approach), the study-specific effect sizes and standard errors are obtained or estimated from the included studies at the first stage, and then they are synthesized at the second stage [
      • Burke D.L.
      • Ensor J.
      • Riley R.D.
      Meta-analysis using individual participant data: one-stage and two-stage approaches, and why they may differ.
      ,
      • Scotti L.
      • Rea F.
      • Corrao G.
      One-stage and two-stage meta-analysis of individual participant data led to consistent summarized evidence: lessons learned from combining multiple databases.
      . Under the two-stage approach, available methods for zero-events include the continuity/empirical correction [
      • Sweeting M.J.
      • Sutton A.J.
      • Lambert P.C.
      What to add to nothing? Use and avoidance of continuity corrections in meta-analysis of sparse data.
      ], Peto's OR [
      • Yusuf S.
      • Peto R.
      • Lewis J.
      • Collins R.
      • Sleight P.
      Beta blockade during and after myocardial infarction: an overview of the randomised trials.
      ], MH [
      • Mantel N.
      • Haenszel W.
      Statistical aspects of the analysis of data from retrospective studies of disease.
      ], two-stage Bayesian methods [
      • Pateras K.
      • Nikolakopoulos S.
      • Roes K.C.B.
      Prior distributions for variance parameters in a sparse-event meta-analysis of a few small trials.
      ,
      • Warn D.E.
      • Thompson S.G.
      • Spiegelhalter D.J.
      Bayesian random effects meta-analysis of trials with binary outcomes: methods for the absolute risk difference and relative risk scales.
      , the exact p-function method [
      • Liu D.
      • Liu R.Y.
      • Xie M.
      Exact meta-analysis approach for discrete data and its application to 2 × 2 tables with rare events.
      ], the arcsine-based transformation (eg, arcsine difference [
      • Rücker G.
      • Schwarzer G.
      • Carpenter J.
      • Olkin I.
      Why add anything to nothing? The arcsine difference as a measure of treatment effect in meta-analysis with zero cells.
      ]), etc. Among these methods, Peto's OR is not applicable for dealing with double-zero-events studies. It should be noted that these methods are estimation methods used at the first stage to obtain the study-specific effects; strictly speaking, they are not meta-analytic methods since meta-analytic methods refer to as a type of “weighted average” procedure [
      ,
      • DerSimonian R.
      • Laird N.
      Meta-analysis in clinical trials.
      ,
      • Doi S.A.
      • Barendregt J.J.
      • Khan S.
      • Thalib L.
      • Williams G.M.
      Advances in the meta-analysis of heterogeneous clinical trials I: the inverse variance heterogeneity model.
      . For the one-stage approaches, there is no need to obtain study-specific effect sizes. Instead, each study is treated as a stratum or cluster, and the overall effect size can be obtained by the population average method [
      • Burke D.L.
      • Ensor J.
      • Riley R.D.
      Meta-analysis using individual participant data: one-stage and two-stage approaches, and why they may differ.
      ,
      • Scotti L.
      • Rea F.
      • Corrao G.
      One-stage and two-stage meta-analysis of individual participant data led to consistent summarized evidence: lessons learned from combining multiple databases.
      , which includes the generalized linear mixed models (GLMMs) [
      • Ju K.
      • Lin L.
      • Chu H.
      • Cheng L.L.
      • Xu C.
      Laplace approximation, penalized quasi-likelihood, and adaptive Gauss-Hermite quadrature for generalized linear mixed models: towards meta-analysis of binary outcome with sparse data.
      ,
      • Simmonds M.C.
      • Higgins J.P.
      A general framework for the use of logistic regression models in meta-analysis.
      , the generalized estimating equations (GEEs) [
      • Zeger S.L.
      • Liang K.Y.
      • Albert P.S.
      Models for longitudinal data: a generalized estimating equation approach.
      ], the stratified exact regression [
      • Kuss O.
      Statistical methods for meta-analyses including information from studies without any events-add nothing to nothing and succeed nevertheless.
      ], one-stage Bayesian methods [
      • McIntosh A.I.
      • Doros G.
      • Jones-López E.C.
      • Gaeddert M.
      • Jenkins H.E.
      • Marques-Rodrigues P.
      • et al.
      Extensions to Bayesian generalized linear mixed effects models for household tuberculosis transmission.
      ], the beta-binominal methods [
      • Bakbergenuly I.
      • Kulinskaya E.
      Beta-binomial model for meta-analysis of odds ratios.
      ], etc.
      One-stage approaches provide a valid solution to synthesize studies with zero-events, especially in the presence of studies with zero-events in both arms [
      • Ju K.
      • Lin L.
      • Chu H.
      • Cheng L.L.
      • Xu C.
      Laplace approximation, penalized quasi-likelihood, and adaptive Gauss-Hermite quadrature for generalized linear mixed models: towards meta-analysis of binary outcome with sparse data.
      ]. However, when the total events count in the meta-analysis is zero in one of the arms or both arms, one-stage approaches cannot be used for synthesis due to the problem of separation, for example, an empirical logistic regression could not be conducted in a single study with no events in one of the arms (or both arms). Therefore, for MA-CSZ, MA-CMZ, and MA-CDZ, most of the one-stage methods (eg, GLMM, GEE, stratified exact regression) are not applicable. As such, we need to distinguish whether the total events count is zero at the meta-level (dimension 1).
      Unlike one-stage approaches, two-stage approaches generally provide a valid solution to synthesizing studies with zero-events in a single arm, but they are less preferable for studies with zero-events in both arms. As a result, without a prior information or post hoc correction, most of the two-stage methods are not appropriate to be used to measure relative risks in MA-MZ, MZ-DZ, MA-CMZ, and MA-CDZ, for example, the Peto’ OR, the MH OR/RR. For this reason, we must consider another dimension to distinguish meta-analysis with single-arm-zero-events studies, double-arm-zero-events studies, or both (dimension 2).
      Under our proposed framework, the applicability of each method can be clearly mapped to guide evidence synthesis practice.

      4. Real-world data for the classification

      We utilized data from Cochrane Database of Systematic Reviews (CDSR) to empirically assess the frequencies and proportions of each type of meta-analysis. Data from the CDSR containing meta-analyses published from January 2003 to May 2018 were collected; meta-analyses with two or more studies were considered. A detailed description for the data collection process has been documented elsewhere [
      • Ma X.
      • Lin L.
      • Qu Z.
      • Zhu M.
      • Chu H.
      Performance of between-study heterogeneity measurements in the Cochrane Library.
      ,
      • Lin L.
      • Chu H.
      • Murad M.H.
      • Hong C.
      • Qu Z.
      • Cole S.R.
      • et al.
      Empirical comparison of publication bias tests in meta-analysis.
      ,
      • Furuya-Kanamori L.
      • Xu C.
      • Lin L.
      • Doan T.
      • Chu H.
      • Thalib L.
      • et al.
      P value-driven methods were underpowered to detect publication bias: analysis of Cochrane review meta-analyses.
      ].
      From 6,781 CDSRs, we obtained a total of 61,090 meta-analyses with binary outcomes, of which 21,288 meta-analyses were identified with zero-events studies (34.85%). We further summarized the frequency and proportion of each meta-analysis subtype with zero-event studies (Fig. 2). The MA-SZ was the most common subset, including 19,308 (90.70%) meta-analyses; the MA-CSZ subset ranked the second, including 1,322 (6.21%) meta-analyses. The MA-MZ subset included 295 (1.39%) meta-analyses, and the MA-DZ subset included 147 (0.69%) meta-analyses. For both MA-CMZ and MA-CDZ subsets, 108 (0.51%) meta-analyses were identified. These real-world data support the relevance and appropriateness of the proposed classification framework.
      Fig. 2
      Fig. 2Proportion of each type of meta-analysis with zero-events studies based on the proposed framework.

      5. How to use the framework for evidence synthesis practice?

      Within the proposed framework, the degree of difficulty in synthesizing evidence increases gradually from subtype 1 to 6, for example, in subtype 1 of MA-SZ, almost all methods can be used to deal with zero-events for the synthesis; however, in subtype 6 of MA-CDZ, only a limited number of methods can be used. To help researchers to better understand how to apply existing methods to perform the research synthesis, we developed a flow diagram to guide synthesis practice under the proposed framework, as seen in Figure 3.
      Fig. 3
      Fig. 3Flow diagram to guide evidence synthesis practice.
      This diagram lists the applicability of the aforementioned methods for each type of meta-analysis, for example, Peto's OR could be used in MA-SZ and MA-CSZ, but is not suitable for MA-MZ, MA-DZ, MA-CMZ, and MA-CDZ where double-arm-zero-events studies are involved. The GLMMs could be used in MA-SZ, MA-MZ, and MA-DZ, but not in MA-CSZ, MA-CMZ, and MA-CDZ where the total events count is zero in either or both arms. Statisticians and methodologists may add other existing methods or novel methods in this flow diagram (where we marked “…” in the diagram) to enhance selection of relevant methods for evidence synthesis. They can also compare the statistical properties of the methods under each subtype to provide more concrete guidance for meta-analysis authors.
      For meta-analysis authors, the flow diagram clearly indicates which methods could be appropriately used under each classification. We recommend meta-analysis authors refer to the following 5 steps in Box 1 to conduct evidence synthesis practice for meta-analysis with zero-events studies.
      Box 1Five steps for meta-analysis with zero-events studies.
      1.Check for zero-events studies;
      2.Check the framework of the classification to identify the meta-analysis belongs to which of MA-SZ, MA-MZ, MA-DZ, MA-CSZ, MA-CMZ, and MA-CDZ;
      3.Check all available methods that could be used under the classification in the flow diagram;
      4.Conduct the meta-analysis by using one or more appropriate methods;
      5.Conduct sensitivity analysis using other available methods to test whether the results are robust.
      We used the meta-analysis by Liu et al [
      • Liu J.
      • Li L.
      • Deng K.
      • et al.
      Incretin based treatments and mortality in patients with type 2 diabetes: systematic review and meta-analysis.
      ] as a real-world example to illustrate how the framework works. This meta-analysis assessed the safety of glucagon-like peptide-1 receptor agonists and dipeptidyl peptidase-4 inhibitors on mortality for patients with type-2 diabetes. We used the data for vildagliptin (vs. placebo, lifestyle modification, or active antihyperglycemic drugs) which consisted of 26 randomized controlled trials (Fig. 4). It should be noted that this is an extreme example because the event rate was very low and it contained a mega trial.
      • Step 1.
        We identified that the dataset contains 13 trials with zero-events occurring in both arms and 6 trials with zero-events occurring in a single arm. The total events count in neither arm is zero.
      • Step 2.
        According to our proposed framework, this meta-analysis is classified as MA-MZ.
      • Step 3.
        Using the flow diagram, we identify that there are at least 9 methods to deal with this type of meta-analysis.
      • Step 4.
        We used the MH methods with RD as an effect measure in the main analysis. In order increase the statistical power to detect a signal of adverse events, we used the fixed-effect analytic model for pooling [
        • Poole C.
        • Greenland S.
        Random-effects meta-analyses are not always conservative.
        ].
      • Step 5.
        An empirical correction [
        • Sweeting M.J.
        • Sutton A.J.
        • Lambert P.C.
        What to add to nothing? Use and avoidance of continuity corrections in meta-analysis of sparse data.
        ] with the fixed-effect analytic model was used as the first sensitivity analysis. The random-intercept GLMM method (random baseline risks but fixed treatment effect) [
        • Xu C.
        • Li L.
        • Lin L.
        • Chu H.
        • Thabane L.
        • Zou K.
        • et al.
        Exclusion of studies with no events in both arms in meta-analysis impacted the conclusions.
        ] under the one-stage framework was used as the second sensitivity analysis.
      Fig. 4
      Fig. 4A real-world example of vildagliptin vs. control on mortality for patients with type-2 diabetes.
      The MetaXL 5.3 software (EpiGear International Pty Ltd, Australian) and R 3.51 (R foundation) were used for the analyses. Figure 4 presents the pooled results for the main analysis. The pooled RD for mortality on vildagliptin compared to control was −0.001 (95%CI: −0.004, 0.002, P = 0.43). Our two sensitivity analyses confirmed the results. We may further show the potential research waste by excluding 13 studies with zero-events occurring in both arms with the same analytic method: the pooled RD was −0.001 (95%CI: −0.005, 0.002, P = 0.36). In this case, excluding these studies slightly underestimated the confidence interval and led to a smaller P-value.

      6. Discussion

      We propose a practical framework to classify meta-analysis with zero-events studies, which is useful for meta-analysis authors when they face the problem of zero-events. Based on this framework, we estimated the frequencies of each subtype of meta-analysis with zero-events studies in a large real-world dataset and found that our theoretical framework had a good match. We further developed a 5-step principle to guide researchers to use the framework and select appropriate methods for meta-analysis. Our framework will provide a reasonable basis for the development of methodological guidelines to deal with zero-events in meta-analysis.
      In the flow diagram, we did not list the methods for relative and absolute measures separately. Instead, we indicated which methods could be used with absolute risk (eg, RD) or relative measures (eg, Peto's OR) for the synthesis under the six subtypes of meta-analysis, for example, we indicated that for MA-MZ, MA-DZ, MA-CMZ, and MA-CDZ, the MH can be used for synthesis when measuring the effect by the RD, because double-arm-zero-events studies would be automatically discarded when the MH method with OR/RR is utilized. It should be noted that, except for the situation that all the included studies have zero cell in the same arm, the single-arm-zero-studies do not require a continuity correction under MH method [
      ,
      • Sweeting M.J.
      • Sutton A.J.
      • Lambert P.C.
      What to add to nothing? Use and avoidance of continuity corrections in meta-analysis of sparse data.
      . While in most of the software, a continuity correction is routinely employed for MH OR/RR. Indeed, the RD is an appealing effect estimator to deal with zero-events over the OR and RR as neither prior information nor post hoc correction is needed. One can also use the RD to estimate the number needed to treat (NNT) to reflect the average number of patients who need to be treated to prevent one additional harmful outcome. While empirical evidence suggested that RD may increase the heterogeneity than OR and RR [
      • Deeks J.J.
      Issues in the selection of a summary statistic for meta-analysis of clinical trials with binary outcomes.
      ], at least, this may lead to the downgrade of the confidence of the evidence [
      • Guyatt G.H.
      • Oxman A.D.
      • Kunz R.
      • Brozek J.
      • Alonso-Coello P.
      • Rind D.
      • et al.
      GRADE guidelines 6. Rating the quality of evidence–imprecision.
      ]. There is currently no consensus about the selection of effect estimator for rare events. But it is clear that the nonreporting problem of the events contributes a major source of systematic error that may biased the effect estimates, regardless of the OR, RR, and RD.
      Notably, the continuity/empirical correction is not routinely employed for studies with no events in both arms; this requires researchers to indicate such a scheme or do it manually in the program. Moreover, several methods (eg, GLMM, beta-binominal) listed in the flow diagram are not available in many meta-analytical programs, but they can be done by other programs (eg, meglm, betabin in Stata, lme4 in R). Researchers are highly recommended to collaborate with methodologists or statisticians when conducting zero-events meta-analysis. The flow diagram provides a group of methods for each type of meta-analysis, thus further investigation is needed to determine which method would perform better within each meta-analysis type.
      It should be highlighted that this framework is based on post hoc classification. This means systematic reviewers can only determine the subtype after the data for a meta-analysis are obtained. Researchers cannot use this framework to help them to make an a priori synthesis plan as they do not know which subtype of meta-analysis (with zero-events) they will be facing. However, it is highly recommended for authors to develop a synthesis plan in advance to avoid selective reporting, including for meta-analysis that may involve zero-events [
      • Zhou Y.
      • Zhu B.
      • Lin L.
      • Kwong J.S.W.
      • Xu C.
      Protocols for meta-analysis of intervention safety seldom specified methods to deal with rare events.
      ,
      • Zorzela L.
      • Loke Y.K.
      • Ioannidis J.P.
      • Golder S.
      • Santaguida P.
      • Altman D.G.
      • et al.
      PRISMA harms checklist: improving harms reporting in systematic reviews.
      . One possible and practical solution is that researchers consider all six possible situations a priori and predefine one or more available methods for each situation in their protocols.
      In the 5-step principle, we recommend that researchers conduct a sensitivity analysis by using other available synthesis methods. We recommend this based on the consideration of the unstable nature of such types of meta-analysis as the total events in both arms are still rare, leading to a large amount of uncertainty for inference. Once there are conflicting results in sensitivity analysis, there is currently no well-established evidence to support a preferred conclusion. However, for safety outcome, the “at worst” principle could be considered that the one presenting a more obvious signal of harms is the first concern [
      ]. In addition, two additional recommendations beyond the framework are worthwhile for consideration. First, before the evidence synthesis, we recommend researchers to estimate the minimal sample size for detecting a potential difference [
      • Wetterslev J.
      • Jakobsen J.C.
      • Gluud C.
      Trial Sequential Analysis in systematic reviews with meta-analysis.
      ,
      • Brok J.
      • Thorlund K.
      • Gluud C.
      • Wetterslev J.
      Trial sequential analysis reveals insufficient information size and potentially false positive results in many meta-analyses.
      . In some situations, statistically nonsignificant results may be due to an insufficient sample size, not to a true nonsignificant difference. Researchers have advocated that putting too much focus on statistical significance is not desirable [
      • Sterne J.A.
      • Davey Smith G.
      Sifting the evidence-what's wrong with significance tests?.
      ]. Second, the GRADE (Grading of Recommendations, Assessment, Development and Evaluations) system [
      • Guyatt G.H.
      • Oxman A.D.
      • Kunz R.
      • Brozek J.
      • Alonso-Coello P.
      • Rind D.
      • et al.
      GRADE guidelines 6. Rating the quality of evidence–imprecision.
      ] or other similar ranking systems should be used to grade the certainty of the results; this can be judged along with the power-analysis results.
      For the selection of concrete methods to deal with zero-events studies, there is currently no universal consensus on which method should be used. A specific recommendation on concrete methods under each subtype may be unrealistic at present. However, we may offer some practices to be avoided as follows, which could be useful for future systematic reviewers. First, it is not recommended to use continuity correction (adding 0.5) when the sample sizes of the two arms are imbalanced (eg, 1:2) because well-established evidence has shown that it would lead to large bias or even reverse the effects [
      • Sweeting M.J.
      • Sutton A.J.
      • Lambert P.C.
      What to add to nothing? Use and avoidance of continuity corrections in meta-analysis of sparse data.
      ]. The empirical correction could be an alternative for continuity correction in this case [
      • Sweeting M.J.
      • Sutton A.J.
      • Lambert P.C.
      What to add to nothing? Use and avoidance of continuity corrections in meta-analysis of sparse data.
      ]. Second, it is not recommended to use Bayesian methods when most or all the included studies are both-arm-zero-events studies. This is because posterior estimates are susceptible to the prior distribution as the weight of sample information is small, even if noninformative prior is used in such case [
      • Xu C.
      • Li L.
      • Lin L.
      • Chu H.
      • Thabane L.
      • Zou K.
      • et al.
      Exclusion of studies with no events in both arms in meta-analysis impacted the conclusions.
      ]. Third, it is not recommended to use the “treat-as-one-trial” method, say, adding the cases and sample sizes together of included studies, as this would lead to Simpson's paradox and mislead the healthcare practice [
      • Altman D.G.
      • Deeks J.J.
      Meta-analysis, Simpson's paradox, and the number needed to treat.
      ]. Fourth, for safety outcomes, it is not suggested to use the conservative analytic model (eg, random-effects model). This has been clarified by Poole & Greenland [
      • Liu J.
      • Li L.
      • Deng K.
      • et al.
      Incretin based treatments and mortality in patients with type 2 diabetes: systematic review and meta-analysis.
      ] and in the Cochrane Handbook [
      ]. As the Cochrane Handbook documented: “incorporation of heterogeneity into an estimate of a treatment effect should be a secondary consideration when attempting to produce estimates of effects from sparse data – the primary concern is to discern whether there is any signal of an effect in the data” [
      ].

      7. Conclusions

      Meta-analysis with zero-events studies represents a difficult and complex issue in evidence synthesis practice. The framework we proposed could be considered for researchers to make better decisions on the selection of the synthesis methods in their meta-analysis. This framework could be useful for statisticians to examine the properties and applicability of various methods for meta-analysis with zero-events studies. In addition, our framework provides a reasonable basis for the development of methodological guidelines to deal with zero-events in meta-analysis.

      Data sharing

      No data involved.

      Credit author statement

      Chang Xu: Conceptualization, Methodology, Data curation, Writing- Original draft preparation.
      Luis Furuya-Kanamori: Methodology, Data curation, Writing- Reviewing and Editing.
      Liliane Zorzela: Methodology, Writing- Reviewing and Editing.
      Lifeng Lin: Data curation, Writing- Reviewing and Editing.
      Sunita Vohra: Methodology, Writing- Reviewing and Editing.

      Acknowledgments

      None.

      References

        • Zorzela L.
        • Golder S.
        • Liu Y.
        • Pilkington K.
        • Hartling L.
        • Joffe A.
        • et al.
        Quality of reporting in systematic reviews of adverse events: systematic review.
        BMJ. 2014; 348: f7668
        • Chou R.
        • Aronson N.
        • Atkins D.
        • Ismaila A.S.
        • Santaguida P.
        • Smith D.H.
        • et al.
        AHRQ series paper 4: assessing harms when comparing medical interventions: AHRQ and the effective health-care program.
        J Clin Epidemiol. 2010; 63: 502-512
        • Vandermeer B.
        • Bialy L.
        • Hooton N.
        • Hartling L.
        • Klassen T.P.
        • Johnston B.C.
        • et al.
        Meta-analyses of safety data: a comparison of exact versus asymptotic methods.
        Stat Methods Med Res. 2009; 18: 421-432
      1. Kuss O., Wandrey M., Kunze M. How frequent are meta-analyses with “double-zero” studies in systematic reviews? 2009. Available from: http://www.egms.de/static/de/meetings/gmds2009/09gmds155.shtml [Accessed at 15th-Jan, 2021]

      2. Higgins JPT Thomas J Chandler J Cumpston M Li T Page M.J. Cochrane handbook for systematic reviews of interventions. 2nd Edition. John Wiley & Sons, Chichester (UK)2019
        • Klingenberg B.
        A new and improved confidence interval for the Mantel-Haenszel risk difference.
        Stat Med. 2014; 33: 2968-2983
        • Sweeting M.J.
        • Sutton A.J.
        • Lambert P.C.
        What to add to nothing? Use and avoidance of continuity corrections in meta-analysis of sparse data.
        Stat Med. 2004; 23: 1351-1375
      3. Cochrane Informatics and Knowledge Management Department. In: RevMan 5.4.1 2020. Available from: https://training.cochrane.org/online-learning/core-software-cochrane-reviews/revman. Accessed by 21-Oct, 2020.

        • Finkelstein M.O.
        • Levin B.
        Meta-analysis of “sparse” data: perspectives from the Avandia cases.
        Jurimetrics J. 2012; 52: 123-153
        • Kuss O.
        Statistical methods for meta-analyses including information from studies without any events-add nothing to nothing and succeed nevertheless.
        Stat Med. 2015; 34: 1097-1116
        • Xu C.
        • Li L.
        • Lin L.
        • Chu H.
        • Thabane L.
        • Zou K.
        • et al.
        Exclusion of studies with no events in both arms in meta-analysis impacted the conclusions.
        J Clin Epidemiol. 2020; 123: 91-99
        • Bradburn M.J.
        • Deeks J.J.
        • Berlin J.A.
        • Russell Localio A.
        Much ado about nothing: a comparison of the performance of meta-analytical methods with rare events.
        Stat Med. 2007; 26: 53-77
        • Burke D.L.
        • Ensor J.
        • Riley R.D.
        Meta-analysis using individual participant data: one-stage and two-stage approaches, and why they may differ.
        Stat Med. 2017; 36: 855-875
        • Scotti L.
        • Rea F.
        • Corrao G.
        One-stage and two-stage meta-analysis of individual participant data led to consistent summarized evidence: lessons learned from combining multiple databases.
        J Clin Epidemiol. 2018; 95: 19-27
        • Yusuf S.
        • Peto R.
        • Lewis J.
        • Collins R.
        • Sleight P.
        Beta blockade during and after myocardial infarction: an overview of the randomised trials.
        Progress Cardiovasc Dis. 1985; 27: 335-371
        • Mantel N.
        • Haenszel W.
        Statistical aspects of the analysis of data from retrospective studies of disease.
        J Natl Cancer Inst. 1959; 22: 719-748
        • Pateras K.
        • Nikolakopoulos S.
        • Roes K.C.B.
        Prior distributions for variance parameters in a sparse-event meta-analysis of a few small trials.
        Pharm Stat. 2020; (10.1002/pst.2053)https://doi.org/10.1002/pst.2053
        • Warn D.E.
        • Thompson S.G.
        • Spiegelhalter D.J.
        Bayesian random effects meta-analysis of trials with binary outcomes: methods for the absolute risk difference and relative risk scales.
        Stat Med. 2002; 21: 1601-1623
        • Liu D.
        • Liu R.Y.
        • Xie M.
        Exact meta-analysis approach for discrete data and its application to 2 × 2 tables with rare events.
        J Am Stat Assoc. 2014; 109: 1450-1465
        • Rücker G.
        • Schwarzer G.
        • Carpenter J.
        • Olkin I.
        Why add anything to nothing? The arcsine difference as a measure of treatment effect in meta-analysis with zero cells.
        Stat Med. 2009; 28: 721-738
        • DerSimonian R.
        • Laird N.
        Meta-analysis in clinical trials.
        Control Clin Trials. 1986; 7: 177
        • Doi S.A.
        • Barendregt J.J.
        • Khan S.
        • Thalib L.
        • Williams G.M.
        Advances in the meta-analysis of heterogeneous clinical trials I: the inverse variance heterogeneity model.
        Contemp Clin Trials. 2015; 45: 130-138
        • Ju K.
        • Lin L.
        • Chu H.
        • Cheng L.L.
        • Xu C.
        Laplace approximation, penalized quasi-likelihood, and adaptive Gauss-Hermite quadrature for generalized linear mixed models: towards meta-analysis of binary outcome with sparse data.
        BMC Med Res Methodol. 2020; 20: 152
        • Simmonds M.C.
        • Higgins J.P.
        A general framework for the use of logistic regression models in meta-analysis.
        Stat Methods Med Res. 2016; 25: 2858-2877
        • Zeger S.L.
        • Liang K.Y.
        • Albert P.S.
        Models for longitudinal data: a generalized estimating equation approach.
        Biometrics. 1988; 44: 1049-1060
        • McIntosh A.I.
        • Doros G.
        • Jones-López E.C.
        • Gaeddert M.
        • Jenkins H.E.
        • Marques-Rodrigues P.
        • et al.
        Extensions to Bayesian generalized linear mixed effects models for household tuberculosis transmission.
        Stat Med. 2017; 36: 2522-2532
        • Bakbergenuly I.
        • Kulinskaya E.
        Beta-binomial model for meta-analysis of odds ratios.
        Stat Med. 2017; 36: 1715-1734
        • Ma X.
        • Lin L.
        • Qu Z.
        • Zhu M.
        • Chu H.
        Performance of between-study heterogeneity measurements in the Cochrane Library.
        Epidemiology. 2018; 29: 821-824
        • Lin L.
        • Chu H.
        • Murad M.H.
        • Hong C.
        • Qu Z.
        • Cole S.R.
        • et al.
        Empirical comparison of publication bias tests in meta-analysis.
        J Gen Intern Med. 2018; 33: 1260-1267
        • Furuya-Kanamori L.
        • Xu C.
        • Lin L.
        • Doan T.
        • Chu H.
        • Thalib L.
        • et al.
        P value-driven methods were underpowered to detect publication bias: analysis of Cochrane review meta-analyses.
        J Clin Epidemiol. 2020; 118: 86-92
        • Liu J.
        • Li L.
        • Deng K.
        • et al.
        Incretin based treatments and mortality in patients with type 2 diabetes: systematic review and meta-analysis.
        BMJ. 2017; 357: j2499
        • Poole C.
        • Greenland S.
        Random-effects meta-analyses are not always conservative.
        Am J Epidemiol. 1999; 150: 469-475
        • Deeks J.J.
        Issues in the selection of a summary statistic for meta-analysis of clinical trials with binary outcomes.
        Stat Med. 2002; 21: 1575-1600
        • Guyatt G.H.
        • Oxman A.D.
        • Kunz R.
        • Brozek J.
        • Alonso-Coello P.
        • Rind D.
        • et al.
        GRADE guidelines 6. Rating the quality of evidence–imprecision.
        J Clin Epidemiol. 2011; 64: 1283-1293
        • Zhou Y.
        • Zhu B.
        • Lin L.
        • Kwong J.S.W.
        • Xu C.
        Protocols for meta-analysis of intervention safety seldom specified methods to deal with rare events.
        J Clin Epidemiol. 2020; (S0895-4356(20)31109-4)
        • Zorzela L.
        • Loke Y.K.
        • Ioannidis J.P.
        • Golder S.
        • Santaguida P.
        • Altman D.G.
        • et al.
        PRISMA harms checklist: improving harms reporting in systematic reviews.
        BMJ. 2016; 352: i157
        • Wetterslev J.
        • Jakobsen J.C.
        • Gluud C.
        Trial Sequential Analysis in systematic reviews with meta-analysis.
        BMC Med Res Methodol. 2017; 17: 39
        • Brok J.
        • Thorlund K.
        • Gluud C.
        • Wetterslev J.
        Trial sequential analysis reveals insufficient information size and potentially false positive results in many meta-analyses.
        J Clin Epidemiol. 2008; 61: 763-769
        • Sterne J.A.
        • Davey Smith G.
        Sifting the evidence-what's wrong with significance tests?.
        BMJ. 2001; 322: 226-231
        • Altman D.G.
        • Deeks J.J.
        Meta-analysis, Simpson's paradox, and the number needed to treat.
        BMC Med Res Methodol. 2002; 2: 3