Series: Living Systematic Review| Volume 91, P38-46, November 2017

Living systematic reviews: 3. Statistical methods for updating meta-analyses

Published:September 10, 2017DOI:


      A living systematic review (LSR) should keep the review current as new research evidence emerges. Any meta-analyses included in the review will also need updating as new material is identified. If the aim of the review is solely to present the best current evidence standard meta-analysis may be sufficient, provided reviewers are aware that results may change at later updates. If the review is used in a decision-making context, more caution may be needed. When using standard meta-analysis methods, the chance of incorrectly concluding that any updated meta-analysis is statistically significant when there is no effect (the type I error) increases rapidly as more updates are performed. Inaccurate estimation of any heterogeneity across studies may also lead to inappropriate conclusions. This paper considers four methods to avoid some of these statistical problems when updating meta-analyses: two methods, that is, law of the iterated logarithm and the Shuster method control primarily for inflation of type I error and two other methods, that is, trial sequential analysis and sequential meta-analysis control for type I and II errors (failing to detect a genuine effect) and take account of heterogeneity. This paper compares the methods and considers how they could be applied to LSRs.


      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to Journal of Clinical Epidemiology
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Elliott J.H.
        • Turner T.
        • Clavisi O.
        • Thomas J.
        • Higgins J.P.
        • Mavergames C.
        • et al.
        Living systematic reviews: an emerging opportunity to narrow the evidence-practice gap.
        PLoS Med. 2017; 11: e1001603
        • Elliott J.H.
        • Synnot A.
        • Turner T.
        • Simmonds M.
        • Akl E.A.
        • McDonald S.
        • et al.
        Living systematic reviews: 1. Introduction—the why, what, when and how.
        J Clin Epidemiol. 2017; 91: 23-30
        • Thomas J.
        • Noel-Storr A.
        • Marshall I.
        • Wallace B.
        • McDonald S.
        • Mavergames C.
        • et al.
        Living systematic reviews: 2. Combining human and machine effort.
        J Clin Epidemiol. 2017; 91: 31-37
        • Ioannidis J.P.
        • Contopoulos-Ioannidis D.G.
        • Lau J.
        Recursive cumulative meta-analysis: a diagnostic for the evolution of total randomized evidence from group and individual patient data.
        J Clin Epidemiol. 1999; 52: 281-291
        • Akl E.A.
        • Meerpohl J.J.
        • Elliott J.
        • Kahale L.A.
        • Schunemann H.J.
        Living systematic reviews: 4. Living guideline recommendations.
        J Clin Epidemiol. 2017; 91: 47-53
        • Borm G.F.
        • Donders A.R.
        Updating meta-analyses leads to larger type I errors than publication bias.
        J Clin Epidemiol. 2009; 62: 825-830.e10
        • Sutton A.J.
        • Cooper N.J.
        • Jones D.R.
        • Lambert P.C.
        • Thompson J.R.
        • Abrams K.R.
        Evidence-based sample size calculations based upon updated meta-analysis.
        Stat Med. 2007; 26: 2479-2500
        • Turner R.M.
        • Bird S.M.
        • Higgins J.P.
        The impact of study size on meta-analyses: examination of underpowered studies in Cochrane reviews.
        PLoS One. 2013; 8: e59202
        • Lan K.K.G.
        • Demets D.L.
        Discrete sequential boundaries for clinical-trials.
        Biometrika. 1983; 70: 659-663
        • Whitehead J.
        A unified theory for sequential clinical trials.
        Stat Med. 1999; 18: 2271-2286
        • Pogue J.M.
        • Yusuf S.
        Cumulating evidence from randomized trials: utilizing sequential monitoring boundaries for cumulative meta-analysis.
        Control Clin Trials. 1997; 18: 580-593
        • Brok J.
        • Thorlund K.
        • Wetterslev J.
        • Gluud C.
        Apparently conclusive meta-analyses may be inconclusive–trial sequential analysis adjustment of random error risk due to repetitive testing of accumulating data in apparently conclusive neonatal meta-analyses.
        Int J Epidemiol. 2009; 38: 287-298
        • Thorlund K.
        • Devereaux P.J.
        • Wetterslev J.
        • Guyatt G.
        • Ioannidis J.P.
        • Thabane L.
        • et al.
        Can trial sequential monitoring boundaries reduce spurious inferences from meta-analyses?.
        Int J Epidemiol. 2009; 38: 276-286
        • Wetterslev J.
        • Thorlund K.
        • Brok J.
        • Gluud C.
        Trial sequential analysis may establish when firm evidence is reached in cumulative meta-analysis.
        J Clin Epidemiol. 2008; 61: 64-75
        • O'Brien P.C.
        • Fleming T.R.
        A multiple testing procedure for clinical trials.
        Biometrics. 1979; 35: 549-556
        • Cook J.A.
        • Hislop J.
        • Altman D.G.
        • Fayers P.
        • Briggs A.H.
        • Ramsay C.R.
        • et al.
        Specifying the target difference in the primary outcome for a randomised controlled trial: guidance for researchers.
        Trials. 2015; 16: 12
        • Wetterslev J.
        • Thorlund K.
        • Brok J.
        • Gluud C.
        Estimating required information size by quantifying diversity in random-effects model meta-analyses.
        BMC Med Res Methodol. 2009; 9: 86
        • Whitehead A.
        A prospectively planned cumulative meta-analysis applied to a series of concurrent clinical trials.
        Stat Med. 1997; 16: 2901-2913
        • Higgins J.P.
        • Whitehead A.
        • Simmonds M.
        Sequential methods for random-effects meta-analysis.
        Stat Med. 2011; 30: 903-921
        • Shuster J.J.
        • Neu J.
        A Pocock approach to sequential meta-analysis of clinical trials.
        Res Synth Methods. 2013; 4: 269-279
        • Hu M.
        • Cappelleri J.C.
        • Lan K.K.
        Applying the law of iterated logarithm to control type I error in cumulative meta-analysis of binary outcomes.
        Clin Trials. 2007; 4: 329-340
        • Lan K.K.G.
        • Hu M.
        • Cappelleri J.C.
        Applying the law of iterated logarithm to cumulative meta-analyses of a continuous endpoint.
        Stat Sin. 2003; 13: 1135-1145
        • Sacks H.S.
        • Chalmers T.C.
        • Blum A.L.
        • Berrier J.
        • Pagano D.
        Endoscopic hemostasis. An effective therapy for bleeding peptic ulcers.
        JAMA. 1990; 264: 494-499
      1. Trial Sequential Analysis 2017. Available at Accessed September 9, 2017.

        • Nikolakopoulou A.
        • Mavridis D.
        • Egger M.
        • Salanti G.
        Continuously updated network meta-analysis and statistical monitoring for timely decision-making.
        Stat Methods Med Res. 2016; (Available at Accessed September 9, 2017)
        • Veroniki A.A.
        • Straus S.E.
        • Soobiah C.
        • Elliott M.J.
        • Tricco A.C.
        A scoping review of indirect comparison methods and applications using individual patient data.
        BMC Med Res Methodol. 2016; 16: 47
        • Imberger G.
        • Gluud C.
        • Wetterslev J.
        Comments on 'Sequential methods for random-effects meta-analysis'.
        Stat Med. 2011; 30: 2965-2966
        • Roloff V.
        • Higgins J.P.
        • Sutton A.J.
        Planning future studies based on the conditional power of a meta-analysis.
        Stat Med. 2013; 32: 11-24
        • Langan D.
        • Higgins J.P.
        • Gregory W.
        • Sutton A.J.
        Graphical augmentations to the funnel plot assess the impact of additional evidence on a meta-analysis.
        J Clin Epidemiol. 2012; 65: 511-519
        • Berkey C.S.
        • Mosteller F.
        • Lau J.
        • Antman E.M.
        Uncertainty of the time of first significance in random effects cumulative meta-analysis.
        Control Clin Trials. 1996; 17: 357-371
        • Greenland S.
        • Senn S.J.
        • Rothman K.J.
        • Carlin J.B.
        • Poole C.
        • Goodman S.N.
        • et al.
        Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations.
        Eur J Epidemiol. 2016; 31: 337-350