Methods for living guidelines: early guidance based on practical experience. Article 5: decisions on methods for evidence synthesis and recommendation development for living guidelines



      Producing living guidelines requires making important decisions about methods for evidence identification, appraisal, and integration to allow the living mode to function. Clarifying what these decisions are and the trade-offs between options is necessary. This article provides living guideline developers with a framework to enable them to choose the most suitable model for their living guideline topic, question, or context.

      Study Design and Setting

      We developed this guidance through an iterative process informed by interviews, feedback, and a consensus process with an international group of living guideline developers.


      Several key decisions need to be made both before commencing and throughout the continual process of living guideline development and maintenance. These include deciding what approach is taken to the systematic review process; decisions about methods to be applied for the evidence appraisal process, including the use of unpublished data; and selection of “triggers” to incorporate new studies into living guideline recommendations. In each case, there are multiple options and trade-offs.


      We identify trade-offs and important decisions to be considered throughout the living guideline development process. The most appropriate, and most sustainable, mode of development and updating will be dependent on the choices made in each of these areas.

      Graphical abstract


      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to Journal of Clinical Epidemiology
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Alderson L.J.H.
        • Alderson P.
        • Tan T.
        Median life span of a cohort of National Institute for Health and Care Excellence clinical guidelines was about 60 months.
        J Clin Epidemiol. 2014; 67: 52-55
        • World Health Organization
        WHO handbook for guideline development. World Health Organisation, Geneva, Switzerland2014
        • SIGN
        SIGN50 A guideline developer’s handbook.
        Scottish Intercollegiate Guidelines Network, Edinburgh2019
        • Steinberg E.
        • Greenfield S.
        • Wolman D.M.
        • Mancher M.
        • Graham R.
        Clinical practice guidelines we can trust.
        National Academies Press(US), Washington, DC2011
        • Akl E.A.
        • Meerpohl J.J.
        • Elliott J.
        • Kahale L.A.
        • Schünemann H.J.
        • Agoritsas T.
        • et al.
        Living systematic reviews: 4. Living guideline recommendations.
        J Clin Epidemiol. 2017; 91: 47-53
        • Elliott J.
        • Lawrence R.
        • Minx J.C.
        • Oladapo O.T.
        • Ravaud P.
        • Tendal Jeppesen B.
        • et al.
        Decision makers need constantly updated evidence synthesis.
        Nature. 2021; 600: 383-385
        • Martínez García L.
        • Arévalo-Rodríguez I.
        • Solà I.
        • Haynes R.B.
        • Vandvik P.O.
        • Alonso-Coello P.
        Strategies for monitoring and updating clinical practice guidelines: a systematic review.
        Implement Sci. 2012; 7: 1-10
        • National COVID-19 Clinical Evidence Taskforce
        Australian guidelines for the clinical care of people with COVID-19.
        (Available at)
        Date accessed: December 1, 2022
        • Fraile Navarro D.
        • Tendal B.
        • Tingay D.
        • Vasilunas N.
        • Anderson L.
        • Best J.
        • et al.
        Clinical care of children and adolescents with COVID-19: recommendations from the national COVID-19 clinical evidence taskforce.
        Med J Aust. 2022; 216: 255-263
        • Vogel J.P.
        • Tendal B.
        • Giles M.
        • Whitehead C.
        • Burton W.
        • Chakraborty S.
        • et al.
        Clinical care of pregnant and postpartum women with COVID-19: living recommendations from the national COVID-19 clinical evidence taskforce.
        Aust N Z J Obstet Gynaecol. 2020; 60: 840-851
        • Cheyne S.
        • Lindley R.I.
        • Smallwood N.
        • Tendal B.
        • Chapman M.
        • Fraile Navarro D.
        • et al.
        Care of older people and people requiring palliative care with COVID-19: guidance from the Australian National COVID-19 Clinical Evidence Taskforce.
        Med J Aust. 2022; 216: 203-208
        • Bhimraj A.
        • Morgan R.L.
        • Shumaker A.H.
        • Lavergne V.
        • Baden L.
        • Cheng V.C.C.
        • et al.
        Infectious Diseases Society of America Guidelines on the Treatment and Management of Patients with COVID-19. Infectious Diseases Society of America, 2022 (Version 10.1.1. Available at)
        • Hanson K.E.
        • Altayar O.
        • Caliendo A.M.
        • Arias C.A.
        • Englund J.A.
        • Hayden M.K.
        • et al.
        The infectious diseases society of America guidelines on the diagnosis of coronavirus disease 2019 (COVID-19): antigen testing.
        Clin Infect Dis. 2021; (ciaa1343)
        • National Institute for Health and Care Excellence (NICE)
        COVID-19 rapid guideline: managing COVID-19.
        (Available at)
        Date accessed: May 11, 2022
        • Hill K.
        • English C.
        • Campbell B.C.V.
        • McDonald S.
        • Pattuwage L.
        • Bates P.
        • et al.
        Feasibility of national living guideline methods: the Australian Stroke Guidelines.
        J Clin Epidemiol. 2022; 142: 184-193
        • Walker R.
        • Palmer S.
        • Tunnicliffe D.
        • Cashmore B.
        • Kostner K.
        • Krishnasamy R.
        • et al.
        Management of cholesterol-lowering therapy for people with chronic kidney disease.
        CARI Guidelines, Sydney, Australia2021
      1. Australian Evidence-Based Clinical Guidelines for Diabetes.
        (Available at)
        • Elliott J.H.
        • Synnot A.
        • Turner T.
        • Simmonds M.
        • Akl E.A.
        • McDonald S.
        • et al.
        Living systematic review: 1. Introduction—the why, what, when, and how.
        J Clin Epidemiol. 2017; 91: 23-30
        • Henderson L.K.
        • Craig J.C.
        • Willis N.S.
        • Tovey D.
        • Webster A.C.
        How to write a Cochrane systematic review.
        Nephrology. 2010; 15: 617-624
        • Higgins J.P.
        • Savović J.
        • Page M.J.
        • Elbers R.G.
        • Sterne J.A.
        Assessing risk of bias in a randomized trial.
        Cochrane Handb Syst Rev Interv. 2019; : 205-228
        • Guyatt G.
        • Oxman A.D.
        • Akl E.A.
        • Kunz R.
        • Vist G.
        • Brozek J.
        • et al.
        GRADE guidelines: 1. Introduction—GRADE evidence profiles and summary of findings tables.
        J Clin Epidemiol. 2011; 64: 383-394
        • Shea B.J.
        • Reeves B.C.
        • Wells G.
        • Thuku M.
        • Hamel C.
        • Moran J.
        • et al.
        AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both.
        BMJ. 2017; 358: j4008
        • Whiting P.
        • Savović J.
        • Higgins J.P.
        • Caldwell D.M.
        • Reeves B.C.
        • Shea B.
        • et al.
        ROBIS: a new tool to assess risk of bias in systematic reviews was developed.
        J Clin Epidemiol. 2016; 69: 225-234
        • Garritty C.
        • Gartlehner G.
        • Nussbaumer-Streit B.
        • King V.J.
        • Hamel C.
        • Kamel C.
        • et al.
        Cochrane Rapid Reviews Methods Group offers evidence-informed guidance to conduct rapid reviews.
        J Clin Epidemiol. 2021; 130: 13-22
        • Palmer S.C.
        • Navaneethan S.D.
        • Craig J.C.
        • Johnson D.W.
        • Perkovic V.
        • Hegbrant J.
        • et al.
        HMG CoA reductase inhibitors (statins) for people with chronic kidney disease not requiring dialysis.
        Cochrane Database Syst Rev. 2014; : CD007784
        • Li D.
        • Wang Z.
        • Wang L.
        • Sohn S.
        • Shen F.
        • Murad M.H.
        • et al.
        A text-mining framework for supporting systematic reviews.
        Am J Inf Manag. 2016 Nov; 1: 1-9
        • Sood M.
        • Sharp S.
        • McFarlane E.
        • Willans R.
        • Hopkins K.
        • Karpusheff J.
        • et al.
        Managing the evidence infodemic: Automation approaches used for developing NICE COVID-19 living guidelines.
        (Available at)
        • Horby P.
        • Lim W.S.
        • Emberson J.
        • Mafham M.
        • Bell J.
        • Linsell L.
        • et al.
        Effect of Dexamethasone in Hospitalized Patients with COVID-19 – Preliminary Report.
        (Available at)
        • Pan H.
        • Peto R.
        • Karim Q.A.
        • Alejandria M.
        • Henao-Restrepo A.M.
        • et al.
        • Consortium WS Trial
        Repurposed antiviral drugs for COVID-19 –interim WHO SOLIDARITY trial results.
        (Available at)
        • Smith R.
        Peer review: a flawed process at the heart of science and journals.
        J R Soc Med. 2006; 99: 178-182
        • Hanson K.E.
        • Caliendo A.M.
        • Arias C.A.
        • Englund J.A.
        • Lee M.J.
        • Loeb M.
        • et al.
        Infectious Diseases Society of America guidelines on the diagnosis of coronavirus disease 2019.
        Clin Infect Dis. 2020; : ciaa760
      2. Covidence - Better systematic review management.
        (Available at)
        Date accessed: April 20, 2022
      3. DistillerSR | Systematic Review Software | Literature Review Software.
        (Available at)
        • Ouzzani M.
        • Hammady H.
        • Fedorowicz Z.
        • Elmagarmid A.
        Rayyan—a web and mobile app for systematic reviews.
        Syst Rev. 2016; 5: 1-10
        • Thomas J.
        • Brunton J.
        EPPI-Reviewer: software for research synthesis.
        (Available at)
      4. GRADEpro.
        (Available at)
        Date accessed: April 20, 2022
      5. MAGICapp.
        (Available at)
        Date accessed: April 20, 2022
        • Li T.
        • Saldanha I.J.
        • Jap J.
        • Smith B.T.
        • Canner J.
        • Hutfless S.M.
        • et al.
        A randomized trial provided new evidence on the accuracy and efficiency of traditional vs. electronically annotated abstraction approaches in systematic reviews.
        J Clin Epidemiol. 2019; 115: 77-89
        • JY E.
        • Saldanha I.J.
        • Canner J.
        • Schmid C.H.
        • Le J.T.
        • Li T.
        Adjudication rather than experience of data abstraction matters more in reducing errors in abstracting data in systematic reviews.
        Res Synth Methods. 2020; 11: 354-362
        • Higgins J.P.
        • Altman D.G.
        • Gøtzsche P.C.
        • Jüni P.
        • Moher D.
        • Oxman A.D.
        • et al.
        The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials.
        BMJ. 2011; 343: d5928
        • Sterne J.A.C.
        • Savović J.
        • Page M.J.
        • Elbers R.G.
        • Blencowe N.S.
        • Boutron I.
        • et al.
        RoB 2: a revised tool for assessing risk of bias in randomised trials.
        BMJ. 2019; 366: l4898
        • Garmendia C.A.
        • Nassar Gorra L.
        • Rodriguez A.L.
        • Trepka M.J.
        • Veledar E.
        • Madhivanan P.
        Evaluation of the inclusion of studies identified by the FDA as having falsified data in the results of meta-analyses: the example of the apixaban trials.
        JAMA Intern Med. 2019; 179: 582-584
      6. Djulbegovic B, Ahmed MM, Hozo I, Koletsi D, Hemkens L, Price A, et al. High quality (certainty) evidence changes less often than low-quality evidence, but the magnitude of effect size does not systematically differ between studies with low versus high-quality evidence. J Eval Clin Pract. ;28(3):353-362.

        • Borm G.F.
        • Donders A.R.T.
        Updating meta-analyses leads to larger type I errors than publication bias.
        J Clin Epidemiol. 2009; 62: 825-830.e10
        • Schmid Christopher C.J.
        Should Cochrane apply error-adjustment methods when conducting repeated meta-analyses?.
        (Available at)
        • Simmonds M.
        • Salanti G.
        • McKenzie J.
        • Elliott J.
        • Agoritsas T.
        • Hilton J.
        • et al.
        Living systematic reviews: 3. Statistical methods for updating meta-analyses.
        J Clin Epidemiol. 2017; 91: 38-46
        • Alonso-Coello P.
        • Schünemann H.J.
        • Moberg J.
        • Brignardello-Petersen R.
        • Akl E.A.
        • Davoli M.
        • et al.
        GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 1: Introduction.
        BMJ. 2016; 353: 166.e1-166.e10
        • Brouwers M.C.
        • Kho M.E.
        • Browman G.P.
        • Burgers J.S.
        • Cluzeau F.
        • Feder G.
        • et al.
        AGREE II: advancing guideline development, reporting and evaluation in health care.
        CMAJ. 2010; 182: E839-E842
        • Tendal B.
        • Vogel J.P.
        • McDonald S.
        • Norris S.
        • Cumpston M.
        • White H.
        • et al.
        Weekly updates of national living evidence-based guidelines: methods for the Australian living guidelines for care of people with COVID-19.
        J Clin Epidemiol. 2021; 131: 11-21
        • El Mikati I.K.
        • Khabsa J.
        • Harb T.
        • Khamis M.
        • Agarwal A.
        • Pardo-Hernandez H.
        • et al.
        A framework for the development of living practice guidelines in health care.
        Ann Intern Med. 2022; 175: 1154-1160