If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
School of Public Health and Preventive Medicine, Monash University, 553 Street Kilda Road, Melbourne, VIC 3004, AustraliaDiscipline of Acute Care Medicine, University of Adelaide, Adelaide, SA 5000, Australia
Corresponding Author: School of Public Health and Preventive Medicine, Monash University, 533 Street Kilda Road, Melbourne, VIC 3004, Australia. Tel.: +61-3-9903-8145; fax: +61-3-9903-0556.
Living methods are an effective way of updating evidence-based guideline recommendations.
•
Living guidelines can be updated rapidly and frequently.
•
Use of preprint data can accelerate dissemination of time-critical research findings.
•
Living guidelines provide access to reliable up-to-date evidence-based guidance.
Abstract
Objectives
To investigate how quickly evidence was incorporated into the Australian living guidelines for COVID-19 during the first 12 months of the pandemic.
Study Design and Setting
For each study concerning drug therapies included in the guideline from April 3, 2020 to April 1, 2021, we extracted the publication date of the study, and the guideline version the study was included in. We analyzed two subgroups of studies as follows: those published in high impact factor journals and those with 100 or more participants.
Results
In the first year, we published 37 major versions of the guidelines, incorporating 129 studies that investigated 48 drug therapies informing 115 recommendations. The median time from first publication of a study to incorporation in the guideline was 27 days (interquartile range [IQR], 16 to 44), ranging from 9 to 234 days. For the 53 studies in the highest impact factor journals, the median was 20 days (IQR 15 to 30), and for the 71 studies with 100 or more participants the median was 22 days (IQR 15 to 36).
Conclusion
Developing and sustaining living guidelines where evidence is rapidly incorporated is a resource- and time-intensive undertaking; however, this study demonstrates that it is feasible, even over a long period.
In the first year of living COVID-19 guidelines, we incorporated 129 studies in 37 major versions, which informed 115 recommendations.
•
Median time from first publication of a study to incorporation in the published guideline was 27 days.
•
Larger and higher profile studies were incorporated more rapidly.
What this adds to what is known?
•
Living guidelines are an effective way of updating recommendations and can be done rapidly and frequently (weekly) when needed and for a sustained period.
What is the implication and what should change now?
•
Our experience shows that living guidelines in which evidence is rapidly incorporated are feasible and appropriate when there is the need, such as with COVID-19.
1. Introduction
Since COVID-19 was first identified, the World Health Organization (WHO) has reported over 500 million cases and at least 6 million deaths globally [
]. The novel nature of COVID-19 meant that little was known initially about the epidemiology or treatment of the disease. High infection rates and severe illness required treatment options to be examined at an unprecedented pace and scale. In response to this global need, a huge volume of research activity was undertaken—in 2020 about 4% of all research produced was related to COVID-19, and by July 2022, more than a quarter of a million articles had been published in PubMed [
]. Rapid translation of trustworthy research was paramount during the pandemic, and guidelines which could keep up with both the rate of research output and rapid progression of COVID-19 were needed.
Although evidence-based guidelines are an established tool for translating new research into improved practice, the typical 3–5 yearly updating interval of traditional guideline models is inappropriate for scenarios where there is clinical uncertainty, and the evidence base is evolving rapidly [
]. Building on the concept of living systematic reviews, the Australian Living Evidence Consortium has been developing living, continually updated guidelines in stroke, diabetes, arthritis, and kidney disease [
With the arrival of the pandemic, it became clear that a rapid, continually updated living guideline would be ideal for COVID-19. The Australian national COVID-19 clinical evidence taskforce was formed in March 2020 to provide regularly updated, evidence-based clinical guidelines for the care of people with COVID-19. The guidelines address the full course of illness, from mild to critical, and all patient populations including specific recommendations for children, pregnant and breastfeeding women, and older people. The taskforce guidelines are updated weekly, using GRADE-based methods and comply with National Health and Medical Research Council (NHMRC) guideline development standards [
]. The taskforce consists of 34-member organizations representing a wide range of clinical disciplines.
Within the taskforce, an evidence team appraises and synthesizes evidence and prepares evidence-to-decision tables to underpin development of recommendations. A guidelines leadership group oversees the development of recommendations by multidisciplinary guideline panels and is advised by a consumer panel. The guidelines are published in MAGIC and disseminated through traditional and social media. New questions are continually sought from clinicians, consumers, and policymakers. For prioritized questions, the evidence is actively monitored using horizon scans and targeted searches.
As of July 2022, the guidelines have been updated more than 100 times and include over 170 recommendations. The COVID-19 taskforce guidelines are the most rapidly updated living guidelines we are aware of, with daily searches for evidence and often weekly updates; however, as members of the evidence team for the guidelines, we were interested to understand how quickly this approach enabled us to update recommendations when new evidence was identified. We aimed to investigate how quickly evidence was incorporated into the guidelines during the first 12 months of the pandemic, i.e., from the time new studies were identified to the time a new or revised recommendation was published.
2. Methods
2.1 Developing recommendations
During the first 12 months of the pandemic, the taskforce's process for developing and publishing new recommendations followed a rolling 3-week cycle (Fig. 1). In brief, new studies identified from daily surveillance of PubMed, select preprint servers, and key journals were assessed by the evidence review team who then prepared the requisite information in MAGIC (i.e., draft recommendations, evidence profiles, evidence-to-decision tables, and evidence summaries) for review and deliberation by the Disease Modifying Treatment & Chemoprophylaxis Panel (DMTC) at their weekly meetings. Once ratified by the DMTC Panel, new recommendations were approved by the Guideline Leadership Group and Steering Committee before publication. Further details are available in a previously published paper [
In exceptional cases, when it was expected that a study would lead to a new recommendation, particularly to use a drug or not, evidence could be submitted to the DMTC panel on the day it convened. Theoretically, the time from publication to incorporation in the guideline could be as little as 9 days.
2.2 Data collection
While the taskforce developed recommendations in several areas, including chemoprophylaxis, pregnancy and perinatal care, child and adolescent care, and respiratory support, the focus of this paper is the main section of the guideline dealing with drug therapies for people with COVID-19. We collected data on the first 12 months of the taskforce (April 2020–April 2021) since this period coincided with the initial phase of the guideline during, which we were committed to developing recommendations for all drug therapies (excluding complementary medicine) as soon as evidence from randomized trials was available.
Using an internally compiled list of all randomized trials on COVID-19 maintained as part of the guideline's evidence surveillance process, we extracted the following information on each study: first author or study name; date first published online or posted to preprint server; name of journal or preprint server; drug therapy; and sample size. We then examined each of the first 37 versions of the guidelines (available in MAGICapp, published between April 3, 2020 and April 1, 2021) and recorded the version number and publication date that each respective study was fully incorporated. We treated trials that reported multiple therapies in a single publication (e.g., WHO Solidarity trial) as separate trials, since they contributed to different recommendations and required independent consideration by the Panel and subsequent sign-off by the steering committee.
2.3 Data analysis
We analyzed the time from study publication to incorporation in the published guideline using medians and interquartile ranges (IQR). Analyses and figures were produced using Stata SE, version 17 (StataCorp).
In addition to calculating an overall median time from publication to incorporation for all studies, we looked at two subgroups: studies with a sample size of greater than or equal to 100 participants and studies published in the top 10 journals by impact factor in the categories of critical care medicine; medicine, general & internal; and respiratory system (Appendix A). These categories were deemed to be the most relevant to COVID-19. Studies initially published as preprints but subsequently published in one of the top 10 impact factor journals were included as part of this subgroup.
3. Results
From April 3, 2020 to April 1, 2021, we incorporated 129 studies into the taskforce guidelines, spanning 37 major versions and informing 115 recommendations. The included studies investigated 48 drug therapies and were published in 110 papers, with some publications reporting multiple treatment arms in the same study.
The median time from first publication of the study to incorporation in the published guideline was 27 days (IQR 16 to 44), ranging from 9 days [
The median time to incorporation was 20 days (IQR 15 to 30) for the 53 (41%) studies published in the highest impact factor journals; this compared with 32 days (IQR 21 to 57) for the 76 (59%) studies published in other journals (Fig. 3). Similarly, studies with more than 100 participants (n = 71) had a quicker incorporation time (median 22 days; IQR 15 to 36) compared with those with fewer than 100 participants (median 35 days; IQR 20 to 63; n = 58) (Fig. 3).
Fig. 3Subgroup analysis of time from publication of treatment studies to incorporation in the first year of guideline by journal type and study size.
Initial publication type of studies included 62 journal articles, 61 preprints, and 6 others (e.g., results posted to ClinicalTrials.gov or first reported in a prospective meta-analysis). For studies published in the top 10 journals by impact factor (either as initial or final publication), the number of studies per journal was as follows: New England Journal of Medicine (23), JAMA (10) The Lancet (6), Lancet Respiratory Medicine (5), BMJ (3), European Respiratory Journal (2), JAMA Internal Medicine (2), Annals of Internal Medicine (1), and Annals of Intensive Care (1) (Appendix A).
Of the 10 studies that took longer than 100 days to incorporate in the guideline, three were micronutrient interventions (vitamin C and zinc), which were not initially, within scope of the guideline and thus were added only when the inclusion criteria changed. Three more were only identified by the evidence surveillance many weeks after the study was first published online—results were first published in ClinicalTrials.gov; study was published in a non-PubMed journal; and study appeared in PubMed 10 weeks after publication. The remaining four were initially posted as preprints, and although they were identified at the time of posting, were deemed low priority compared with other studies (three of the four had fewer than 50 participants).
4. Discussion
4.1 Summary of main findings
In this research, we sought to understand how quickly new evidence was being incorporated into our living COVID-19 guidelines, by determining the duration between the publication of new research, its appraisal and synthesis with existing research, and its subsequent inclusion in the updated guideline. During the first year of the pandemic, we added 129 studies, with a median time to incorporation of 27 days (IQR 16 to 44). Studies published in the highest impact factor journals were more rapidly incorporated compared with those published in other journals (20 vs. 32 days). Studies with 100 or more participants were also more rapidly incorporated than those with fewer than 100 participants (22 vs. 35 days). In both cases, this quicker incorporation reflects the more substantial impact these studies were likely to have on our recommendations.
A notable feature of the pandemic has been the prominence of preprints as a means to accelerate the dissemination of time-critical research findings [
]. This is borne out in our guidelines, with almost half the studies included in the first year of the guidelines being published initially as preprints. From the outset, when the first trials of COVID-19 began to appear as preprints, the taskforce developed a policy which allowed for the inclusion of preprints, with a proviso added to the evidence summary in MAGIC indicating that the study had yet to be peer-reviewed. Following publication of the preprint, data are checked against the final publication and the evidence profiles updated accordingly.
The longer time taken to incorporate some studies either reflected changes to the inclusion criteria for the guidelines (e.g., around micronutrient interventions), delays in identifying the studies through our evidence surveillance system, or were deemed low priority given the existence of other studies to incorporate.
4.2 Living guidelines
The concept of living guidelines is relatively new, methods are still in development, and the limitations and applications of the approach are yet to be fully understood [
Australian living evidence Consortium methods and processes working group and collaborators. Methods for living guidelines: early guidance based on practical experience. Paper 1: introduction.
ALEC methods and processes working group and collaborators. Methods for living guidelines: early guidance based on practical experience. Paper 3: selecting and prioritising questions for living guidelines.
Methods for living guidelines: early guidance based on practical experience. Article 5: decisions on methods for evidence synthesis and recommendation development for living guidelines: early guidance based on practical experience. Article 5: decisions on methods for evidence synthesis and recommendation development for living guidelines.
Maintaining the taskforce guidelines in “living” mode has proven to be appropriate and important given the impact these guidelines continue to have on decision-making, the frequency with which new or updated recommendations were developed in response to new evidence, and the continuing flow of evidence throughout the year [
Several factors have enabled this rapid evidence synthesis process. The taskforce received timely funding to begin its operations and benefited from the availability of experienced, skilled staff seconded from other living guideline, and evidence synthesis projects within Cochrane Australia and Monash University [
]. Equally important have been the taskforce's communications team and the collaboration of the 34 health professional organizations that formed the taskforce, which has ensured the rapid dissemination of updated guidance to those making decisions about patient care using a range of traditional and social media channels.
The evidence synthesis undertaken during the first 12 months of the taskforce demonstrates that it is feasible for large volumes of research to be rapidly incorporated into living guidelines to ensure clinicians always have access to reliable, up-to-date, evidence-based guidance. This however was an intensive undertaking, which led to a new phase of the guidelines focusing on prioritization to optimize resources and relevance.
4.3 Sustain phase
From July 2021, the taskforce entered a sustainability phase, the objective of which was to direct resources toward higher priority topics rather than develop new (or update existing) recommendations that would have little or no impact on the treatment or care of people with COVID-19. In this phase, evidence work is guided by levels of priority assigned to existing guideline recommendations (in consultation with the expert panels), with high-priority recommendations updated as new evidence becomes available, moderate-priority recommendations for which we incorporate evidence on the advice of our expert panels, and low-priority recommendations that we are unlikely to update.
This priority setting approach, although not implemented during the period covered by this study, can be seen to be informally happening already, with studies of 100 or more participants or those published in high-impact factor journals being more rapidly incorporated in the guidelines.
4.4 Limitations
Given the rapid and responsive nature of the taskforce, this study was not planned a priori, raising potential for bias in the design. However, detailed records were kept as a result of our standard of operating processes and provided a reliable data source for data extraction.
5. Conclusions
Developing and sustaining living guidelines in which evidence is rapidly incorporated is a resource- and time-intensive undertaking, however this study demonstrates that it is feasible, even over a long period. Living approaches to guideline development lead to the production of highly relevant guidelines, ensuring that clinicians have access to the latest available evidence to guide their decisions, enabling the best possible outcomes for patients.
Australian living evidence Consortium methods and processes working group and collaborators. Methods for living guidelines: early guidance based on practical experience. Paper 1: introduction.
ALEC methods and processes working group and collaborators. Methods for living guidelines: early guidance based on practical experience. Paper 3: selecting and prioritising questions for living guidelines.
Methods for living guidelines: early guidance based on practical experience. Article 5: decisions on methods for evidence synthesis and recommendation development for living guidelines: early guidance based on practical experience. Article 5: decisions on methods for evidence synthesis and recommendation development for living guidelines.
Declaration of Interest: Weekly updating of guideline recommendations is feasible: the Australian National COVID-19 Clinical Evidence Taskforce.
Funding: The taskforce is funded by the Australian Government Department of Health. Foundation funding was provided by the Victorian Government Department of Health and Human Services, The Ian Potter Foundation, Walter Cottman Endowment Fund, managed by Equity Trustees, Lord Mayors' Charitable Foundation. TT is director of the National Clinical Evidence Taskforce, and JH, SM, AP, and HW are members of the taskforce evidence team. All are either fully or partially funded by the taskforce.
Declaration of competing interests: The authors declare that they have no competing interests.
CRediT author statement: JH, SM, AP, HW and TT developed the study. JH, SM, AP and HW undertook data checking and cleaning. ST undertook the statistical analysis. JH wrote the first draft of the paper, which was revised by SM and TT. All authors revised subsequent drafts. All authors approved the final version.
00040-9&id=gr1.jpg){kind=link}
00040-9&id=gr2.jpg){kind=link}
00040-9&id=gr3.jpg){kind=link}