Abbreviations:
AGREE (Appraisal of Guidelines for REsearch & Evaluation), COVID-19 (Coronavirus disease of 2019), COVID-END (COVID-19 Evidence Network to support Decision-making), CPG (Clinical Practice Guidelines), GRADE-ADOLOPMENT’ (Approach to guideline production combines adoption, adaptation, and, as needed, de novo development of recommendations), HTA (Health Technology Assessment), RecMap (COVID19 Recommendations and Gateway to Contextualization), RG (Rapid guidelines)Keywords
1. Introduction
The COVID-19 global pandemic led to a substantial investment into the funding, execution, and publication/dissemination of research at unprecedented speeds and volumes. This, in turn, has created an urgent need to manage and curate this evidence to inform public health, clinical and health-system decision-making. Eager to provide some type of guidance in a scenario of uncertainty, many organizations started developing guidelines to provide recommendations to manage COVID-19 patients. Clinical practice guidelines (CPGs) have been published since the early stages of the pandemic, often in very short time frames, with a scarcity of evidence, with evidence that in other contexts would be considered of questionable quality and using methods that do not meet traditional development and reporting norms.
Some of these early CPGs were labeled as “rapid advice”, “rapid guidelines”, or “interim recommendations” [
1
, 2
, 3
] implying they were developed under a pressuring situation, their methods were expedited, or developers applied methodological shortcuts. However, most of the CPGs developed during 2020, presumably under time constraints, were not labeled as rapid guidelines. Rather, they were labeled or presented as regular guidelines, using the usual terms: “guidelines”, “statements”, or “recommendations.”Since the development of traditional (de novo) evidence-based guidelines usually require long periods of time and significant funding, rapid guidelines are necessary, and considered acceptable, in cases of an emergency scenario or where urgent guidance is required. Rapid Guidelines (RGs) are defined as those developed in short timeframes (ie, 1–3 months) [
[1]
,[2]
,[4]
], although during the COVID-19 pandemic they were often developed in even shorter timeframes. To produce evidence-based guidance in a short time frame, because of the urgency, RGs often are developed with methodological shortcuts. The challenge in these processes is to develop guidance at a high speed without compromising the methodological rigor and validity, and therefore, their trustworthiness.Early in the pandemic, the COVID-Evidence Network to support decision-making (COVID-END) initiative (https://www.mcmasterforum.org/networks/covid-end) was created to facilitate the dissemination of evidence synthesis results to inform decision-making and to reduce duplication of efforts in evidence synthesis and evidence-based guidance. Additionally, a group of international researchers interested in the quality of the guidelines partnered with members of the COVID-END recommending working group to monitor and regularly assess the COVID-19 guidelines quality. A living review of guidelines was designed to assess the CPGs focused on the management of critically ill patients with COVID-19 and its first report is published in this journal's volume [
[5]
]. This commentary discusses the insights from the mentioned assessment, examines the key concepts related to the quality of guidelines developed in emergency situations, the problem of duplication with rapid guidelines during the pandemic, and provides some ideas about solutions and further steps for this type of guidelines.2. Quality of guidelines
2.1 Quality of COVID-19 guidelines
Previous publications have raised concerns about the quality of guidelines related to COVID-19. Two assessments of guidelines developed very early in the pandemic (guidelines published before April 2020) [
[6]
,[7]
] found that their methodological quality was poor in almost all the cases. Stamm et al., found that only 8 out of 188 guidelines could be considered as of high quality [[7]
]. These authors found that most of the guidelines lacked appropriate systematic reviews, failed in performing evidence quality assessment, and their editorial independence was unclear, among other methodological flaws [[6]
]. Although we could have expected that the quality would improve over time, this was not the case. Several authors have shown how the quality of guidelines published later in the pandemic and even more recently is still suboptimal [8
, - Gascon L
- Fournier I
- Chiesa-Estomba C
- Russo G
- Fakhry N
- Lechien JR
- et al.
Systematic review of international guidelines for head and neck oncology management in COVID-19 patients.
Eur Arch Otorhinolaryngol. 2021; 23 (PMID: 33891167; PMCID: PMC8062612): 1-37https://doi.org/10.1007/s00405-021-06823-4
9
, 10
, 11
, - Arieta-Miranda J
- Alcaychahua AS
- Santos GP
- Sevillano MC
- Verástegui RL
- Victorio DB
- et al.
Quality assessment of clinical practice guidelines for the management of paediatric dental emergencies applicable to the COVID-19 pandemic, using the AGREE II instrument. A systematic review.
Heliyon. 2020; 6: e05612
12
].2.2 Assessment of the quality of rapid guidelines
The Appraisal of Guidelines for REsearch & Evaluation II, the AGREE II, is the most widely used tool to appraise the quality of CPGs [
13
, 14
, 15
]. The AGREE II tool was developed to assess “conventional” or de novo guidelines; it has been suggested that in a pandemic scenario its standards may be too strict and impossible to meet [[16]
]. We disagree. The AGREE II is useful for assessing rapid guidelines as it provides a blueprint to highlight the strengths and limitations of a guideline, and provides an estimate of trustworthiness, and a mechanism for public health decision-makers, clinicians and health system leaders to have a better sense of likely impact on patients and populations and successful implementation [[14]
,[15]
]. Using the AGREE II to assess rapid guidelines does not presume those with lower scores (than perhaps what would be possible in non-pandemic times) are not useful – but it can serve to manage expectations and inform guideline updates or revisions. Moreover, it should be noted that an assessment of the quality of rapid guidelines before the pandemic showed that a good AGREE II score is possible even for guidelines developed in a very short time frame [[1]
,[17]
]. Therefore, even under emergent situations, the development of trustworthy recommendations with the highest possible methodological standards is possible and should be a goal of any guideline development or adaptation process, especially in times where the stakes are high.2.3 Methodological problems
The above-mentioned methodological problems are not different from the quality assessment performed for other diseases before the pandemic. Using different thresholds on the AGREE II rigour of development or the overall domain scores as criteria for defining a high-quality guideline, previous reports have highlighted that most of guidelines are of limited to poor quality. Thus, in the context of the pandemic, under a unique scenario where there is extreme pressure for developing guidance, it is not surprising that the quality of COVID-19 guidance ended up being suboptimal.
However, there are additional factors to consider that worsen the situation in a pandemic. First, in the beginning of the pandemic there was a very large degree of uncertainty about the disease, and thus, in this situation recommendations provided by well-known experts would be easily adopted and implemented by many stakeholders regardless of the quality of the process or evidence that underpinned them. Second, the disease has a high lethality among some population groups. This increased the anxiety of developers and clinicians (among other decision-makers) which may have led them to recommend anything regardless of the lack of evidence of safety and efficacy. In some cases, so-called "compassionate use" became a justification for the use of interventions of unknown safety and efficacy [
[18]
]. Thus, the pandemic became an excuse for implementing non-systematic shortcuts to develop guidelines in a short time with the risk of recommending non effective or harmful interventions.3. Duplication of efforts
Although quality is a significant methodological problem, duplication of efforts should not be overlooked. Duplication has been a significant problem for research and evidence synthesis processes during the pandemic [
[19]
,[20]
]. While guidelines, as it occurs with health technology assessments (HTAs), are unique as recommendations need to be tailored to specific contexts and may not be applicable to all jurisdictions, unnecessary duplication is not desirable, especially if high-quality guidelines on the same topic exist. Duplication of efforts in guidelines results in guidelines organizations, professional societies and governments struggling simultaneously to evaluate and analyze the same evidence [[21]
]. This, in turn, results in wasted money and time, and increased in delays to release of, in most of the cases, similar recommendations. This can be taxing in contexts where the impact of COVID-19 is particularly acute and exaggerated by complex challenges, such as in low- and middle-income countries (LMIC). Even in often nuanced and unique setting in LMICs, where contextualization of the evidence is required and guidelines specific for those contexts is a need, there is some degree of duplication that could be avoided. The evidence synthesis process, required in any guideline's development process (eg, GRADE Evidence Profiles and Evidence-to-Decision [EtD] judgments) should be shared among guidelines organizations and they can benefit from shorter and less expensive development processes.4. Potential solutions
We have witnessed how the international guidelines community was not prepared to deal with a situation like the COVID-19 pandemic. While finding low quality in the COVID-19 CPGs is not surprising, it was avoidable, and we need to learn from some of our mistakes and experiences. Potential solutions and lessons for future emergency situations and scenarios where rapid guidelines are a need can be summarized in: i) developing clear methodological guidance and templates for rapid guidance and systematic reviews, ii) reducing duplication of efforts via encouraging adoption/adaptation when possible, and iii) enhancing CPGs registration and collaboration; iv) enhancing the coordination with evidence synthesis teams; and v) developing and maintaining appropriate evidence synthesis repositories; and vi) strengthening the CPGs editorial processes. Some of these issues that have been discussed at length by the CPGs’ community for years with insufficient progress made toward a solution; the COVID-19 pandemic amplified the consequences of this lack of progress, and we should take the opportunity to improve and prepare for future emergency scenarios.
4.1 Clear methodological guidance for rapid guidelines development
Methods for CPGs’ development have received substantial input in the last decades. Methods and tools such as Guidelines 2.0 [
[22]
], GRADE [[23]
], and AGREE [[14]
], have boosted this development. However, before the pandemic started there was not enough methodological guidance on how to develop evidence-based guidance in a rapid way maintaining the rigor and trustworthiness. Before the 2020, there was scarcity of resources on the methodological elements for developing rapid and trustworthy guidelines and provided some guidance for their development [1
, 2
, 3
]. With the pandemic some guidance has been published. For instance, recent papers have provided insights and guidance on how to develop trustworthy recommendations using GRADE as part of an urgent response [[17]
,[24]
] or guidance on methods for conducting rapid systematic reviews [[25]
]. We agree that, applying high methodological standards in the development of evidence-based guidance is even more important in times of crisis [[26]
] with the overall aim of balancing rigor and speed. The availability of this guidance should discourage the development of low-quality guidelines with the rationale that high methodological standards cannot be achieved in emergency situations. The need for explicit and transparent descriptions of the methods used, and steps skipped, or short-cuts taken in response to an urgent manner, is critical to ensuring users understand the strengths and limitations of the recommendations they plan to implement.4.2 Encouraging adoption and adaptation
Methodological approaches for adopting or adapting CPGs, with the aim of developing recommendations in a more efficient way have been developed in the last decade. Approaches as ADAPTE [
[27]
], CAN-IMPLEMENT [[28]
], GRADE-ADOLOPMENT [[29]
], among others [[30]
] can be applied to reduce duplication and efficiently develop recommendations. During pandemic, for instance, identifying high-quality guidance for specific questions could facilitate the work for organizations that are planning to develop a new CPG. Adapting those CPGs can provide high-quality, trustworthy, and contextualized recommendations in a short time frame. Moreover, identifying high-quality CPGs’ recommendations from trustworthy organizations which are also considered implementable and feasible in specific context, can lead decision-makers to adopt or endorse some or all the recommendations without major changes. The benefit of this approach is evident, as users can have trustworthy recommendations almost immediately.Toward reducing duplication and research waste, and supporting the evidence-demand, COVID-END has produced useful resources for evidence synthesis and CPG/HTAs, including highlighting various tools, technologies, and CPG adaptation examples (https://www.mcmasterforum.org/networks/covid-end/resources-for-researchers/supports-for-guidance-developers). This resource facilitates accessing to useful information and tools for supporting guidelines’ developers, adapters, and users that will enhance methodological decisions, and may facilitate collaboration.
Moreover, early this year the COVID19 Recommendations and Gateway to Contextualization, or eCOVID-19 RecMap (https://covid19.recmap.org/) initiative was recently launched [
[31]
]. It provides recommendations from CPGs that have been assessed with the AGREE II tool, and its content is routinely updated. It facilitates the identification of the best recommendations for specific questions to be adopted and used and also provides a platform that facilitates the adaptation and contextualization through the GRADE-ADOLOPMENT framework [- Lotfi T
- Stevens A
- Akl EA
- Falavigna M
- Kredo T
- Mathew JL
- et al.
Getting trustworthy guidelines into the hands of decision-makers and supporting their consideration of contextual factors for implementation globally: recommendation mapping of COVID-19 guidelines.
J Clin Epidemiol. 2021; 135 (Epub 2021 Apr 6. PMID: 33836255; PMCID: PMC8022513): 182-186https://doi.org/10.1016/j.jclinepi.2021.03.034
[29]
]. This initiative has the potential to facilitate adoption, adaptation and developing contextualized recommendations, and therefore, to reduce duplication and enhance the quality of future CPGs.4.3 Encouraging registration and collaboration
The high number of guidelines produced in a short timeframe during 2020 and 2021 may reflect a low degree of collaboration among CPGs organizations. Collaboration among CPG developers from different organizations, with varied scopes and structures and different countries may be a challenge, much more in the context of an emergency. However, collaboration may allow organizations to identify areas of common interest, distribute work (eg, systematic reviews, questions, among others), exchange evidence and information, and in some cases, to develop joint guidance [
[32]
]. This can reduce duplication of efforts, decrease development time, and make a faster and more efficient process. Registering guidelines, for instance, is an initiative that may facilitate the identification of groups working on specific topics or questions for guidelines and can be the start of a collaboration between organizations to reduce duplication [- Sultan S
- Siedler MR
- Morgan RL
- Ogunremi T
- Dahm P
- Fatheree LA
- et al.
An international needs assessment survey of guideline developers demonstrates variability in resources and challenges to collaboration between organizations.
J Gen Intern Med. 2021; (Epub ahead of print. PMID: 34545466)https://doi.org/10.1007/s11606-021-07112-w
[33]
,- Chen Y
- Guyatt GH
- Munn Z
- Florez ID
- Marušić A
- Norris SL
- et al.
Clinical practice guidelines registry: toward reducing duplication, improving collaboration, and increasing transparency.
Ann Intern Med. 2021; 174 (Epub 2021 Mar 16. PMID: 33721516): 705-707https://doi.org/10.7326/M20-7884
[34]
], and promote guideline adoption or adaptation. Existing guidelines registries include the International Practice Guideline Registry Platform (www.guidelines-registry.org/), the Australian Clinical Practice Guidelines portal (www.clinicalguidelines.gov.au/register) and the Guidelines-International-Network portal (https://guidelines.ebmportal.com/). Contacting groups working on similar guidelines identified in these registries may lead to sharing outputs or summaries, such as summary of findings tables, risk of bias assessments, or GRADE evidence-to Decision frameworks (EtD) among developers, which will make more efficient development processes.4.4 Strengthening guidelines editorial processes
Guidelines are published in scientific journals or in the official website of developing and endorsing organizations. Regardless of the publication strengthening the editorial and peer review process may help in the process of increasing the quality of future rapid guidelines. For instance, the Enhancing the QUAlity and Transparency Of health Research (EQUATOR) network [
[35]
] recommends the use of AGREE reporting and RIGHT statement checklists to support reporting [EQUATOR Network. https://www.equator-network.org/about-us/how-to-support-equator/2021 [Available from: https://www.equator-network.org/about-us/how-to-support-equator/. Accessed on dec 2nd 2021.
[36]
,[37]
]. These checklists may improve the guidelines’ reporting and enhance their methodology and trustworthiness. Additionally, as described above, encouraging previous registration by journals’ editors, as an expectation of guidelines seeking publication, would likely impact quality and transparency.Moreover, another way to improve guidelines is through a rigorous external review process including both a content and a methodological review. This process may provide an opportunity to identify evidence that has not been previously included, and increases the accountability among CPGs’ panels [
[38]
]. Encouraging external reviewers to use established methodologies to perform the draft CPG's review (eg, using the AGREE II tool) might benefit the final CPG manuscript. We have no data to evaluate the peer review process of rapid guidelines developed during the pandemic, but considering the low quality that has been described, very likely a good number of guidelines had a review of their content and had scarce or no review of their methodologies during the review process.4.5 Coordination with evidence synthesis teams
Evidence synthesis processes have faced similar challenges as those described for CPGs. Low quality and duplication of efforts have also been reported during the pandemic. Nonetheless, the pandemic has also brought some benefits, such as, for instance, advances in the living systematic reviews methods. We witnessed international collaboration in production of living reviews, for example, those focused on vaccine effectiveness [
[39]
] or those that are linked to living guidelines [[40]
], such is the case of the living review and network meta-analysis on pharmacological treatments [[41]
]. Facilitating coordination between evidence synthesis teams and CPG developers, streamlines and makes the process more efficient, and may reduce duplication of efforts.4.6 Developing and maintaining appropriate evidence synthesis repositories
Pandemic also saw the rise of international efforts and cooperation to create and main repositories of relevant evidence resources. For example, the COVID-END initiative (https://www.mcmasterforum.org/networks/covid-end) has developed a repository of ‘best evidence syntheses’, many of which are regularly updated. Guidelines’ developers in different contexts may benefit from these repositories to inform their guidelines. As a result, CPGs process can be expedited, and recommendations can benefit from this. Further research about how CPGs’ developers used these repositories is warranted.
5. Conclusion
We hereby recommend promoting more inclusiveness and collaboration in CPG projects among decision-makers and methodologists using formal evidence-based methodologies for de novo, or adaptation of rapid guidelines, especially for high priority urgent and emergent health topics like COVID-19. Agreeing on the best methodology for rapid guidelines development, encouraging adaptation when possible, registering the CPG project and strengthening the editorial process, enhancing coordination between evidence synthesis and CPG development or adaptation teams, and maintaining and using evidence synthesis repositories are some actions that may facilitate collaboration and reduce duplication to developing CPGs in a more efficient way.
Credit author statement
IDF: Conceptualization, Writing - Original Draft, Writing - Review & Editing, Supervision.
YSA: Conceptualization, Validation, Writing - Review & Editing.
MM: Validation, Writing - Review & Editing.
JNL: Validation, Writing - Review & Editing.
MB: Validation, Writing - Review & Editing.
Financial support
This work did not have any external financial support.
Disclosures
IDF, MB and JNL are part of the AGREE collaboration.
IDF, MB and JNL are co-investigators and Principal investigator, respectively, of the COVID-END Canada initiative which is funded by the Canadian Institute of Health Research.
References
- Development of rapid guidelines: 1. Systematic survey of current practices and methods.Health Res Policy Syst. 2018; 16: 61
- Development of rapid guidelines: 3. GIN-McMaster Guideline Development Checklist extension for rapid recommendations.Health Res Policy Syst. 2018; 16: 63
- Development of rapid guidelines: 2. A qualitative study with WHO guideline developers.Health Res Policy Syst. 2018; 16: 62
- Using GRADE to respond to health questions with different levels of urgency.Environ Int. 2016; 92-93: 585-589
- International alliance and AGREE-ment of 71 clinical practice guidelines on the management of critical care patients with COVID-19: a living systematic review.J Clin Epidemiol. 2021;
- Scope, quality, and inclusivity of clinical guidelines produced early in the covid-19 pandemic: rapid review.BMJ. 2020; 369: m1936
- The methodological quality is insufficient in clinical practice guidelines in the context of COVID-19: systematic review.J Clin Epidemiol. 2021; 135: 125-135
- Systematic review of international guidelines for head and neck oncology management in COVID-19 patients.Eur Arch Otorhinolaryngol. 2021; 23 (PMID: 33891167; PMCID: PMC8062612): 1-37https://doi.org/10.1007/s00405-021-06823-4
- Consistency of recommendations and methodological quality of guidelines for the diagnosis and treatment of COVID-19.J Evid Based Med. 2021; 14: 40-55
- Quality and consistency of clinical practice guidelines for treating children with COVID-19.Ann Transl Med. 2021; 9: 633
- Quality assessment of clinical practice guidelines for the management of paediatric dental emergencies applicable to the COVID-19 pandemic, using the AGREE II instrument. A systematic review.Heliyon. 2020; 6: e05612
- Review of guidelines and recommendations from 17 countries highlights the challenges that clinicians face caring for neonates born to mothers with COVID-19.Acta Paediatrica. 2020; 109: 2192-2207
- Guideline appraisal with AGREE II: online survey of the potential influence of AGREE II items on overall assessment of guideline quality and recommendation for use.BMC Health Serv Res. 2018; 18: 143
- AGREE II: advancing guideline development, reporting and evaluation in health care.Can Med Assoc J. 2010; 182: E839-EE42
- What to do with all the AGREEs? The AGREE portfolio of tools to support the guideline enterprise.J Clin Epidemiol. 2020; 125: 191-197
- EMERGE': construction of a simple quality appraisal tool for rapid review of laparoscopic surgery guidelines during COVID-19 pandemic.Br J Surg. 2020; 107 (e518-e9)
- Developing trustworthy recommendations as part of an urgent response (1–2 weeks): a GRADE concept paper.J Clin Epidemiol. 2021; 129: 1-11
- Treating COVID-19—off-label drug use, compassionate use, and randomized clinical trials during pandemics.JAMA. 2020; 323: 1897-1898
- Waste in COVID-19 research.BMJ. 2020; 369: m1847
- Assessment of duplicate evidence in systematic reviews of imaging findings of children with COVID-19.JAMA Network Open. 2021; 4 (e2032690-e)
- Future of evidence ecosystem series: evidence ecosystems and learning health systems: why bother?.J Clin Epidemiol. 2020; 123: 166-170
- Guidelines 2.0: systematic development of a comprehensive checklist for a successful guideline enterprise.CMAJ. 2014; 186: E123-EE42
- GRADE guidelines: a new series of articles in the.J Clin Epidemiol. 2011; 64: 380-382
- Using GRADE in situations of emergencies and urgencies: certainty in evidence and recommendations matters during the COVID-19 pandemic, now more than ever and no matter what.J Clin Epidemiol. 2020; 127: 202-207
- Cochrane Rapid Reviews Methods Group offers evidence-informed guidance to conduct rapid reviews.J Clin Epidemiol. 2021; 130: 13-22
- Evidence-based medicine in times of crisis.J Clin Epidemiol. 2020; 126: 164-166
- Guideline adaptation: an approach to enhance efficiency in guideline development and improve utilisation.BMJ Qual Saf. 2011; 20: 228-236
- Guideline adaptation and implementation planning: a prospective observational study.Implement Sci. 2013; 8: 49
- GRADE Evidence to Decision (EtD) frameworks for adoption, adaptation, and de novo development of trustworthy recommendations: GRADE-ADOLOPMENT.J Clin Epidemiol. 2017; 81: 101-110
- A methodological survey identified eight proposed frameworks for the adaptation of health related guidelines.J Clin Epidemiol. 2017; 86: 3-10
- Getting trustworthy guidelines into the hands of decision-makers and supporting their consideration of contextual factors for implementation globally: recommendation mapping of COVID-19 guidelines.J Clin Epidemiol. 2021; 135 (Epub 2021 Apr 6. PMID: 33836255; PMCID: PMC8022513): 182-186https://doi.org/10.1016/j.jclinepi.2021.03.034
- An international needs assessment survey of guideline developers demonstrates variability in resources and challenges to collaboration between organizations.J Gen Intern Med. 2021; (Epub ahead of print. PMID: 34545466)https://doi.org/10.1007/s11606-021-07112-w
- Clinical practice guidelines registry: toward reducing duplication, improving collaboration, and increasing transparency.Ann Intern Med. 2021; 174 (Epub 2021 Mar 16. PMID: 33721516): 705-707https://doi.org/10.7326/M20-7884
- Clinical practice guidelines registry.Ann Intern Med. 2021; 174: 1346-1347
EQUATOR Network. https://www.equator-network.org/about-us/how-to-support-equator/2021 [Available from: https://www.equator-network.org/about-us/how-to-support-equator/. Accessed on dec 2nd 2021.
- A reporting tool for practice guidelines in health care: the RIGHT statement.Ann Intern Med. 2016; 166: 128-132
- The AGREE Reporting Checklist: a tool to improve reporting of clinical practice guidelines.BMJ. 2016; 352: i1152
- Clinical practice guidelines: tools to support high quality patient care.Semin Nucl Med. 2019; 49: 145-152
- Vaccines to prevent COVID-19: a protocol for a living systematic review with network meta-analysis including individual patient data (The LIVING VACCINE Project).Syst Rev. 2020; 9: 262
- A living WHO guideline on drugs for COVID-19.BMJ. 2020; 370: m3379
- Drug treatments for covid-19: living systematic review and network meta-analysis.BMJ. 2020; 370: m2980
Article info
Publication history
Published online: November 12, 2021
Accepted:
November 6,
2021
Received:
October 25,
2021
Footnotes
Declaration of interests: The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Non-financial Conflicts of interest: IF, MB and JL are part of the AGREE collaboration; IF, MB and JL are co-investigators and Principal investigator, respectively, of the COVID-END Canada initiative which is funded by the Canadian Institute of Health Research.
Identification
Copyright
© 2021 Elsevier Inc. All rights reserved.
