The methodological quality is insufficient in clinical practice guidelines in the context of COVID-19: systematic review

Open AccessPublished:March 07, 2021DOI:https://doi.org/10.1016/j.jclinepi.2021.03.005

      Highlights

      • Our study explored basic methodological standards in COVID-19 guideline development.
      • Only 4% of the guidelines were classified as the highest methodological quality.
      • Insufficient methodological standards could lead to harmful actions for patients.

      Objectives

      The number of published clinical practice guidelines related to COVID-19 has rapidly increased. This study explored if basic methodological standards of guideline development have been met in the published clinical practice guidelines related to COVID-19.

      Study Design and Setting

      Rapid systematic review from February 1 until April 27, 2020 using MEDLINE [PubMed], CINAHL [Ebsco], Trip and manual search, including all types of healthcare workers providing any kind of healthcare to any patient population in any setting.

      Results

      There were 1342 titles screened and 188 guidelines included. The highest average AGREE II domain score was 89% for scope and purpose, the lowest for rigor of development (25%). Only eight guidelines (4%) were based on a systematic literature search and a structured consensus process by representative experts (classified as the highest methodological quality). The majority (156; 83%) was solely built on an informal expert consensus. A process for regular updates was described in 27 guidelines (14%). Patients were included in the development of only one guideline.

      Conclusion

      Despite clear scope, most publications fell short of basic methodological standards of guideline development. Clinicians should use guidelines that include up-to-date information, were informed by stakeholder involvement, and employed rigorous methodologies.

      Graphical Abstract

      Keywords

      What is new?

         Key findings

      • Despite clear scope, most guidelines for SARS CoV-2 infections and for other care in the context of COVID-19 fell short of basic methodological standards. Only 4% were based on a systematic literature search and a structured consensus process by representative experts (classified as the highest methodological quality). Patients were included in the development of one guideline. A process for regular updates was described in 14%.

         What this adds to what is known?

      • Our study explored if basic methodological standards of guideline development have been met in the published clinical practice guidelines related to COVID-19.

         What is the implication/what should change now?

      • An insufficient consideration of appropriate methodologies in the guideline development process could lead to misleading information, uncertainty among the professionals, and potentially harmful actions for patients. This paper provides an important benchmark for the future assessment of the quality of COVID-19 guidelines.

      1. Introduction

      The novel Coronavirus Disease 2019 (COVID-19) spread rapidly worldwide and the number of cases increased globally at an accelerated rate [
      • Zhu N
      • Zhang D
      • Wang W
      • Li X
      • Yang B
      • Song J.
      • et al.
      A novel coronavirus from patients with pneumonia in China, 2019.
      ,

      World Health Organization. Clinical management of severe acute respiratory infection when novel coronavirus (2019-nCoV) infection is suspected—interim guidance. World Health Organization. Available at: https://www.who.int/publications-detail/clinical-management-of-severe-acute-respiratory-infection-when-novel-coronavirus-(ncov)-infection-is-suspected. 2020. Accessed April 5, 2020.

      . While measures are essential to mitigate the impact of the pandemic specifically, we also need immediate and targeted action to continue effective and safe healthcare generally. Necessary actions to avoid collateral damages to patients range from adapting healthcare to maximizing safety while providing continuation of usual healthcare to rapidly restart routine care in the areas where healthcare has been reduced due to mandated lockdowns. Moreover, healthcare workers in close physical contact with patients could be infected or could be unknowingly carriers of the SARS-CoV-2 virus themselves.
      Clinical practice guidelines and recommendations are needed to support therapeutic decisions and provide an essential knowledge source, especially in such a new and challenging situation. Clinical practice guidelines are statements that include recommendations intended to optimize patient care which are informed by a systematic review of evidence and an assessment of the benefits and harms of alternative care options [

      American Academy of Family Physicians (AAFP). Clinical practice guideline manual. Available at: https://www.aafp.org/patient-care/clinical-recommendations/cpg-manual.html. 2017. Accessed May 22, 2020.

      ,
      Institute of Medicine
      Committee on standards for developing trustworthy clinical practice guidelines.
      .
      The number of published guidelines and recommendations related to COVID-19 has rapidly increased since February 2020. However, insufficient consideration of appropriate methodologies and rigorous strategies in the guideline development process could lead to misleading information, uncertainty among professionals, and potentially harmful actions for patients [
      Institute of Medicine
      Committee on standards for developing trustworthy clinical practice guidelines.
      ,
      • Brouwers MC
      • Kho ME
      • Browman GP
      • et al.
      AGREE II: advancing guideline development, reporting and evaluation in health care.
      . The aim of our study was therefore to systematically review and critically appraise clinical practice guidelines related to SARS-CoV-2 infections and the delivery of healthcare in the current context of COVID-19 from a methodological perspective.

      2. Methods

       2.1. Data sources and searches

      We performed a rapid systematic review informed by the Cochrane Rapid Reviews Interim Guidance from the Cochrane Rapid Reviews Methods Group [
      • Garritty C GG
      • Kamel C
      • King VJ
      • Nussbaumer-Streit B
      • Stevens A
      • Hamel C
      • et al.
      Cochrane rapid reviews interim guidance from the Cochrane rapid reviews methods group.
      ] in collaboration with Cochrane Austria. We will provide yearly updates during the pandemic and thereafter, as appropriate, given the currently increasing evidence and the rising number of published guidelines in the context of COVID-19. An international expert task force including representatives from each of the six WHO regions (Europe, America, Africa, Eastern-Mediterranean, South-East Asia, Western-Pacific) defined research questions, keywords, Medical Subject Headings (MeSH) terms and a search strategy for the rapid systematic review. We searched the MEDical Literature Analysis and Retrieval System OnLINE (MEDLINE) [PubMed], the Cumulative Index to Nursing & Allied Health Literature (CINAHL) [Ebsco] and the Turning Research Into Practice (Trip) database on April 27, 2020 (calendar week 17) for medical guidelines and recommendations in the context of the COVID-19 pandemic published after February 1, 2020. An initial limited search was conducted to verify our selection of search terms and proposed strategy. The results of the initial pilot search were discussed with the task force. The search strategy (Supplement Table A) was adapted accordingly and approved before conducting the literature search in MEDLINE, CINAHL and Trip. To identify grey literature [
      • Peters MD
      • Godfrey CM
      • Khalil H
      • McInerney P
      • Parker D
      • Soares CB.
      Guidance for conducting systematic scoping reviews.
      ], the task force members were asked to indicate any additional guidelines and recommendations not identified in the database queries. In addition, we examined the websites of cross-regional associations and institutions at continent level. The study protocol is publicly available at www.researchgate.net (DOI:10.13140/RG.2.2.21293.51689). All task force members completed conflict of interest forms.

       2.2. Study selection

      Inclusion criteria were (i) guidelines and recommendations related to the COVID-19 situation, (ii) all types of healthcare workers providing any kind of healthcare to any patient population, (iii) any setting (acute care, rehabilitation, long-term care, practice, home visits, etc.). We did not restrict to any specific authors, e.g. from a certain group of health professionals. We excluded documents that only discuss (i) infection control and exposure safety information, (ii) experimental pharmaceutical treatments, (iii) clinical research recommendations, (iv) ethical guidelines, (v) diagnostic procedures or strategies, (vi) results not produced in direct relation to the COVID-19 situation, (vii) textbooks, theoretical exercises, case studies, experience write-ups, and (viii) documents not available in English.
      Results from the searches in MEDLINE and CINAHL were downloaded in Covidence (www.covidence.org) and duplicates removed thereafter. One reviewer (MA) screened titles and abstracts for basic inclusion and exclusion criteria as described above. To ensure accuracy of the selection process, 60 percent of the titles and abstracts were independently screened by other reviewers (EMos, VR, TS). Disagreements were resolved through consensus. Reasons for exclusion are provided in a Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flowchart [
      • Moher D
      • Liberati A
      • Tetzlaff J
      • Altman DG
      • The PG.
      Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement.
      ] in Figure 1 and in more detail in the supplementary material (Supplement Table B). Following a training session, full text screening was carried out by at least two reviewers per document (MA, LL, EMos, JM, MO, VR, TS). For the planned updates (living review), we will use automated text mining (tm) based on the tm package in R (www.r-project.org) to support abstract screening and study selection.

       2.3. Data extraction and quality assessment

      Data extracted included study characteristics (Supplement Table C), a critical appraisal according to The Appraisal of Guidelines for Research & Evaluation Instrument (AGREE II) [
      • Brouwers MC
      • Kho ME
      • Browman GP
      • et al.
      AGREE II: advancing guideline development, reporting and evaluation in health care.
      ] and classification of the guidelines according to the Association of the Scientific Medical Societies (AWMF) Guidance Manual and Rules for Guideline Development [
      • Muche-Borowski C
      • Selbmann H
      • Nothacker M
      • Müller W
      • Kopp I.
      AWMF guidance manual and rules for guideline development.
      ]. The AGREE II is the gold-standard protocol for clinical guideline assessment, development, and reporting [
      • Brouwers MC
      • Kho ME
      • Browman GP
      • et al.
      AGREE II: advancing guideline development, reporting and evaluation in health care.
      ]. It comprises 23 items assessing six domains including scope and purpose, stakeholder involvement, rigor of development, clarity of presentation, applicability, and editorial independence. The AWMF Guidance Manual and Rules for Guideline Development classifies guidelines as being based either on (i) a systematic literature review including a subsequent synthesis of the evidence and a structured consensus process completed by a representative committee (S3), (ii) a systematic literature review and synthesis of the evidence only (S2e), (iii) a structured consensus process completed by a representative committee only (S2k) or (iv) an informal consensus process by a group of experts (S1) [
      • Muche-Borowski C
      • Selbmann H
      • Nothacker M
      • Müller W
      • Kopp I.
      AWMF guidance manual and rules for guideline development.
      ].
      Reviewers pilot-tested the data extraction form on 19 (10%) of the records (Supplement Table C) then independently extracted the data. AGREE II scores were ranked on a Likert scale from 1=strongly disagree to 7=strongly agree. Mean scores between the two reviewers were calculated for each AGREE II item. AGREE II items were summarized into the six domain scores [
      • Brouwers MC
      • Kho ME
      • Browman GP
      • et al.
      AGREE II: advancing guideline development, reporting and evaluation in health care.
      ]. An overall rating and a qualitative recommendation were assigned and disagreements were resolved through consensus. AGREE domain-level scores were calculated for each publication and mean domain-level scores were formed for the entire dataset and the S3 guidelines separately.

       2.4. Data synthesis and analysis

      Descriptive statistics were calculated for relevant extraction fields for the entire dataset using Microsoft Excel, including geographic affiliation of authors, publication status, type of recommendation, target population, focus of recommendation, disease/condition, and setting. Other fields are presented narratively, where appropriate. The world map was created in the Free and Open Source Geographic Information System QGIS (https://qgis.org/en/site/). We used the PRISMA checklist for standardized reporting in systematic reviews and meta-analyses as a reference for the reporting of our results (Supplement Table F).

      3. Results

      In total, we identified 819 records in MEDLINE, 89 in CINAHL and 434 in Trip through our systematic search, and four additional records through other sources (Fig. 1). We retained 397 documents for abstract and full text screening. Of these, 188 guidelines and recommendations met the inclusion criteria and were selected for data extraction and critical appraisal.

       3.1. Target areas, professionals, and country representation of authors

      The most frequent medical areas addressed in the guidelines were acute COVID-19 care (n = 46; 24%), surgery (n = 41; 22%), oncology (n = 20; 11%), radiology (n = 14; 7%) and cardiology (n = 10; 5%). A third (n = 57; 30%) of the guidelines focused on miscellaneous other medical areas (Supplement Table D). All except four guidelines (184; 98%) targeted physicians. The four guidelines not targeting physicians referred to care delivered by non-physician health professionals, namely pharmacy practice management for hematopoietic cell transplantation and cellular therapy [
      • Mahmoudjafari Z
      • Alexander M
      • Roddy J
      • Shaw R
      • Shigle TL
      • Timlin C
      • et al.
      American Society for Transplantation and Cellular Therapy Pharmacy Special Interest Group Position Statement on pharmacy practice management and clinical management for COVID-19 in hematopoietic cell transplant and cellular therapy patients in the United States.
      ], physiotherapy management for COVID-19 in the acute hospital settings [
      • Thomas P
      • Baldwin C
      • Bissett B
      • Boden I
      • Gosselink R
      • Granger CL
      • et al.
      Physiotherapy management for COVID-19 in the acute hospital setting: clinical practice recommendations.
      ], and two articles on pharmaceutical care [
      • Song Z
      • Hu Y
      • Zheng S
      • Yang L
      • Zhao R.
      Hospital pharmacists’ pharmaceutical care for hospitalized patients with COVID-19: recommendations and guidance from clinical experience.
      ,
      • S-q Zheng
      • L Yang
      • P-x Zhou
      • H-b Li
      • Liu F
      • R-s Zhao
      Recommendations and guidance for providing pharmaceutical care services during COVID-19 pandemic: a China perspective.
      . Overall, 84 guidelines (45%) mentioned the work of non-physician health professionals. This includes naming specific professions and/or grouping all non-physician health professionals under members of a healthcare team. Thirty-two (17%) guidelines also targeted patients and/or the general public. Patients were included as reviewers in the development of only one guideline [
      • Sultan S
      • Lim JK
      • Altayar O
      • Davitkov P
      • Feuerstein JD
      • Siddique SM
      • et al.
      AGA Institute rapid recommendations for gastrointestinal procedures during the COVID-19 pandemic.
      ]. Experts from 54 countries participated in the guidelines development, with Africa being the least represented continent and North America being the most represented continent (Fig. 2 and Supplement Table G). A process for regular updates was described in 27 guidelines (14%).
      Fig 2
      Fig. 2Country representation of the guideline authors. Absolute and relative frequencies per country are shown in .

       3.2. Critical methodological appraisal

      The highest average AGREE II [
      • Brouwers MC
      • Kho ME
      • Browman GP
      • et al.
      AGREE II: advancing guideline development, reporting and evaluation in health care.
      ] domain score was given for scope and purpose (89%), followed by clarity of presentation (70%) and editorial independence (61%). Stakeholder involvement was on average fulfilled by 43% and applicability by 35%. The lowest average AGREE domain score (25%) was given for rigor of development due to the fact that methods of development were insufficiently or not reported. The domain scores of each guideline are shown in Supplement Table E. The mean overall quality rating of the guidelines was 3.8 (standard deviation +/-1.4) of a maximum score 7.
      Eight of all 188 guidelines (4%) [
      • Thomas P
      • Baldwin C
      • Bissett B
      • Boden I
      • Gosselink R
      • Granger CL
      • et al.
      Physiotherapy management for COVID-19 in the acute hospital setting: clinical practice recommendations.
      ,
      • Sultan S
      • Lim JK
      • Altayar O
      • Davitkov P
      • Feuerstein JD
      • Siddique SM
      • et al.
      AGA Institute rapid recommendations for gastrointestinal procedures during the COVID-19 pandemic.
      • Zhao H-M
      • Xie Y-X
      • Wang C.
      Recommendations for respiratory rehabilitation in adults with COVID-19.
      were classified as having the highest methodological level (systematic literature search and structured consensus process by representative experts; S3) [
      • Muche-Borowski C
      • Selbmann H
      • Nothacker M
      • Müller W
      • Kopp I.
      AWMF guidance manual and rules for guideline development.
      ] (Table 1). Twelve guidelines (6%) were solely based on a systematic literature review and synthesis of the evidence (S2e), while a further 12 (6%) were developed using a structured consensus process completed by a representative committee only (S2k). The majority of the guidelines and recommendations, namely 156 (83%), were developed based on an informal consensus process by a group of experts (S1). Papers which referred to a consensus process or expert opinion without providing any details were also assigned to the group of S1 guidelines. An overview about the number of published guidelines regarding their S-classification is depicted separately for each week in Figure 3. The eight S3 guidelines had an average overall quality rating of 6.7 (standard deviation +/-0.5) of a maximum score 7 and are depicted Table 2.
      Table 1Data extraction for the eight publications classified as the highest methodological level (systematic literature search and structured consensus process by representative experts; S3)
      Authors, reference (ID)Author countriesPublish dateRegular updateHistory shownMethodsPeer-reviewRisk of bias - AGREE II scoreCare focusType focusDisease/condition, if applicableSetting
      Jin, Cai, Cheng, Cheng, Deng, Fan, Fang, Huang, Huang, Huang
      • Jin Y-H
      • Cai L
      • Cheng Z-S
      • Cheng H
      • Deng T
      • Fan Y-P
      • et al.
      A rapid advice guideline for the diagnosis and treatment of 2019 novel coronavirus (2019-nCoV) infected pneumonia (standard version).
      (358)
      China06.02.20 (Up-date published on 04.09.2020
      • Jin Y-H
      • Zhan Q-Y
      • Peng Z-Y
      • Ren X-Q
      • Yin X-T
      • Cai L
      • et al.
      Chemoprophylaxis, diagnosis, treatments, and discharge management of COVID-19: An evidence-based clinical practice guideline (updated version).
      )
      nonoLiterature review and expert consensusyes6COVID-19COVID-19PneumoniaGeneric
      Gralnek, Hassan, Beilenhoff, Antonelli, Ebigbo, Pellisè, Arvanitakis, Bhandari, Bisschops, Van Hooft
      • Gralnek IM
      • Hassan C
      • Beilenhoff U
      • Antonelli G
      • Ebigbo Alanna
      • Pellisé Maria
      • et al.
      ESGE and ESGENA position statement on gastrointestinal endoscopy and the COVID-19 pandemic.
      (302)
      Israel, Italy, Germany, Spain, Belgium, UK, the Netherlands, Poland, Greece, Portugal, France, Denmark, Estonia, Slovenia, Croatia, Switzerland, USA18.03.20 (Up-dates published on 17.04.2020 and on 08.07.2020
      • Gralnek IM
      • Hassan C
      • Beilenhoff U
      • Antonelli G
      • Ebigbo A
      • Pellisé M
      • et al.
      ESGE and ESGENA position statement on gastrointestinal endoscopy and COVID-19: an update on guidance during the post-lockdown phase and selected results from a membership survey.
      )
      yesyesSystemic literature review, followed by a formal consensus process by an international group.yes7Routine careDisease-specificGastrointestinal endoscopyInstitution
      Alhazzani, Møller, Arabi, Loeb, Gong, Fan, Oczkowski, Levy, Derde, Dzierba
      • Alhazzani W
      • Møller MH
      • Arabi YM
      • Loeb M
      • Gong Michelle Ng
      • Fan E
      • et al.
      Surviving sepsis campaign: guidelines on the management of critically ill adults with coronavirus disease 2019 (COVID-19).
      (76)
      Canada, Denmark, Saudi Arabia, USA, Netherlands, China, Italy, UAE, Korea, Australia, UK28.03.20 (Update published on 28.01.2021
      • Alhazzani W
      • Evans L
      • Alshamsi F
      • Møller MH
      • Ostermann M
      • Prescott HC
      • et al.
      Surviving sepsis campaign guidelines on the management of adults with coronavirus disease 2019 (COVID-19) in the ICU: first update.
      )
      nonoSystematic literature review and consensus processyes7COVID-19COVID-19Management of critically ill adults with Coronavirus Disease 2019Acute / ICU
      Thomas, Baldwin, Bissett, Boden, Gosselink, Granger, Hodgson, Jones, Kho, Moses
      • Thomas P
      • Baldwin C
      • Bissett B
      • Boden I
      • Gosselink R
      • Granger CL
      • et al.
      Physiotherapy management for COVID-19 in the acute hospital setting: clinical practice recommendations.
      (696)
      Australia, Belgium, Canada, UK30.03.20yesnoWeb search for evidence and expert consensus processyes7COVID-19COVID-19PhysiotherapyAcute / ICU
      Sultan, Lim, Altayar, Davitkov, Feuerstein, Siddique, Falck-Ytter, El-Serag
      • Sultan S
      • Lim JK
      • Altayar O
      • Davitkov P
      • Feuerstein JD
      • Siddique SM
      • et al.
      AGA Institute rapid recommendations for gastrointestinal procedures during the COVID-19 pandemic.
      (678)
      USA31.03.20
      This paper itself was not updated, but AGA released a formal recommendation on 11.05.20 titled ‘AGA Institute Rapid Review of the Gastrointestinal and Liver Manifestations of COVID-19, Meta-Analysis of International Data, and Recommendations for the Consultative Management of Patients with COVID-19’ with the same purpose [24].
      yesnoRapid review and expert consensusyes7COVID-19Disease-specificGastrointestinal proceduresGeneric
      Motlagh, Yamrali, Azghandi, Azadeh, Vaezi, Ashrafi, Zendehdel, Mirzaei, Basi, Rakhsha
      • Motlagh A
      • Yamrali M
      • Azghandi S
      • Azadeh P
      • Vaezi M
      • Ashrafi F
      • et al.
      COVID19 prevention & care; a cancer specific guideline.
      (518)
      Iran01.04.20yesyesLiterature search and nominal group techniqueyes6Routine careDisease-specificCancerGeneric
      Rubin, Ryerson, Haramati, Sverzellati, Kanne, Raoof, Schluger, Volpi, Yim, Martin
      • Rubin GD
      • Ryerson CJ
      • Haramati LB
      • Sverzellati N
      • Kanne JP
      • Raoof S
      • et al.
      The role of chest imaging in patient management during the COVID-19 pandemic: a multinational consensus statement from the Fleischner Society.
      (613)
      USA, Canada, South Korea, UK, France, Japan, Germany, Netherlands, Italy, China07.04.20 (e-pub ahead of print; official publication in July 2020)nonoA systematic review to inform the discussion of a 15-person expert panel to address 14 questions, corresponding to 11 decision points within 3 scenarios and 3 additional clinical situations.unclear7COVID-19COVID-19Chest imagingGeneric
      Zhao, Xie, Wang
      • Zhao H-M
      • Xie Y-X
      • Wang C.
      Recommendations for respiratory rehabilitation in adults with COVID-19.
      (789)
      China09.04.20 (e-pub ahead of print; official publication in July 2020)yesnoLiterature review and expert consensusyes6.5COVID-19COVID-19Respiratory rehabilitationGeneric
      COVID-19, coronavirus disease 2019; ICU, intensive care unit; UAE, United Arab Emirates; UK, United Kingdom; USA, United States of America.
      Conflict of interest was declared in all these S3 guidelines. Options for comments or external review were not given in any of these. AGREE II scores were ranked on a Likert scale from 1 = strongly disagree to 7 = strongly agree. We included up-dates of the guidelines until 08.02.2021.
      a This paper itself was not updated, but AGA released a formal recommendation on 11.05.20 titled ‘AGA Institute Rapid Review of the Gastrointestinal and Liver Manifestations of COVID-19, Meta-Analysis of International Data, and Recommendations for the Consultative Management of Patients with COVID-19’ with the same purpose
      • Sultan S
      • Altayar O
      • Siddique SM
      • Davitkov P
      • Feuerstein JD
      • Lim JK
      • et al.
      AGA Institute rapid review of the GI and liver manifestations of COVID-19, meta-analysis of international data, and recommendations for the consultative management of patients with COVID-19.
      .
      Fig 3
      Fig. 3Numbers of published guidelines which increased per week separately for each S-level
      [
      • Muche-Borowski C
      • Selbmann H
      • Nothacker M
      • Müller W
      • Kopp I.
      AWMF guidance manual and rules for guideline development.
      ]
      Guidelines were classified according to the Association of the Scientific Medical Societies (AWMF) Guidance Manual and Rules for Guideline Development
      [
      • Muche-Borowski C
      • Selbmann H
      • Nothacker M
      • Müller W
      • Kopp I.
      AWMF guidance manual and rules for guideline development.
      ]
      . S3 is a systematic literature review including a subsequent synthesis of the evidence and a structured consensus process completed by a representative committee. S2e refers to a systematic literature review and synthesis of the evidence only; S2k guidelines are based on a structured consensus process completed by a representative committee only. S2e and S2k are summarized in this graph as S2. S1 in this work includes both, documents based on an informal consensus process by a group of experts as well as expert opinions.
      Table 2Mean AGREE II domain scores for the eight publications classified as the highest methodological level (systematic literature search and structured consensus process by representative experts; S3)
      Authors, reference (ID)Domain 1 score: scope & purposeDomain 2 score: stakeholder involvementDomain 3 score: rigor of developmentDomain 4 score: clarity of presentationDomain 5 score: applicabilityDomain 6 score: editorial independenceRate the overall quality of this guidelineI would recommend this guideline for use
      Jin, Cai, Cheng, Cheng, Deng, Fan, Fang, Huang, Huang, Huang
      • Jin Y-H
      • Cai L
      • Cheng Z-S
      • Cheng H
      • Deng T
      • Fan Y-P
      • et al.
      A rapid advice guideline for the diagnosis and treatment of 2019 novel coronavirus (2019-nCoV) infected pneumonia (standard version).
      (358)
      100%67%77%83%58%100%6Yes
      Gralnek, Hassan, Beilenhoff, Antonelli, Ebigbo, Pellisè, Arvanitakis, Bhandari, Bisschops, Van Hooft
      • Gralnek IM
      • Hassan C
      • Beilenhoff U
      • Antonelli G
      • Ebigbo Alanna
      • Pellisé Maria
      • et al.
      ESGE and ESGENA position statement on gastrointestinal endoscopy and the COVID-19 pandemic.
      (302)
      100%78%94%100%58%100%7Yes
      Alhazzani, Møller, Arabi, Loeb, Gong, Fan, Oczkowski, Levy, Derde, Dzierba
      • Alhazzani W
      • Møller MH
      • Arabi YM
      • Loeb M
      • Gong Michelle Ng
      • Fan E
      • et al.
      Surviving sepsis campaign: guidelines on the management of critically ill adults with coronavirus disease 2019 (COVID-19).
      (76)
      100%47%85%83%67%100%7Yes
      Thomas, Baldwin, Bissett, Boden, Gosselink, Granger, Hodgson, Jones, Kho, Moses
      • Thomas P
      • Baldwin C
      • Bissett B
      • Boden I
      • Gosselink R
      • Granger CL
      • et al.
      Physiotherapy management for COVID-19 in the acute hospital setting: clinical practice recommendations.
      (696)
      100%67%69%100%71%100%7Yes
      Sultan, Lim, Altayar, Davitkov, Feuerstein, Siddique, Falck-Ytter, El-Serag
      • Sultan S
      • Lim JK
      • Altayar O
      • Davitkov P
      • Feuerstein JD
      • Siddique SM
      • et al.
      AGA Institute rapid recommendations for gastrointestinal procedures during the COVID-19 pandemic.
      (678)
      100%89%96%94%63%100%7Yes
      Motlagh, Yamrali, Azghandi, Azadeh, Vaezi, Ashrafi, Zendehdel, Mirzaei, Basi, Rakhsha
      • Motlagh A
      • Yamrali M
      • Azghandi S
      • Azadeh P
      • Vaezi M
      • Ashrafi F
      • et al.
      COVID19 prevention & care; a cancer specific guideline.
      (518)
      97%69%49%67%46%83%6Yes
      Rubin, Ryerson, Haramati, Sverzellati, Kanne, Raoof, Schluger, Volpi, Yim, Martin
      • Rubin GD
      • Ryerson CJ
      • Haramati LB
      • Sverzellati N
      • Kanne JP
      • Raoof S
      • et al.
      The role of chest imaging in patient management during the COVID-19 pandemic: a multinational consensus statement from the Fleischner Society.
      (613)
      100%67%81%100%67%100%7Yes
      Zhao, Xie, Wang
      • Zhao H-M
      • Xie Y-X
      • Wang C.
      Recommendations for respiratory rehabilitation in adults with COVID-19.
      (789)
      100%56%65%67%75%50%6.5Yes
      AGREE II scores were ranked on a Likert scale from 1 = strongly disagree to 7 = strongly agree.

      4. Discussion

      A considerable number of medical guidelines and recommendations related to the COVID-19 pandemic were published from February to April 2020. Despite high AGREE II scores for scope and purpose as well as for clarity of presentation, the majority of the guidelines and recommendations fell short of basic standards. This was mainly due to a lack of appropriate methodologies and rigorous strategies in the guideline development process. The quality and methodological limitations of the guidelines should be placed in the context of the current unprecedented situation and the human resources required to produce high-quality guidelines in a timely manner. While systematically developed statements reflecting the current state of knowledge and supporting health professionals in their work during the COVID-19 pandemic have been urgently needed, basic methodological standards in the guideline development process are nevertheless essential to avoid misleading information and potentially harmful actions for patients and the healthcare system as a whole. One S3 guideline [
      • Jin Y-H
      • Cai L
      • Cheng Z-S
      • Cheng H
      • Deng T
      • Fan Y-P
      • et al.
      A rapid advice guideline for the diagnosis and treatment of 2019 novel coronavirus (2019-nCoV) infected pneumonia (standard version).
      ] with high overall methodological ratings had been published in calendar week six from when we found the first guidelines in our search (Fig. 3), demonstrating that it was possible to produce high-quality work in a short time. Taking more time to develop guidelines therefore may not always lead to a more rigorous methodological basis. Rather, the methodological commitment, expertise of the authors and engagement of appropriate stakeholders might be important for a rigorous development process, independent of the time period after the outbreak when the guideline was published. Many guidelines have been published since the end of April 2020, with better knowledge, experience and more robust literature accompanying the evolution of the pandemic. Further research should monitor the evolving methodological quality of the guidelines and their updates over time. Similar to our study, a review of prediction models for diagnosis and prognosis of COVID-19 concluded that the published models were poorly reported and at high risk of bias [
      • Wynants L
      • Van Calster B
      • Collins GS
      • Riley RD
      • Heinze G
      • Schuit E
      • et al.
      Prediction models for diagnosis and prognosis of covid-19 infection: systematic review and critical appraisal.
      ].
      Clinical practice guidelines with the highest methodological quality, classified as S3 according to AWMF, are built on both a systemic literature review and a structured consensus process by representative experts. Only 4% of all guidelines in our study were S3. Another 6% reported solely a structured consensus process without a systematic literature review (classified as S2k according to AWMF). The majority (83%) was based on an informal consensus by a group of experts only. However, even if some guidelines developers might argue that well-designed studies were lacking at a certain time point and a systemic literature review as a basis for the guideline development was therefore not conducted, a structured consensus process could still have been done.
      A clear updating process, patient and public involvement, a systematic development processes incorporating systematic review, an explicit recommendation development process (rigor of development domain), and guidance on implementation of these guidelines (applicability domain) are needed areas to improve guideline development quality. Given the increasing evidence related to the COVID-19 pandemic, a clear updating process, as well as information of where previous and later versions of the guideline can be found might be essential information which should be explicitly stated in each document. Although patient and public involvement is recognized as a key component of clinical practice guideline development [
      • Armstrong MJ
      • Mullins CD
      • Gronseth GS
      • Gagliardi AR.
      Impact of patient involvement on clinical practice guideline development: a parallel group study.
      ], almost no citizens, patients or their public representatives were involved in the development of any of the documents, except for one guideline [
      • Sultan S
      • Lim JK
      • Altayar O
      • Davitkov P
      • Feuerstein JD
      • Siddique SM
      • et al.
      AGA Institute rapid recommendations for gastrointestinal procedures during the COVID-19 pandemic.
      ].
      Non-physician health professionals provide essential care, especially in the area of non-pharmacological interventions including physical activity/exercise, activity pacing, pain management, complex chronic disease management, nutrition, speech/aural care, personal care, disability, rehabilitation, substance use, aged care and psychosocial health [
      • Frenk J
      • Chen L
      • Bhutta ZA
      • Cohen J
      • Crisp N
      • Evans T
      • et al.
      Health professionals for a new century: transforming education to strengthen health systems in an interdependent world.
      ]. Some of these interventions have been continuously provided during the current COVID-19 situation, some health services have not been delivered due to home quarantine and physical distancing [

      Pan American Health Organization. Rehabilitation considerations during the COV ID-19 outbreak Available at: https://iris.paho.org/handle/10665.2/52035. 2020. Accessed May 22, 2020.

      ]. Guidelines and recommendations on how to manage and respond to this situation are often missing. Currently, the majority of the published guidelines, which we reviewed, did not focus on the specific needs of non-physician health professionals who are usually in direct/face-to-face patient contact. However, due to often unclear information about the background and role of each author in the guideline development process, transparency on stakeholder involvement was sometimes lacking and judgment was left to the discretion of the reviewers. Balanced stakeholder involvement as well as explicit information about this process in the paper would be necessary for a transparent guideline development that takes into account different treatment perspectives.
      Our study focused on quality criteria of the published guidelines and recommendations. We did not extract, compare and synthesize the medical content of the recommendations, due to the heterogeneity of medical disciplines included in this work. This might be regarded as a limitation of our work. Nevertheless, our results might also contribute to a further conversation on how best to mobilize national and international resources to develop high-quality guidelines in crisis situations in the future.

      5. Conclusion

      A considerable number of medical guidelines and recommendations were published in the context of the COVID-19 pandemic from February to April 2020. Despite high AGREE II scores for scope and purpose as well as for clarity of presentation, the majority of the guidelines and recommendations fell short of basic methodological standards. This was mainly due to a lack of appropriate methodologies and rigorous strategies in the guideline development process. Updated guidelines should include up-to-date information, transparent stakeholder involvement, consideration of various patient populations, and rigorous strategies and methodologies. Our study will allow clinicians to better select and appraise clinical practice guidelines. It will create awareness and inspire guideline developers, researchers and journal editors to focus on methodological accuracy instead of publishing large quantities of documents which lack basic methodological standards. Future research should monitor the evolving methodological quality of the guidelines and their updates over time. This paper provides an important benchmark for the future quality assessment of COVID-19 guidelines.

      Details of ethics approval

      Not required.

      Consent for publication

      All authors have taken an active part in the study and take responsibility for its contents. The publication of this manuscript is approved by all authors

      Availability of data and material

      All relevant data are up-loaded as supplemental files.

      Funding

      This study was partly funded by the COVID-19 Rapid Response Funding Scheme of the Vienna Science and Technology Fund (project number COV20-028 ). The funding institution had no influence on the results of this work.

      Patient and public involvement

      Three co-authors (JH, SMa and EMor) are patient representatives/advocates and participated in setting up the study concept and design, interpreting the data and writing and reviewing the manuscript.

      Protocol

      The protocol was published at www.researchgate.net, DOI: 10.13140/RG.2.2.21293.51689.
      Funding Source: This study was partly funded by the COVID-19 Rapid Response Funding Scheme of the Vienna Science and Technology Fund (project number COV20-028). The funding institution had no influence on the results of this work.
      Registration/Publication: Protocol published at www.researchgate.net, DOI:10.13140/RG.2.2.21293.51689; preliminary results available at https://www.medrxiv.org/content/10.1101/2020.06.19.20134767v1.

      Authors’ contributions

      MA, EMos, VR, LL, JM, ME, NF, RF, GGä, SMa, SMi, RMo, SMu, SN, MO, JH, GGa, and TS were responsible for the study conceptualization and design. MA, LL, EMos, JM, MO, VR and TS conducted the literature review and extracted and analyzed data. MA, EMos, VR, LL, JM, AA, SB, NB, ME, NF, RF, GGä, SMa, SMi, RMo, EMor, SMu, SN, MO, MSP, JH, GGa, and TS interpreted the data, wrote and reviewed the manuscript. MA, VR and TS were responsible for the visualization, including tables and figures. All authors have been involved in the revision of the manuscript.

      Acknowledgments

      We would like to thank Mohammed Gamal, Judith Höllebrand, Claudia Oppenauer, Romualdo Ramos, Sinisa Stefanac and Thea Vliet Vlieland for their support in data management and thoughtful advice.

      Appendix. Supplementary materials

      References

        • Zhu N
        • Zhang D
        • Wang W
        • Li X
        • Yang B
        • Song J.
        • et al.
        A novel coronavirus from patients with pneumonia in China, 2019.
        N Engl J Med. 2020; 382: 727-733
      1. World Health Organization. Clinical management of severe acute respiratory infection when novel coronavirus (2019-nCoV) infection is suspected—interim guidance. World Health Organization. Available at: https://www.who.int/publications-detail/clinical-management-of-severe-acute-respiratory-infection-when-novel-coronavirus-(ncov)-infection-is-suspected. 2020. Accessed April 5, 2020.

      2. American Academy of Family Physicians (AAFP). Clinical practice guideline manual. Available at: https://www.aafp.org/patient-care/clinical-recommendations/cpg-manual.html. 2017. Accessed May 22, 2020.

        • Institute of Medicine
        Committee on standards for developing trustworthy clinical practice guidelines.
        in: Graham R Mancher M. Clinical practice guidelines we can trust. National Academies Press, Washington, DC2011
        • Brouwers MC
        • Kho ME
        • Browman GP
        • et al.
        AGREE II: advancing guideline development, reporting and evaluation in health care.
        CMAJ. 2010; 182: E839-E842
        • Garritty C GG
        • Kamel C
        • King VJ
        • Nussbaumer-Streit B
        • Stevens A
        • Hamel C
        • et al.
        Cochrane rapid reviews interim guidance from the Cochrane rapid reviews methods group.
        2020
        • Peters MD
        • Godfrey CM
        • Khalil H
        • McInerney P
        • Parker D
        • Soares CB.
        Guidance for conducting systematic scoping reviews.
        Int J Evid Based Healthc. 2015; 13: 141-146
        • Moher D
        • Liberati A
        • Tetzlaff J
        • Altman DG
        • The PG.
        Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement.
        PLoS Med. 2009; 6e1000097
        • Muche-Borowski C
        • Selbmann H
        • Nothacker M
        • Müller W
        • Kopp I.
        AWMF guidance manual and rules for guideline development.
        2012 (English version)
        • Mahmoudjafari Z
        • Alexander M
        • Roddy J
        • Shaw R
        • Shigle TL
        • Timlin C
        • et al.
        American Society for Transplantation and Cellular Therapy Pharmacy Special Interest Group Position Statement on pharmacy practice management and clinical management for COVID-19 in hematopoietic cell transplant and cellular therapy patients in the United States.
        Biol Blood Marrow Transplant. 2020; 26: 1043-1049
        • Thomas P
        • Baldwin C
        • Bissett B
        • Boden I
        • Gosselink R
        • Granger CL
        • et al.
        Physiotherapy management for COVID-19 in the acute hospital setting: clinical practice recommendations.
        J Physiother. 2020; 66: 73-82
        • Song Z
        • Hu Y
        • Zheng S
        • Yang L
        • Zhao R.
        Hospital pharmacists’ pharmaceutical care for hospitalized patients with COVID-19: recommendations and guidance from clinical experience.
        Res Soc Admin Pharm. 2021; 17: 2027-2031
        • S-q Zheng
        • L Yang
        • P-x Zhou
        • H-b Li
        • Liu F
        • R-s Zhao
        Recommendations and guidance for providing pharmaceutical care services during COVID-19 pandemic: a China perspective.
        Res Soc Admin Pharm. 2021; 17: 1819-1824
        • Sultan S
        • Lim JK
        • Altayar O
        • Davitkov P
        • Feuerstein JD
        • Siddique SM
        • et al.
        AGA Institute rapid recommendations for gastrointestinal procedures during the COVID-19 pandemic.
        Gastroenterology. 2020; 159: 739-758
        • Alhazzani W
        • Møller MH
        • Arabi YM
        • Loeb M
        • Gong Michelle Ng
        • Fan E
        • et al.
        Surviving sepsis campaign: guidelines on the management of critically ill adults with coronavirus disease 2019 (COVID-19).
        Intensive Care Med. 2020; : 1-34
        • Gralnek IM
        • Hassan C
        • Beilenhoff U
        • Antonelli G
        • Ebigbo Alanna
        • Pellisé Maria
        • et al.
        ESGE and ESGENA position statement on gastrointestinal endoscopy and the COVID-19 pandemic.
        Endoscopy. 2020; 52: 1-818
        • Motlagh A
        • Yamrali M
        • Azghandi S
        • Azadeh P
        • Vaezi M
        • Ashrafi F
        • et al.
        COVID19 prevention & care; a cancer specific guideline.
        Arch Iran Med. 2020; 23: 255-264
        • Rubin GD
        • Ryerson CJ
        • Haramati LB
        • Sverzellati N
        • Kanne JP
        • Raoof S
        • et al.
        The role of chest imaging in patient management during the COVID-19 pandemic: a multinational consensus statement from the Fleischner Society.
        Chest. 2020; 158: 106-116
        • Jin Y-H
        • Cai L
        • Cheng Z-S
        • Cheng H
        • Deng T
        • Fan Y-P
        • et al.
        A rapid advice guideline for the diagnosis and treatment of 2019 novel coronavirus (2019-nCoV) infected pneumonia (standard version).
        Mil Med Res. 2020; 7: 4
        • Zhao H-M
        • Xie Y-X
        • Wang C.
        Recommendations for respiratory rehabilitation in adults with COVID-19.
        Chin Med J (Engl). 2020; 133: 1595-1602
        • Jin Y-H
        • Zhan Q-Y
        • Peng Z-Y
        • Ren X-Q
        • Yin X-T
        • Cai L
        • et al.
        Chemoprophylaxis, diagnosis, treatments, and discharge management of COVID-19: An evidence-based clinical practice guideline (updated version).
        Mil Med Res. 2020; 7: 1-33
        • Gralnek IM
        • Hassan C
        • Beilenhoff U
        • Antonelli G
        • Ebigbo A
        • Pellisé M
        • et al.
        ESGE and ESGENA position statement on gastrointestinal endoscopy and COVID-19: an update on guidance during the post-lockdown phase and selected results from a membership survey.
        Endoscopy. 2020; 52891–81
        • Alhazzani W
        • Evans L
        • Alshamsi F
        • Møller MH
        • Ostermann M
        • Prescott HC
        • et al.
        Surviving sepsis campaign guidelines on the management of adults with coronavirus disease 2019 (COVID-19) in the ICU: first update.
        Crit Care Med. 2021; 49: e219-e234
        • Sultan S
        • Altayar O
        • Siddique SM
        • Davitkov P
        • Feuerstein JD
        • Lim JK
        • et al.
        AGA Institute rapid review of the GI and liver manifestations of COVID-19, meta-analysis of international data, and recommendations for the consultative management of patients with COVID-19.
        Gastroenterology. 2020; 159: 320-334
        • Wynants L
        • Van Calster B
        • Collins GS
        • Riley RD
        • Heinze G
        • Schuit E
        • et al.
        Prediction models for diagnosis and prognosis of covid-19 infection: systematic review and critical appraisal.
        BMJ. 2020; 369m1328
        • Armstrong MJ
        • Mullins CD
        • Gronseth GS
        • Gagliardi AR.
        Impact of patient involvement on clinical practice guideline development: a parallel group study.
        Implement Sci. 2018; 13: 55
        • Frenk J
        • Chen L
        • Bhutta ZA
        • Cohen J
        • Crisp N
        • Evans T
        • et al.
        Health professionals for a new century: transforming education to strengthen health systems in an interdependent world.
        Lancet North Am Ed. 2010; 376: 1923-1958
      3. Pan American Health Organization. Rehabilitation considerations during the COV ID-19 outbreak Available at: https://iris.paho.org/handle/10665.2/52035. 2020. Accessed May 22, 2020.