Advertisement

Challenges in applying the GRADE approach in public health guidelines and systematic reviews: a concept article from the GRADE Public Health Group

  • Michele Hilton Boon
    Correspondence
    Correspondence: Michele Hilton Boon, MRC/CSO Social and Public Health Sciences Unit, Berkeley Square, 99 Berkeley Street, University of Glasgow, G3 7HR, UK. Tel: +44 141 353 7500.
    Affiliations
    MRC/CSO Social and Public Health Sciences Unit, Berkeley Square, 99 Berkeley Street, University of Glasgow, Glasgow G3 7HR, UK
    Search for articles by this author
  • Hilary Thomson
    Affiliations
    MRC/CSO Social and Public Health Sciences Unit, Berkeley Square, 99 Berkeley Street, University of Glasgow, Glasgow G3 7HR, UK
    Search for articles by this author
  • Beth Shaw
    Affiliations
    Center for Evidence-based Policy, Oregon Health & Science University, Portland, OR 97201 USA
    Search for articles by this author
  • Elie A. Akl
    Affiliations
    Department of Health Research Methods, Evidence, and Impact, McMaster University, 1280 Main Street W, Hamilton, Ontario L8S 4K1, Canada

    Department of Internal Medicine, American University of Beirut, Beirut, Lebanon
    Search for articles by this author
  • Stefan K. Lhachimi
    Affiliations
    Department for Health Services Research, Institute of Public Health and Nursing Research, University of Bremen, Grazer Straße 4, 28359 Bremen, Germany

    Health Sciences Bremen, University of Bremen, 28359 Bremen, Germany
    Search for articles by this author
  • Jesús López-Alcalde
    Affiliations
    Department of Paediatrics, Obstetrics & Gynaecology and Preventative Medicine, Universitat Autònoma de Barcelona; Faculty of Health Sciences, Universidad Francisco de Vitoria (UFV)-Madrid; Clinical Biostatistics Unit, Hospital Universitario Ramón y Cajal (IRYCIS); CIBER Epidemiology and Public Health; Cochrane Associate Centre of Madrid, Madrid, Spain
    Search for articles by this author
  • Miloslav Klugar
    Affiliations
    Faculty of Medicine, Czech National Centre for Evidence-Based Healthcare and Knowledge Translation (Cochrane Czech Republic, The Czech Republic Centre for Evidence-Based Healthcare; JBI Centre of Excellence, Masaryk University GRADE Centre), Institute of Biostatistics and Analyses, Masaryk University, 625 00 Brno, Czechia
    Search for articles by this author
  • Leslie Choi
    Affiliations
    The Department of Vector Biology, Partnership for Increasing the Impact of Vector Control, Liverpool School of Tropical Medicine, Liverpool, UK
    Search for articles by this author
  • Zuleika Saz-Parkinson
    Affiliations
    European Commission, Joint Research Centre, Ispra, Italy
    Search for articles by this author
  • Reem A. Mustafa
    Affiliations
    Department of Health Research Methods, Evidence, and Impact, McMaster University, 1280 Main Street W, Hamilton, Ontario L8S 4K1, Canada

    Departments of Medicine and Biomedical & Health Informatics, University of Missouri-Kansas City, Kansas City, MO 66160 USA
    Search for articles by this author
  • Miranda W. Langendam
    Affiliations
    Department of Clinical Epidemiology, Amsterdam University Medical Centres, University of Amsterdam, Biostatistics and Bioinformatics, Amsterdam Public Health Research Institute, Amsterdam, the Netherlands
    Search for articles by this author
  • Olivia Crane
    Affiliations
    National Institute for Health and Care Excellence (NICE), Level 1A, City Tower, Piccadilly Plaza, Manchester M1 4BT, UK
    Search for articles by this author
  • Rebecca L. Morgan
    Affiliations
    Department of Health Research Methods, Evidence, and Impact, McMaster University, 1280 Main Street W, Hamilton, Ontario L8S 4K1, Canada
    Search for articles by this author
  • Eva Rehfuess
    Affiliations
    Institute for Medical Informatics, Biometry and Epidemiology, Pettenkofer School of Public Health, LMU Munich, Munich, Germany
    Search for articles by this author
  • Bradley C. Johnston
    Affiliations
    Department of Nutrition, Texas A&M University, College Station, TX, USA
    Search for articles by this author
  • Lee Yee Chong
    Affiliations
    Cochrane Public Health and Health Systems Network, University of Oxford, Oxford, UK
    Search for articles by this author
  • Gordon H. Guyatt
    Affiliations
    Department of Health Research Methods, Evidence, and Impact, McMaster University, 1280 Main Street W, Hamilton, Ontario L8S 4K1, Canada
    Search for articles by this author
  • Holger J. Schünemann
    Affiliations
    Department of Health Research Methods, Michael G DeGroote Cochrane Canada and McMaster GRADE Centres, and WHO Collaborating Centre for Infectious Diseases, Research Methods and Recommendations, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada
    Search for articles by this author
  • Srinivasa Vittal Katikireddi
    Affiliations
    MRC/CSO Social and Public Health Sciences Unit, Berkeley Square, 99 Berkeley Street, University of Glasgow, Glasgow G3 7HR, UK
    Search for articles by this author
  • For theGRADE Working Group
Open AccessPublished:January 18, 2021DOI:https://doi.org/10.1016/j.jclinepi.2021.01.001

      Abstract

      Background and Objective

      This article explores the need for conceptual advances and practical guidance in the application of the GRADE approach within public health contexts.

      Methods

      We convened an expert workshop and conducted a scoping review to identify challenges experienced by GRADE users in public health contexts. We developed this concept article through thematic analysis and an iterative process of consultation and discussion conducted with members electronically and at three GRADE Working Group meetings.

      Results

      Five priority issues can pose challenges for public health guideline developers and systematic reviewers when applying GRADE: (1) incorporating the perspectives of diverse stakeholders; (2) selecting and prioritizing health and “nonhealth” outcomes; (3) interpreting outcomes and identifying a threshold for decision-making; (4) assessing certainty of evidence from diverse sources, including nonrandomized studies; and (5) addressing implications for decision makers, including concerns about conditional recommendations. We illustrate these challenges with examples from public health guidelines and systematic reviews, identifying gaps where conceptual advances may facilitate the consistent application or further development of the methodology and provide solutions.

      Conclusion

      The GRADE Public Health Group will respond to these challenges with solutions that are coherent with existing guidance and can be consistently implemented across public health decision-making contexts.

      Keywords

      What is new?

        Key findings

      • Nine studies of guideline developers’ experiences reported challenges when using GRADE in public health and related fields.
      • The GRADE Public Health project group will respond to these challenges by producing guidance relating to social determinants, population-level outcomes, and rating certainty of evidence from interrupted time series.

        What this adds to what was known

      • Thematic analysis indicates the challenges relate to diverse perspectives and outcomes, decision-making thresholds, and nonrandomized studies.
      • Case studies drawing on Cochrane reviews and public health guidelines provide examples of how these challenges are encountered in practice.

        What is the implication and what should change now?

      • Detailed examples, new guidance, and targeted training materials should be developed to support public health and public policy audiences using GRADE.

      1. Introduction

      1.1 Overview

      For the past decade, the GRADE Working Group has produced tools and guidance with the aim of reducing unnecessary confusion and variation in rating the certainty in synthesized evidence and improving the transparency and methodological rigor of guidelines. Several GRADE outputs and project groups have specifically addressed public health topics and used examples from public health guidelines [
      • Alonso-Coello P.
      • Schünemann H.J.
      • Moberg J.
      • Brignardello-Petersen R.
      • Akl E.A.
      • Davoli M.
      • et al.
      GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 1: Introduction.
      ,
      • Moberg J.
      • Oxman A.D.
      • Rosenbaum S.
      • Schünemann H.J.
      • Guyatt G.
      • Flottorp S.
      • et al.
      The GRADE Evidence to Decision (EtD) framework for health system and public health decisions.
      ,
      • Schünemann H.J.
      • Mustafa R.
      • Brozek J.
      • Santesso N.
      • Alonso-Coello P.
      • Guyatt G.
      • et al.
      GRADE Guidelines: 16. GRADE evidence to decision frameworks for tests in clinical practice and public health.
      ]. At the same time, public health guideline developers and others using GRADE in public health-related contexts have identified ongoing challenges. To address these challenges and consider the solutions, the GRADE Working Group approved in 2017 the formation of the GRADE Public Health Project Group. This article, which is the first output of this project group, is not GRADE guidance but a GRADE concept article. A GRADE concept article is a new type of communication from the GRADE Working Group that discusses the need for conceptual advances in GRADE methodology.
      This article reports the results of activities undertaken to identify some of the challenges relating to GRADE and public health guidelines, specifically through an initial stakeholder workshop and a scoping review. We propose priority areas for further consideration and explain the rationale for these choices. We present four case studies of public health guidelines and systematic reviews (Case study 1, Case study 2, Case study 3, Case study 4) to illustrate how these challenges have been encountered in practice. The article concludes by summarizing the types of solutions that may be proposed for these challenges and the next steps for the GRADE Public Health Group.
      Cochrane systematic reviews on unconditional cash transfers
      Tabled 1
      Contributed by: Stefan Lhachimi
      Background
      Unconditional cash transfers (UCTs, money provided without obligation) are an intervention that addresses a key social determinant of health, i.e., income. Two recent Cochrane reviews [
      • Pega F.
      • Liu S.Y.
      • Walter S.
      • Lhachimi S.K.
      Unconditional cash transfers for assistance in humanitarian disasters: effect on use of health services and health outcomes in low- and middle-income countries.
      ,
      • Pega F.
      • Liu S.Y.
      • Walter S.
      • Pabayo R.
      • Saith R.
      • Lhachimi S.K.
      Unconditional cash transfers for reducing poverty and vulnerabilities: effect on use of health services and health outcomes in low- and middle-income countries.
      ] assessed the health effects of UCTs: the first in the context of (humanitarian or natural) disasters and the second as a social protection intervention for reducing poverty and vulnerabilities (e.g., orphanhood, old age, or HIV infection). Both reviews included studies conducted in low- and middle-income countries only and had as primary outcomes the use of health services and health outcomes. Secondary outcomes were relevant social determinants of health (e.g., assets, education, labor force participation, parenting quality, and extreme poverty) and health care expenditure.
      Examples of challenges
      Challenges 2 and 3: Most studies included in the review of UCTs as a social protection intervention were conducted by economists. Outcome reporting in the economic literature may not take into account considerations necessary for evidence synthesis, such as choosing and defining outcomes in common with already existing studies on the same topic or reporting intracluster correlation coefficients in cluster randomized trials. Some studies have dozens of very similar outcomes (e.g., number of goats, number of cows, number of chickens, etc.), making it difficult to choose which one to report in a systematic review. Moreover, for many outcome measures used (in particular indices of food security and diet diversity), no agreed minimal important difference exists. The Summary of Findings table went through several iterations during the review process before agreement was reached on which and how many outcomes to report.
      Challenges 4 and 5: In the review of UCTs for disaster relief, no high-quality evidence was identified. In addition, all included studies investigated UCTs as a response to only one type of disaster, namely droughts, which are different from most disasters in that droughts can be anticipated with a relatively long lead time before the consequences of the disaster materialize. Accordingly, the evidence was seriously indirect as well as low quality. Nevertheless, the UN at the World Humanitarian Summit in Istanbul 2016 called for making UCTs the default option for help during disasters. Intuitively, this seems plausible as UCTs in a short-term disaster context are (compared to in-kind transfers) swift and fairly easily to administer. However, this view does not account for potential crowding out effects of UCTs, i.e., unintended negative effects of UCT provision on other types of financial assistance during disasters.
      Cochrane systematic reviews to inform WHO malaria vector control guidelines
      Tabled 1
      Contributed by: Leslie Choi
      Background
      In 2016, the WHO commissioned the development of new guidelines for malaria vector control [
      World Health Organization
      Guidelines for malaria vector control.
      ] in partnership with the Cochrane Infectious Diseases Group (CIDG) based at the Liverpool School of Tropical Medicine. CIDG contributed eight systematic reviews [
      • Choi L.
      • Pryce J.
      • Garner P.
      The combination of indoor residual spraying with insecticide-treated nets versus insecticide-treated nets alone for preventing malaria.
      ,
      • Choi L.
      • Wilson A.
      Larviciding to control malaria.
      ,
      • Gleave K.
      • Lissenden N.
      • Richardson M.
      • Choi L.
      • Ranson H.
      Piperonyl butoxide (PBO) combined with pyrethroids in insecticide-treated nets to prevent malaria in Africa.
      ,
      • Maia M.F.
      • Kliner M.
      • Richardson M.
      • Lengeler C.
      • Moore S.J.
      Mosquito repellents for malaria prevention.
      ,
      • Pluess B.
      • Tanser F.C.
      • Lengeler C.
      • Sharp B.L.
      Indoor residual spraying for preventing malaria.
      ,
      • Pryce J.
      • Choi L.
      • Richardson M.
      • Malone D.
      Insecticide space spraying for preventing malaria transmission.
      ,
      • Pryce J.
      • Richardson M.
      • Lengeler C.
      Insecticide-treated nets for preventing malaria.
      ,
      • Walshe D.P.
      • Garner P.
      • Adeel A.A.
      • Pyke G.H.
      • Burkot T.R.
      Larvivorous fish for preventing malaria transmission.
      ] (five de novo, two updates, and one previously published review) that were used to inform the recommendations made in these guidelines.
      Examples of challenges
      Challenges 2 and 3: Vector control tools are typically public health interventions distributed at a community level. To evaluate efficacy at a community level, appropriate study designs with applicable outcomes are required. The main challenge encountered with these guidelines is how to tailor the guidelines for the correct target audience. Are they for individuals wanting to protect themselves from malaria or for national malaria program planners? Paradoxically, increased protection for some individuals may translate into increased risk for others in the community who are not as well protected. Leading on from this, it is difficult to assess whether the study evidence included in the systematic reviews demonstrates community protection. For example, the systematic review on topical repellents combined studies that distributed the intervention at an individual level and those that distributed at a community level. Therefore, it was unclear if the conclusions drawn from that review were applicable to both contexts.
      Challenge 4: Some modelling studies have suggested that poor coverage of vector control tools leads to more harm than good in a community by protecting a few individuals at the expense of the majority. However, these findings were difficult to capture and reconcile with the evidence from RCTs within the Summary of Findings and by extension the Evidence to Decision (EtD) framework.
      Czech national public health guideline on 1-day surgery
      Tabled 1
      Contributed by: Miloslav Klugar
      Background
      One-day surgery was identified as a public health topic within the national Clinical Practice Guidelines project by the Guarantor Committee (representatives of important policymakers in the Czech Republic from the Ministry of Health, Institute of Health Information and Statistics, Czech Health Research Council, and health insurance organizations). One-day surgery has various definitions in different health systems worldwide; the definition for the Czech health system agreed by guideline panellists is “surgical performance (diagnostic and therapeutic) for hospitalization not exceeding 1 day (up to 24 hours of hospitalization, including one overnight stay).” In the Czech Republic, there are several surgical procedures in some hospitals in the 1-day surgery regime, while in other hospitals patients may be routinely hospitalized for the same surgical procedures for 3-6 days. One-day surgery clearly involves surgeons as a key audience, but also health care users, health care providers, and policymakers. In addition, following GRADE guidance, it was clear that social work and community care stakeholders needed to be engaged, as timely discharge from hospital and patient safety following discharge depend on social and community care.
      Examples of challenges
      Challenge 1: Initially the guideline panel membership included only surgeons and methodologists and the focus was on identifying which surgical procedures should be covered by 1-day surgery. There was initial resistance to inviting allied health professionals, patients, and other relevant stakeholders to join the panel. Furthermore, after framing the guideline questions the panel realized that social work/community care stakeholders should also be on the panel and that representation was needed from health and social insurance organizations, as well as homecare organizations. Identifying representatives from different organizations/sectors and managing the work of such a diverse guideline panel presented a challenge.
      Challenge 2: Although the Population, Intervention, Comparison, Outcome format is straightforward in its application to public health questions, the panel faced a challenge in prioritizing clinical versus social and community care outcomes, particularly in terms of the identification and management of postsurgical complications and patient safety after discharge from hospital.
      European Breast Guidelines
      Tabled 1
      Contributed by: Jesús López Alcalde and Zuleika Saz Parkinson
      Background
      The European Commission's Joint Research Center (JRC) supports European Union policies with independent scientific evidence [
      European Commission
      JRC in brief 2019.
      ]. The JRC coordinates the development of the European guidelines for breast cancer screening and diagnosis [
      European Commission
      European Breast Guidelines 2019.
      ] (in short, the European Breast Guidelines). These guidelines follow GRADE [
      European Commission
      Methods of the European guidelines for breast cancer screening and diagnosis 2016.
      ]. The European Breast Guidelines provide recommendations on population-based breast cancer screening programs and diagnostic procedures for breast cancer [
      • Schunemann H.J.
      • Lerda D.
      • Dimitrova N.
      • Alonso-Coello P.
      • Grawingholt A.
      • Quinn C.
      • et al.
      Methods for development of the European commission initiative on Breast cancer guidelines recommendations in the era of guideline transparency.
      ,
      European Commission
      Scope of the European guidelines for breast cancer screening and diagnosis.
      ,
      • Schünemann H.J.
      • Lerda D.
      • Quinn C.
      • Follmann M.
      • Alonso-Coello P.
      • Rossi P.G.
      • et al.
      Breast cancer screening and diagnosis: a synopsis of the European Breast guidelines.
      ]. The target audience of the guidelines is heterogeneous, covering health care providers, service users, and policymakers.
      Examples of challenges
      Challenges 1 and 2: To define the perspective of the European Breast Guidelines was not straightforward. Some questions are purely clinical, taking an individual perspective, while others have a population perspective. These aspects influence the selection of the guideline questions and outcomes. The prioritization of the outcomes to determine the effects of breast cancer screening programs is also challenging; some panellists may take the safety of screening interventions for granted and be reluctant to include harms as outcomes.
      Challenges 3 and 4: The GRADE approach requires a multidisciplinary team with specific skills, which may not be available in all public health institutions. Therefore, certain tasks must be outsourced. The first guideline developed with GRADE in a public health institution can be logistically complicated, but the replication in future guidelines is expected to be more straightforward.
      Challenge 5: As RCTs for some questions are scarce, or even thought to be unethical, the use of GRADE is expected to generate conditional recommendations. The misconception that GRADE gives too much weight to RCT evidence may partly explain this.

      1.2 Background

      Public health is concerned with “preventing disease, prolonging life, and promoting health through the organized efforts of society.” [
      • Acheson D.
      Public Health in England: The Report of the Committee of Inquiry into the Future Development to the Public Health Function.
      ]. Public health encompasses three domains—health protection, health services, and health improvement [
      • Griffiths S.
      • Jewell T.
      • Donnelly P.
      Public health in practice: the three domains of public health.
      ]—within a disciplinary approach which recognizes the impact of social determinants on individual and population health outcomes and the importance of reducing health inequalities. Decisions made by public health practitioners, policymakers, and organizations may relate to interventions that are not implemented at the individual level, including health system reform, regulation of unhealthy commodities (such as taxation of alcohol or sugar-sweetened beverages), infrastructure development, and social security policies. Depending on the approach taken, policy goals may be competing or may produce contradictory results, for example, when addressing health inequalities [
      • Benach J.
      • Malmusi D.
      • Yasui Y.
      • Martínez J.M.
      A new typology of policies to tackle health inequalities and scenarios of impact based on Rose's population approach.
      ]. Decisions about such interventions and situations involve stakeholders from many different professions and organizations. Consensus on critical outcomes may be challenging to achieve and evidence from well-conducted randomized studies may be lacking. However, despite any lack of randomized studies, decisions must be based on the best available evidence.
      In this context, the GRADE approach has strengths which are well recognized. GRADE's structured and transparent methods for integrating evidence into decisions have been tested and implemented by guideline developers worldwide. Previous research has reported the adoption and further development of GRADE methods in public health contexts [
      • Ahmed F.
      • Temte J.L.
      • Campos-Outcalt D.
      • Schünemann H.J.
      • Recommendation A.E.B.
      • Group A.E.B.R.W.
      Methods for developing evidence-based recommendations by the advisory committee on immunization practices (ACIP) of the U.S. Centers for disease control and prevention (CDC).
      ,
      • Dahm P.
      • Oxman A.D.
      • Djulbegovic B.
      • Guyatt G.H.
      • Murad M.H.
      • Amato L.
      • et al.
      Stakeholders apply the GRADE evidence-to-decision framework to facilitate coverage decisions.
      ,
      • Schunemann H.J.
      • Hill S.R.
      • Kakad M.
      • Vist G.E.
      • Bellamy R.
      • Stockman L.
      • et al.
      Transparent development of the WHO rapid advice guidelines.
      ,
      • Schunemann H.J.
      • Lerda D.
      • Dimitrova N.
      • Alonso-Coello P.
      • Grawingholt A.
      • Quinn C.
      • et al.
      Methods for development of the European commission initiative on Breast cancer guidelines recommendations in the era of guideline transparency.
      ,
      • Thornton J.
      • Alderson P.
      • Tan T.
      • Turner C.
      • Latchem S.
      • Shaw E.
      • et al.
      Introducing GRADE across the NICE clinical guideline program.
      ]. Previous research has also investigated stakeholder views and experiences with GRADE and confirmed that GRADE principles and processes are applicable to public health systematic reviews and guidelines [
      • Akl E.A.
      • Kennedy C.
      • Konda K.
      • Caceres C.F.
      • Horvath T.
      • Ayala G.
      • et al.
      Using GRADE methodology for the development of public health guidelines for the prevention and treatment of HIV and other STIs among men who have sex with men and transgender people.
      ,
      • Alexander P.E.
      • Gionfriddo M.R.
      • Li S.A.
      • Bero L.
      • Stoltzfus R.J.
      • Neumann I.
      • et al.
      A number of factors explain why WHO guideline developers make strong recommendations inconsistent with GRADE guidance.
      ,
      • Rehfuess E.A.
      • Akl E.A.
      Current experience with applying the GRADE approach to public health interventions: an empirical study.
      ]. However, challenges have also been identified for which various solutions have been proposed [
      • Akl E.A.
      • Kennedy C.
      • Konda K.
      • Caceres C.F.
      • Horvath T.
      • Ayala G.
      • et al.
      Using GRADE methodology for the development of public health guidelines for the prevention and treatment of HIV and other STIs among men who have sex with men and transgender people.
      ,
      • Rehfuess E.A.
      • Akl E.A.
      Current experience with applying the GRADE approach to public health interventions: an empirical study.
      ,
      • Dedios M.C.
      • Esperato A.
      • De-Regil L.M.
      • Pena-Rosas J.P.
      • Norris S.L.
      Improving the adaptability of WHO evidence-informed guidelines for nutrition actions: results of a mixed methods evaluation.
      ,
      European Centre for Disease Prevention and Control
      Evidence-based methodologies for public health [Technical report].
      ,
      • Harder T.
      • Abu Sin M.
      • Bosch-Capblanch X.
      • Bruno C.
      • de Carvalho Gomes H.
      • Duclos P.
      • et al.
      Towards a framework for evaluating and grading evidence in public health.
      ,
      • Thayer K.A.
      • Schünemann H.J.
      Using GRADE to respond to health questions with different levels of urgency.
      ,
      • Florez I.D.
      • Morgan R.L.
      • Falavigna M.
      • Kowalski S.C.
      • Zhang Y.
      • Etxeandia-Ikobaltzeta I.
      • et al.
      Development of rapid guidelines: 2. A qualitative study with WHO guideline developers.
      ,
      • Schünemann H.
      • Hill S.
      • Guyatt G.
      • Akl E.A.
      • Ahmed F.
      The GRADE approach and Bradford Hill's criteria for causation.
      ]. These challenges, along with the existence of multiple proposed solutions, represent areas in which further elaboration of GRADE guidance and production of exemplars may be beneficial to ensure consistent application of the methodology, support the production of high-quality systematic reviews and evidence-based guidelines, and allow for further improvement of the GRADE system.
      To explore experiences of GRADE users and lay the foundations for a GRADE public health project group, guideline developers and systematic reviewers were invited to participate in a workshop on GRADE for public health at the Global Evidence Summit in Cape Town, South Africa, in September 2017. Twenty participants contributed to a facilitated discussion of challenges encountered when applying GRADE to public health topics. Participants reported issues with agreeing on critical outcomes, assessing certainty when the body of evidence includes findings from disciplines other than health, making appropriate use of nonrandomized studies, and addressing stakeholder concerns about weak or conditional recommendations. In addition, participants raised the issue of the broad range of stakeholders involved in public health decision-making and how to address the differences and competing interests that may arise when guidelines apply to sectors other than health. Further challenges in applying GRADE included the need to integrate the perspectives of different disciplines and professions in areas such as planning, housing, transport, and social security; the need to address competing policy priorities and resource considerations across governmental agencies and geographical boundaries; and the involvement of both public sector and commercial interests.
      The formation of the GRADE Public Health project group was subsequently approved by the GRADE Guidance Group in October 2017 with the overarching goal to advance GRADE methodology in the development of public health guidelines. The work of the GRADE Public Health Group builds on previously developed and in-development GRADE guidance and related research on topics including equity, complex interventions, modeling studies, nonrandomized studies (NRS), and environmental and occupational health [
      • Akl E.A.
      • Welch V.
      • Pottie K.
      • Eslava-Schmalbach J.
      • Darzi A.
      • Sola I.
      • et al.
      GRADE equity guidelines 2: considering health equity in GRADE guideline development: equity extension of the guideline development checklist.
      ,
      • Cuello-Garcia C.A.
      • Morgan R.L.
      • Brozek J.
      • Santesso N.
      • Verbeek J.
      • Thayer K.
      • et al.
      A scoping review and survey provides the rationale, perceptions, and preferences for the integration of randomized and nonrandomized studies in evidence syntheses and GRADE assessments.
      ,
      • Morgan R.L.
      • Thayer K.A.
      • Bero L.
      • Bruce N.
      • Falck-Ytter Y.
      • Ghersi D.
      • et al.
      GRADE: assessing the quality of evidence in environmental and occupational health.
      ,
      • Pottie K.
      • Welch V.
      • Morton R.
      • Akl E.A.
      • Eslava-Schmalbach J.H.
      • Katikireddi V.
      • et al.
      GRADE equity guidelines 4: considering health equity in GRADE guideline development: evidence to decision process.
      ,
      • Schünemann H.J.
      • Cuello C.
      • Akl E.A.
      • Mustafa R.A.
      • Meerpohl J.J.
      • Thayer K.
      • et al.
      GRADE guidelines: 18. How ROBINS-I and other tools to assess risk of bias in nonrandomized studies should be used to rate the certainty of a body of evidence.
      ,
      • Welch V.A.
      • Akl E.A.
      • Guyatt G.
      • Pottie K.
      • Eslava-Schmalbach J.
      • Ansari M.T.
      • et al.
      GRADE equity guidelines 1: considering health equity in GRADE guideline development: introduction and rationale.
      ,
      • Welch V.A.
      • Akl E.A.
      • Pottie K.
      • Ansari M.T.
      • Briel M.
      • Christensen R.
      • et al.
      GRADE equity guidelines 3: considering health equity in GRADE guideline development: rating the certainty of synthesized evidence.
      ]. Here, we particularly focus on interventions that impact on whole populations or large population groups (i.e., are not delivered at the individual level or whose benefits are realized at a population level). Examples of such interventions include regulation of unhealthy commodities (e.g., restrictions on transfats in processed foods and minimum unit pricing of alcohol); the provision of new infrastructure (e.g., clean water and sanitation, affordable housing, and modifications to transport); vaccination programs; and health system reforms (e.g., the integration of health and social care services).

      2. Methods

      We conducted a scoping review with the aim of investigating (i) experiences of applying GRADE within public health systematic reviews and guideline development and (ii) existing research activity in this area. We included peer-reviewed published research on experiences, barriers, and facilitators encountered in public health contexts in the adoption of GRADE. We did not include methodological or technical documents that reported only the methods and not the experiences or reflections of reviewers or developers. We searched Medline and Embase (date range 2000–2018; see online supplement for search strategies); used the snowballing technique to identify additional relevant studies; searched the reference lists of included articles; and contacted experts in the field. We performed a narrative synthesis using thematic analysis to identify key issues and challenges reported in the literature. We then developed the analysis through an iterative process of consultation and discussion conducted with GRADE Working Group members electronically and at three GRADE Working Group meetings.

      3. Results

      We identified nine studies [
      • Akl E.A.
      • Kennedy C.
      • Konda K.
      • Caceres C.F.
      • Horvath T.
      • Ayala G.
      • et al.
      Using GRADE methodology for the development of public health guidelines for the prevention and treatment of HIV and other STIs among men who have sex with men and transgender people.
      ,
      • Alexander P.E.
      • Gionfriddo M.R.
      • Li S.A.
      • Bero L.
      • Stoltzfus R.J.
      • Neumann I.
      • et al.
      A number of factors explain why WHO guideline developers make strong recommendations inconsistent with GRADE guidance.
      ,
      • Rehfuess E.A.
      • Akl E.A.
      Current experience with applying the GRADE approach to public health interventions: an empirical study.
      ,
      • Dedios M.C.
      • Esperato A.
      • De-Regil L.M.
      • Pena-Rosas J.P.
      • Norris S.L.
      Improving the adaptability of WHO evidence-informed guidelines for nutrition actions: results of a mixed methods evaluation.
      ,
      European Centre for Disease Prevention and Control
      Evidence-based methodologies for public health [Technical report].
      ,
      • Alexander P.E.
      • Li S.-A.
      • Gionfriddo M.R.
      • Stoltzfus R.J.
      • Neumann I.
      • Brito J.P.
      • et al.
      Senior GRADE methodologists encounter challenges as part of WHO guideline development panels: an inductive content analysis.
      ,
      • Burford B.J.
      • Rehfuess E.
      • Schunemann H.J.
      • Akl E.A.
      • Waters E.
      • Armstrong R.
      • et al.
      Assessing evidence in public health: the added value of GRADE.
      ,
      • Gopinathan U.
      • Hoffman S.J.
      Institutionalising an evidence-informed approach to guideline development: progress and challenges at the World Health Organization.
      ,
      • Harder T.
      • Abu Sin M.
      • Bosch-Capblanch X.
      • Bruno C.
      • de Carvalho Gomes H.
      • Duclos P.
      • et al.
      Towards a framework for evaluating and grading evidence in public health.
      ] that specifically described the experiences of guideline development agencies or systematic review groups in applying GRADE to public health topics.
      Rehfuess and Akl gathered insights from 25 stakeholders in 12 organizations about their experiences and views of using GRADE in the field of public health [
      • Rehfuess E.A.
      • Akl E.A.
      Current experience with applying the GRADE approach to public health interventions: an empirical study.
      ]. They grouped the challenges identified by participants into six categories:
      • Complexity of public health interventions.
      • Choice of outcomes and outcome measures.
      • Ability to discriminate between different types of observational studies.
      • Use of nonepidemiological evidence.
      • GRADE terminology.
      • GRADE and guideline development process.
      Issues relating to GRADE terminology included definitions of “quality of evidence,” the use of the term “observational study,” and different understandings of terms such as “low-quality evidence” and “weak recommendations.” Participants reported challenges relating to the methodological difficulties of systematically reviewing nonrandomized studies, applying GRADE criteria to narrative synthesis in the absence of a pooled effect estimate, difficulties with the evidence-to-recommendations process, and concerns about the resources required to implement GRADE methodology [
      • Rehfuess E.A.
      • Akl E.A.
      Current experience with applying the GRADE approach to public health interventions: an empirical study.
      ].
      Five reports [
      • Akl E.A.
      • Kennedy C.
      • Konda K.
      • Caceres C.F.
      • Horvath T.
      • Ayala G.
      • et al.
      Using GRADE methodology for the development of public health guidelines for the prevention and treatment of HIV and other STIs among men who have sex with men and transgender people.
      ,
      • Alexander P.E.
      • Gionfriddo M.R.
      • Li S.A.
      • Bero L.
      • Stoltzfus R.J.
      • Neumann I.
      • et al.
      A number of factors explain why WHO guideline developers make strong recommendations inconsistent with GRADE guidance.
      ,
      • Dedios M.C.
      • Esperato A.
      • De-Regil L.M.
      • Pena-Rosas J.P.
      • Norris S.L.
      Improving the adaptability of WHO evidence-informed guidelines for nutrition actions: results of a mixed methods evaluation.
      ,
      • Alexander P.E.
      • Li S.-A.
      • Gionfriddo M.R.
      • Stoltzfus R.J.
      • Neumann I.
      • Brito J.P.
      • et al.
      Senior GRADE methodologists encounter challenges as part of WHO guideline development panels: an inductive content analysis.
      ,
      • Gopinathan U.
      • Hoffman S.J.
      Institutionalising an evidence-informed approach to guideline development: progress and challenges at the World Health Organization.
      ] have described experiences of using GRADE in the context of World Health Organization (WHO) guideline development. Akl et al reported their experiences of applying GRADE in the development of a WHO Department of HIV/AIDS guideline on public health interventions for men who have sex with men and transgender people [
      • Akl E.A.
      • Kennedy C.
      • Konda K.
      • Caceres C.F.
      • Horvath T.
      • Ayala G.
      • et al.
      Using GRADE methodology for the development of public health guidelines for the prevention and treatment of HIV and other STIs among men who have sex with men and transgender people.
      ]. They describe the solutions they developed within a single guideline for the following nine challenges:
      • Heterogenous and complex interventions.
      • Paucity of trial data.
      • Selecting outcomes of interest.
      • Using indirect evidence.
      • Integrating values and preferences.
      • Considering resource use.
      • Addressing social and legal barriers.
      • Wording of recommendations.
      • Developing global guidelines.
      Dedios et al. evaluated the methodological quality and implementability of guidelines produced by the WHO Department of Nutrition for Health and Development [
      • Dedios M.C.
      • Esperato A.
      • De-Regil L.M.
      • Pena-Rosas J.P.
      • Norris S.L.
      Improving the adaptability of WHO evidence-informed guidelines for nutrition actions: results of a mixed methods evaluation.
      ]. They suggested that difficulties encountered in their evaluation may reflect limitations of GRADE guidance, specifically citing NRS, time requirements, and wording or strength of recommendations as issues. Alexander et al. investigated why WHO guideline developers make “discordant recommendations,” i.e., strong recommendations based on low or very low certainty evidence. Their interviews with guideline panel members identified several challenges or barriers to GRADE application, including insufficient guidance on prioritizing outcomes and rating NRS, different professional backgrounds among panel members, and a political environment that favors strong recommendations [
      • Alexander P.E.
      • Gionfriddo M.R.
      • Li S.A.
      • Bero L.
      • Stoltzfus R.J.
      • Neumann I.
      • et al.
      A number of factors explain why WHO guideline developers make strong recommendations inconsistent with GRADE guidance.
      ]. In a further study by Alexander et al, interviews with GRADE methodologists also identified the political environment as a driver of “discordant” recommendations as well as issues with the management of conflicts of interest, urgent timescales, and the feasibility of implementing recommendations [
      • Alexander P.E.
      • Li S.-A.
      • Gionfriddo M.R.
      • Stoltzfus R.J.
      • Neumann I.
      • Brito J.P.
      • et al.
      Senior GRADE methodologists encounter challenges as part of WHO guideline development panels: an inductive content analysis.
      ]. Most recently, Gopinathan and Hoffman interviewed 35 senior WHO staff on their views about changes in WHO guideline development because its processes were reformed starting in 2007 [
      • Gopinathan U.
      • Hoffman S.J.
      Institutionalising an evidence-informed approach to guideline development: progress and challenges at the World Health Organization.
      ]. Among several themes identified in the interviews, challenges were identified relating to the assessment of NRS and qualitative studies, concerns which the authors note were subsequently addressed with the introduction of ROBINS-I [
      • Sterne J.A.C.
      • Hernan M.A.
      • Reeves B.C.
      • Savovic J.
      • Berkman N.D.
      • Viswanathan M.
      • et al.
      ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions.
      ] and GRADE-CERQual [
      • Lewin S.
      • Glenton C.
      • Munthe-Kaas H.
      • Carlsen B.
      • Colvin C.J.
      • Gulmezoglu M.
      • et al.
      Using qualitative evidence in decision making for health and social interventions: an approach to assess confidence in findings from qualitative evidence syntheses (GRADE-CERQual).
      ].
      Two reports [
      European Centre for Disease Prevention and Control
      Evidence-based methodologies for public health [Technical report].
      ,
      • Harder T.
      • Abu Sin M.
      • Bosch-Capblanch X.
      • Bruno C.
      • de Carvalho Gomes H.
      • Duclos P.
      • et al.
      Towards a framework for evaluating and grading evidence in public health.
      ] relate to guidelines or projects at the European Center for Disease Prevention and Control (ECDC). Within a review of evidence-based guideline methodology, the authors described issues with GRADE's terminology and treatment of NRS and noted an absence of guidance relating to economic and epidemiological models, studies of incidence and prevalence of disease, and microbiological studies. They further noted the need to integrate other aspects of decision-making with GRADE processes, such as prioritization of topics and selection of experts [
      European Centre for Disease Prevention and Control
      Evidence-based methodologies for public health [Technical report].
      ]. The ECDC Project on a Framework for Rating Evidence in Public Health subsequently reported the discussions and conclusions of an international expert meeting on evaluating and grading evidence on infectious disease prevention and control [
      • Harder T.
      • Abu Sin M.
      • Bosch-Capblanch X.
      • Bruno C.
      • de Carvalho Gomes H.
      • Duclos P.
      • et al.
      Towards a framework for evaluating and grading evidence in public health.
      ]. This report concluded that GRADE can be applied successfully to guideline questions concerning the incidence and prevalence of diseases and suggested that some types of nonrandomized evidence, such as a body of evidence from interrupted time series (ITS) designs, should be rated more highly in GRADE assessments. This is an area for which the GRADE Working Group has sought examples and operationalization since its inception but failed to receive or identify clear methodological suggestions that would allow rating ITS higher.
      Finally, Burford et al. (2012) summarized the benefits and challenges when applying GRADE to Cochrane public health systematic reviews [
      • Burford B.J.
      • Rehfuess E.
      • Schunemann H.J.
      • Akl E.A.
      • Waters E.
      • Armstrong R.
      • et al.
      Assessing evidence in public health: the added value of GRADE.
      ]. These challenges related to NRS and rating certainty of evidence, as well as what to do in the absence of a pooled effect size. The article addressed some misconceptions relating to initial ratings of randomized and observational studies and noted that concerns about low certainty of evidence are not unique to public health. The authors further noted that GRADE is particularly helpful in addressing certain issues commonly encountered when synthesizing public health evidence, including the importance of context and the heterogeneity that arises when combining findings from different public health programs and health systems.
      The studies identified in the scoping review range in publication date from 2011 to 2018. During this period, developments in GRADE methodology occurred that addressed many, if not most, of the challenges identified. For example, GRADE changed its terminology about the quality of evidence to be more flexible. Furthermore, GRADE had already altered the traditional evidence hierarchy by allowing a body of evidence from observational studies to be rated as high-quality evidence. In addition, articles have since addressed the use of GRADE in the absence of meta-analyses [
      • Thayer K.A.
      • Schünemann H.J.
      Using GRADE to respond to health questions with different levels of urgency.
      ,
      • Murad M.H.
      • Mustafa R.A.
      • Schünemann H.J.
      • Sultan S.
      • Santesso N.
      Rating the certainty in evidence in the absence of a single estimate of effect.
      ]. Overall, the existing literature is consistent in identifying certain challenges in using GRADE within public health systematic reviews and guidelines, particularly regarding diverse stakeholder perspectives within political environments, rating the certainty of nonrandomized evidence and wording or strength of recommendations. Most of the included studies make suggestions or recommendations to address these issues, and many solutions have been developed. However, additional solutions require further investigation, development, and testing to ensure a systematic approach that is coherent with existing GRADE guidance while also meeting the needs of stakeholders.

      4. Analysis: key challenges and solutions when applying GRADE guidance to public health topics

      To formulate an approach to addressing the issues identified through the scoping review, the GRADE Public Health Group organized the challenges into themes, mapped these to current GRADE Working Group activities, identified gaps, discussed and presented potential activities and outputs, and reached consensus on a program of work including publications, contributions to other project groups, and training. The project group also identified and discussed case studies through which the relevance and applicability of the five themes were explored and confirmed in a variety of public health contexts, including both systematic reviews and guidelines. Figure 1 summarizes the process of developing the scope and priorities of the GRADE Public Health Group, leading to a focus on five key challenges.
      Figure thumbnail gr1
      Fig. 1Process of developing the scope and priorities of the GRADE Public Health Group.
      The five key challenges are described narratively below. Table 1 summarizes the analysis of the challenges, gaps, and the project group response, including existing solutions, planned outputs, and other activities. Case study 1, Case study 2, Case study 3, Case study 4 present the case studies, which have been selected to represent a diversity of public health topics (social determinants of health, infectious diseases, organization of health care, and screening) drawn from both systematic reviews and guidelines (ten Cochrane reviews, one national public health guideline, and one pan-national guideline development program).
      Table 1Key challenges, examples, and proposed solutions in applying GRADE guidance to public health topics
      ChallengeExamplesSolutions
      1. Incorporating diverse perspectivesConvening guideline panels with diverse perspectives from different sectors, e.g., Czech national public health guideline on 1-day surgery (Box 3)

      Combining individual and population perspectives, e.g., malaria vector control reviews (Box 2) and European Breast Guidelines (Box 4)
      Forthcoming GRADE Public Health article,

      Addressing social determinants in public health systematic reviews and guidelines using GRADE

      Adapt GRADE training materials for nonhealth and policy audiences
      2. Selecting and prioritizing outcomesAddressing individual and population outcomes in the same guideline or review, particularly when benefits and harms differ depending on the perspective taken, e.g., Cochrane reviews and WHO guideline on malaria vector control (Box 2)

      Achieving consensus on critical outcomes across different sectors, e.g., health care, social work, community care, and insurers in Czech national public health guideline on 1-day surgery (Box 3)

      Health vs. nonhealth outcomes, e.g., in transport policy
      3. Interpreting outcomes and identifying a threshold for decision-makingOutcomes that have no defined thresholds, e.g., Cochrane reviews of unconditional cash transfers (Box 1)

      Small changes may have important health impacts at population level but not at individual level, e.g., taxation to address health externalities

      Different thresholds will have implications for modelling of the population health effects of policies and public health interventions
      Forthcoming GRADE Public Health article,

      Considerations when applying GRADE to population-level outcomes in public health systematic reviews and guidelines
      4. Assessing certainty of evidence from diverse sources, including nonrandomized studiesCochrane reviews of environmental interventions to reduce sugar-sweetened beverage consumption and interventions to reduce ambient air pollution found large differences in internal validity both within and across NRS

      Natural experiments may involve randomization or allocation that is “as good as” random, and may be the best possible evidence for a policy-related question but are treated as nonrandomized or observational studies and generally start with a rating of low-quality evidence
      Contribute worked examples from public health reviews and guidelines to other GRADE project groups, especially the GRADE NRS group

      Operationalize criteria for when interrupted time series may avoid typical selection and confounding bias and be rated as moderate certainty
      5. Addressing implications for decision makers, including concerns about conditional recommendationsMaking strong recommendations despite very low certainty evidence, e.g., unconditional cash transfers for disaster relief (Box 1)Collaboration with GRADE EtD group ongoing since May 2019

      4.1 Challenge 1: incorporating diverse perspectives

      Many health guidelines are intended for multidisciplinary audiences, including different health professions, health care managers, health policymakers, patients, and carers. The Health in All Policies approach is arguably where this becomes most challenging, for example, when considering the health impacts of migration policy [
      • Juarez S.P.
      • Honkaniemi H.
      • Dunlavy A.C.
      • Aldridge R.W.
      • Barreto M.L.
      • Katikireddi S.V.
      • et al.
      Effects of non-health-targeted policies on migrant health: a systematic review and meta-analysis.
      ]. In addition, to address the wider determinants of health, public health guidelines may be targeted at professionals, policymakers, and other stakeholders from nonhealth disciplines and sectors whose perspectives on health evidence may vary; indeed, sometimes the primary responsibility for financing and/or implementing public health interventions lies within these sectors. For example, the National Institute for Health and Care Excellence guideline on physical activity and the environment is aimed at local government and their contractors, employers, and community organizations responsible for public spaces, housing authorities, public transport planners and providers, and organizations that support people with limited mobility [
      National Institute for Health and Care Excellence
      Physical activity and the environment. NICE guideline [NG90].
      ]. The way that such varied audiences frame policy questions, the extent to which these audiences value health protection and improvement, and the priorities placed on various policy approaches may all differ substantially [
      • Stone D.
      Policy paradox: The art of political decision making.
      ], but need to be understood if the guideline is to achieve its aims. Furthermore, different professions and stakeholder groups may represent different “cultures of evidence” in which legislation, regulations, and other contextual factors may take priority over scientific research [
      • Lorenc T.
      • Tyner E.F.
      • Petticrew M.
      • Duffy S.
      • Martineau F.P.
      • Phillips G.
      • et al.
      Cultures of evidence across policy sectors: systematic review of qualitative evidence.
      ].

      4.2 Challenge 2: selecting and prioritizing outcomes

      Differences in perspectives and in cultures of evidence also translate into challenges in selecting relevant outcomes and agreeing which outcomes are critical to decision-making. For example, in transportation policy any recommendations about traffic management would ideally consider all relevant outcomes, including respiratory health, changes in physical activity levels, emergency admissions, and safety for vulnerable road users. However, transport policymakers might not view these outcomes as critical when weighed against traffic flow, commuting times, road traffic collisions, costs, provision of goods, or public opinion. Relatedly, challenges may arise when benefits are realized to a greater extent by communities than by individuals who receive treatment (e.g., immunizations which lead to herd immunity) [
      • Welch V.A.
      • Akl E.A.
      • Guyatt G.
      • Pottie K.
      • Eslava-Schmalbach J.
      • Ansari M.T.
      • et al.
      GRADE equity guidelines 1: considering health equity in GRADE guideline development: introduction and rationale.
      ]. A further challenge relates to the potential trade-off between population health and health equity, in which a population health benefit may be achieved at the expense of widening inequalities. For example, a public health intervention such as screening or wellness checks may succeed in improving a population's overall morbidity and mortality, but if the improved outcomes are predominantly achieved in a subgroup of higher socioeconomic status (SES), health inequality may inadvertently be increased by widening the difference between the outcomes of high versus low SES subgroups. The GRADE equity guidelines [
      • Akl E.A.
      • Welch V.
      • Pottie K.
      • Eslava-Schmalbach J.
      • Darzi A.
      • Sola I.
      • et al.
      GRADE equity guidelines 2: considering health equity in GRADE guideline development: equity extension of the guideline development checklist.
      ,
      • Pottie K.
      • Welch V.
      • Morton R.
      • Akl E.A.
      • Eslava-Schmalbach J.H.
      • Katikireddi V.
      • et al.
      GRADE equity guidelines 4: considering health equity in GRADE guideline development: evidence to decision process.
      ,
      • Welch V.A.
      • Akl E.A.
      • Guyatt G.
      • Pottie K.
      • Eslava-Schmalbach J.
      • Ansari M.T.
      • et al.
      GRADE equity guidelines 1: considering health equity in GRADE guideline development: introduction and rationale.
      ,
      • Welch V.A.
      • Akl E.A.
      • Pottie K.
      • Ansari M.T.
      • Briel M.
      • Christensen R.
      • et al.
      GRADE equity guidelines 3: considering health equity in GRADE guideline development: rating the certainty of synthesized evidence.
      ] are helpful in addressing this challenge. Finally, advancing health in the context of the Sustainable Development Goals implies that health benefits may need to be weighed against broader environmental, economic, and social impacts and their potential interactions [
      • Griffiths S.M.
      The Sustainable Development Goals: an agenda for us all.
      ,
      • Morton S.
      • Pencheon D.
      • Squires N.
      Sustainable Development Goals (SDGs), and their implementation: a national global framework for health, development and equity needs a systems approach at every level.
      ,
      • Rehfuess E.A.
      • Stratil J.M.
      • Scheel I.B.
      • Portela A.
      • Norris S.L.
      • Baltussen R.
      The WHO-INTEGRATE evidence to decision framework version 1.0: integrating WHO norms and values and a complexity perspective.
      ].

      4.3 Challenge 3: interpreting outcomes and identifying a threshold for decision-making

      The interpretation of evidence for a given outcome needs to be informed by knowledge of what change in that outcome would make a meaningful difference in a given context. A change in a health outcome that might be deemed too small to be of importance to a patient may be perceived differently from a whole population perspective, where small effects in many individuals may be magnified to differences that are important for public health interventions and policy [
      • Rose G.
      Strategy of prevention: lessons from cardiovascular disease.
      ], as can be seen, for example, in population-wide salt reduction programs [
      • Webster J.
      • Waqanivalu T.
      • Arcand J.
      • Trieu K.
      • Cappuccio F.P.
      • Appel L.J.
      • et al.
      Understanding the science that supports population-wide salt reduction programs.
      ]. Furthermore, many interventions at the population level are implemented for primarily nonhealth reasons but act on key determinants of health, for example, transport infrastructure, housing, and general taxation, and therefore may have profound health and health equity impacts. In these circumstances, the effect size of an intervention may not be as informative for decision-making as the expected direction of overall health impact (e.g., likely results in health gains or health harms). Relatedly, equipoise may not exist for the primary nonhealth outcome [
      • Petticrew M.
      • McKee M.
      • Lock K.
      • Green J.
      • Phillips G.
      In search of social equipoise.
      ]. These considerations of population-level impact, social determinants of health and inequalities, and lack of equipoise may all influence views on what constitutes an important difference and the required degree of precision to support decision-making.

      4.4 Challenge 4: assessing certainty of evidence from diverse sources, including NRS

      In public health, where randomized studies are less common and often infeasible in comparison with other areas of health, some types of NRS may provide greater certainty than others when investigating the health effects of policy or social interventions. For example, natural experiment studies may address selection bias and confounding through designs such as ITS or regression discontinuity, which may support stronger causal inference than other observational designs such as cohort and case-control studies [
      • Craig P.
      • Katikireddi S.V.
      • Leyland A.
      • Popham F.
      Natural experiments: an overview of methods, approaches, and contributions to public health intervention research.
      ]. Two recent Cochrane reviews, one on environmental interventions to reduce sugar-sweetened beverage consumption [
      • von Philipsborn P.
      • Stratil J.M.
      • Burns J.
      • Busert L.K.
      • Pfadenhauer L.M.
      • Polus S.
      • et al.
      Environmental interventions to reduce the consumption of sugar-sweetened beverages and their effects on health.
      ] and one on interventions to reduce ambient air pollution [
      • Burns J.
      • Boogaard H.
      • Polus S.
      • Pfadenhauer L.M.
      • Rohwer A.C.
      • van Erp A.M.
      • et al.
      Interventions to reduce ambient particulate matter air pollution and their effect on health.
      ] used nonrandomized intervention studies where no randomized studies were available; however, there were large differences in internal validity both within and across NRS. Existing GRADE guidance has noted that the potential exists for different NRS designs to provide moderate-quality evidence but that examples are yet to be identified [
      • Schünemann H.J.
      • Cuello C.
      • Akl E.A.
      • Mustafa R.A.
      • Meerpohl J.J.
      • Thayer K.
      • et al.
      GRADE guidelines: 18. How ROBINS-I and other tools to assess risk of bias in nonrandomized studies should be used to rate the certainty of a body of evidence.
      ]. An active research agenda exists in the area of NRS and evidence synthesis, including the activity of the GRADE NRS group, the development of ROBINS-I within the Cochrane Risk of Bias Methods Group, and further adaptations of ROBINS-I for different NRS designs [
      • Sterne J.A.C.
      • Hernan M.A.
      • Reeves B.C.
      • Savovic J.
      • Berkman N.D.
      • Viswanathan M.
      • et al.
      ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions.
      ,
      • Morgan R.L.
      • Thayer K.A.
      • Santesso N.
      • Holloway A.C.
      • Blain R.
      • Eftim S.E.
      • et al.
      A risk of bias instrument for non-randomized studies of exposures: a users' guide to its application in the context of GRADE.
      ]. Addressing this challenge by identifying relevant examples from public health policy and applying these new tools therefore seems a promising catalyst through which solutions to other challenges may also emerge.

      4.5 Challenge 5: addressing implications for decision makers, including concerns about conditional recommendations

      As demonstrated in the scoping review, the predominance of “weak” recommendations in public health guidelines has been of concern to guideline developers and policymakers. GRADE has addressed this issue by changing the terminology (weak recommendations are in public health primarily labeled conditional and the conditions should be specified). It is already recognized that public health produces several situations in which action may be strongly recommended despite low or very low certainty, including:
      • Life-threatening situations.
      • Uncertain benefit but certain harm.
      • Potential equivalence of effectiveness in which one option is clearly more or less risky or costly.
      • Potential for catastrophic harm [
        • Andrews J.C.
        • Schünemann H.J.
        • Oxman A.D.
        • Pottie K.
        • Meerpohl J.J.
        • Coello P.A.
        • et al.
        GRADE guidelines: 15. Going from evidence to recommendation—determinants of a recommendation's direction and strength.
        ].
      Early strong recommendations in the context of very low certainty evidence emphasized that GRADE had recognized this problem [
      • Schunemann H.J.
      • Hill S.R.
      • Kakad M.
      • Bellamy R.
      • Uyeki T.M.
      • Hayden F.G.
      • et al.
      WHO rapid advice guidelines for pharmacological management of sporadic human infection with avian influenza A (H5N1) virus.
      ]. However, examples may still be encountered, such as in fiscal policy, where economists might strongly recommend austerity policies while public health might produce a weak recommendation in terms of health outcomes. Grading recommendations as “conditional” rather than “weak” may partially address this issue, but guideline developers may also need to better contextualize and ground such recommendations in “what else we might know.” A Bayesian decision theory approach has been proposed in the context of this problem [
      • Fischer A.J.
      • Threlfall A.
      • Meah S.
      • Cookson R.
      • Rutter H.
      • Kelly M.P.
      The appraisal of public health interventions: an overview.
      ,
      • Hultcrantz M.
      • Rind D.
      • Akl E.A.
      • Treweek S.
      • Mustafa R.A.
      • Iorio A.
      • et al.
      The GRADE Working Group clarifies the construct of certainty of evidence.
      ,
      • Threlfall A.G.
      • Meah S.
      • Fischer A.J.
      • Cookson R.
      • Rutter H.
      • Kelly M.P.
      The appraisal of public health interventions: the use of theory.
      ] and could be further explored. A key challenge for GRADE in public health is therefore to identify how to reconcile the tension between the methodologically correct presentation of evidence and recommendations as per GRADE and the implications of strong vs. conditional recommendations from the perspective of decision makers in political environments.
      [Case study 1, Case study 2, Case study 3, Case study 4: Case studies: four examples of challenges of applying GRADE in public health systematic reviews and guidelines].

      5. Discussion and conclusion

      The views sourced from a participatory workshop, a scoping review, and four case studies suggest that key public health-related challenges in application of GRADE methodology include the incorporation of stakeholder perspectives from outside the health sector; agreement on the interpretation and prioritization of diverse outcomes; definition and interpretation of thresholds for population health and nonhealth outcomes; assessing the certainty of evidence from diverse sources and NRS designs; and addressing implications for decision makers, including the presentation of strength of recommendations to ensure utility and acceptability for stakeholders.
      These challenges may be categorized into three different types, as per the different types of solutions that may be proposed. The first type of solution is to identify the training needs of public health guideline developers (and their stakeholders) in understanding and using the GRADE concept, i.e., as per existing guidance. As different disciplines or “cultures of evidence” come together in public health, it will take time to speak a common (GRADE) language. The second type of solution has to do with applying existing GRADE guidance, including current developments, to the public health context. As identified within the scoping review, most of the existing GRADE guidance, both on rating certainty of the evidence and moving from evidence to decision, is already applicable to the public health context but would benefit from translation and examples, so that its relevance to public health practice and policy can be more readily recognized and a population perspective can be consistently reflected in all steps. The third type of solution is to advance the GRADE methodology where existing guidance falls short, i.e., to develop new GRADE guidance where necessary and appropriate.
      The GRADE Public Health Group has considered the challenges in the context of other GRADE outputs and project groups, as well as other ongoing research in guideline methodology [
      • Rehfuess E.A.
      • Stratil J.M.
      • Scheel I.B.
      • Portela A.
      • Norris S.L.
      • Baltussen R.
      The WHO-INTEGRATE evidence to decision framework version 1.0: integrating WHO norms and values and a complexity perspective.
      ,
      • Montgomery P.
      • Movsisyan A.
      • Grant S.P.
      • Macdonald G.
      • Rehfuess E.A.
      Considerations of complexity in rating certainty of evidence in systematic reviews: a primer on using the GRADE approach in global health.
      ], with the aim of avoiding unnecessary duplication of effort. The solutions proposed by the group include the following package of work:
      • Further GRADE concept articles or guidance to address social determinants in public health systematic reviews and guidelines using GRADE, considerations when applying GRADE to population-level outcomes and criteria to assess certainty of interrupted time series.
      • The ongoing development and dissemination of detailed examples of the application of GRADE to public health topics in coordination with other GRADE project groups, including the NRS and EtD groups.
      • Adaptation of GRADE training materials for public health and public policy audiences.
      By providing a single, carefully documented, rigorous, and transparent method of evaluating evidence, GRADE serves to increase the trustworthiness, implementation, and adaptability of systematic reviews and guidelines, while also helping to reduce research waste. It is therefore our aim to address and mitigate any barriers to using GRADE in public health and, through the work of the GRADE Public Health group, to build bridges and increase mutual understanding across the wide range of policy and research areas that intersect in the population health context.

      CRediT authorship contribution statement

      Michele Hilton Boon: Conceptualization, Methodology, Writing - original draft, Writing - review & editing. Hilary Thomson: Conceptualization, Writing - review & editing. Beth Shaw: Conceptualization, Investigation, Writing - review & editing. Elie A. Akl: Conceptualization, Methodology, Writing - review & editing. Stefan K. Lhachimi: Writing - original draft, Writing - review & editing. Jesús López-Alcalde: Writing - original draft, Writing - review & editing. Miloslav Klugar: Writing - original draft, Writing - review & editing. Leslie Choi: Writing - original draft, Writing - review & editing. Zuleika Saz-Parkinson: Writing - original draft, Writing - review & editing. Reem A. Mustafa: Methodology, Writing - review & editing. Miranda W. Langendam: Methodology, Writing - review & editing. Olivia Crane: Writing - review & editing. Rebecca L. Morgan: Writing - review & editing. Eva Rehfuess: Writing - review & editing. Bradley C. Johnston: Writing - review & editing. Lee Yee Chong: Writing - review & editing. Gordon H. Guyatt: Methodology, Writing - review & editing. Holger J. Schünemann: Conceptualization, Methodology, Writing - review & editing. Srinivasa Vittal Katikireddi: Conceptualization, Methodology, Investigation, Writing - review & editing.

      Acknowledgments

      The authors gratefully acknowledge the participants at the initial GRADE Public Health workshop at the Global Evidence Summit in Cape Town, South Africa, for sharing their experiences and views. The authors also thank the following members of the GRADE Working Group for comments on previous drafts of this article: Paul Garner, Thomas Harder, Rachel Kettle, Hugh McGuire, Paul Montgomery, Susan Norris, Juan Pablo Peña-Rosas, Nancy Santesso, Ein-Soon Shin, Toni Tan, Gregory Traversy, and Vivian Welch. Article history: Concepts and cases in development were presented on slides at GRADE Working Group meetings in Rome (2017), Manchester (2018), and Hamilton (2019).

      References

        • Alonso-Coello P.
        • Schünemann H.J.
        • Moberg J.
        • Brignardello-Petersen R.
        • Akl E.A.
        • Davoli M.
        • et al.
        GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 1: Introduction.
        BMJ. 2016; : 353
        • Moberg J.
        • Oxman A.D.
        • Rosenbaum S.
        • Schünemann H.J.
        • Guyatt G.
        • Flottorp S.
        • et al.
        The GRADE Evidence to Decision (EtD) framework for health system and public health decisions.
        Health Res Policy Syst. 2018; 16: 45
        • Schünemann H.J.
        • Mustafa R.
        • Brozek J.
        • Santesso N.
        • Alonso-Coello P.
        • Guyatt G.
        • et al.
        GRADE Guidelines: 16. GRADE evidence to decision frameworks for tests in clinical practice and public health.
        J Clin Epidemiol. 2016; 76: 89-98
        • Acheson D.
        Public Health in England: The Report of the Committee of Inquiry into the Future Development to the Public Health Function.
        The Stationery Office, London1988
        • Griffiths S.
        • Jewell T.
        • Donnelly P.
        Public health in practice: the three domains of public health.
        Public Health. 2005; 119: 907-913
        • Benach J.
        • Malmusi D.
        • Yasui Y.
        • Martínez J.M.
        A new typology of policies to tackle health inequalities and scenarios of impact based on Rose's population approach.
        J Epidemiol Community Health. 2013; 67: 286-291
        • Ahmed F.
        • Temte J.L.
        • Campos-Outcalt D.
        • Schünemann H.J.
        • Recommendation A.E.B.
        • Group A.E.B.R.W.
        Methods for developing evidence-based recommendations by the advisory committee on immunization practices (ACIP) of the U.S. Centers for disease control and prevention (CDC).
        Vaccine. 2011; 29: 9171-9176
        • Dahm P.
        • Oxman A.D.
        • Djulbegovic B.
        • Guyatt G.H.
        • Murad M.H.
        • Amato L.
        • et al.
        Stakeholders apply the GRADE evidence-to-decision framework to facilitate coverage decisions.
        J Clin Epidemiol. 2017; 86: 129-139
        • Schunemann H.J.
        • Hill S.R.
        • Kakad M.
        • Vist G.E.
        • Bellamy R.
        • Stockman L.
        • et al.
        Transparent development of the WHO rapid advice guidelines.
        PLoS Med. 2007; 4: 786-793
        • Schunemann H.J.
        • Lerda D.
        • Dimitrova N.
        • Alonso-Coello P.
        • Grawingholt A.
        • Quinn C.
        • et al.
        Methods for development of the European commission initiative on Breast cancer guidelines recommendations in the era of guideline transparency.
        Ann Intern Med. 2019; 171: 273-280
        • Thornton J.
        • Alderson P.
        • Tan T.
        • Turner C.
        • Latchem S.
        • Shaw E.
        • et al.
        Introducing GRADE across the NICE clinical guideline program.
        J Clin Epidemiol. 2013; 66: 124-131
        • Akl E.A.
        • Kennedy C.
        • Konda K.
        • Caceres C.F.
        • Horvath T.
        • Ayala G.
        • et al.
        Using GRADE methodology for the development of public health guidelines for the prevention and treatment of HIV and other STIs among men who have sex with men and transgender people.
        BMC Public Health. 2012; 12: 386
        • Alexander P.E.
        • Gionfriddo M.R.
        • Li S.A.
        • Bero L.
        • Stoltzfus R.J.
        • Neumann I.
        • et al.
        A number of factors explain why WHO guideline developers make strong recommendations inconsistent with GRADE guidance.
        J Clin Epidemiol. 2016; 70: 111-122
        • Rehfuess E.A.
        • Akl E.A.
        Current experience with applying the GRADE approach to public health interventions: an empirical study.
        BMC Public Health. 2013; 13: 9
        • Dedios M.C.
        • Esperato A.
        • De-Regil L.M.
        • Pena-Rosas J.P.
        • Norris S.L.
        Improving the adaptability of WHO evidence-informed guidelines for nutrition actions: results of a mixed methods evaluation.
        Implementation Sci. 2017; 12: 39
        • European Centre for Disease Prevention and Control
        Evidence-based methodologies for public health [Technical report].
        (Available at)
        • Harder T.
        • Abu Sin M.
        • Bosch-Capblanch X.
        • Bruno C.
        • de Carvalho Gomes H.
        • Duclos P.
        • et al.
        Towards a framework for evaluating and grading evidence in public health.
        Health Policy. 2015; 119: 732-736
        • Thayer K.A.
        • Schünemann H.J.
        Using GRADE to respond to health questions with different levels of urgency.
        Environ Int. 2016; 92-93: 585-589
        • Florez I.D.
        • Morgan R.L.
        • Falavigna M.
        • Kowalski S.C.
        • Zhang Y.
        • Etxeandia-Ikobaltzeta I.
        • et al.
        Development of rapid guidelines: 2. A qualitative study with WHO guideline developers.
        Health Res Policy Syst. 2018; 16: 62-69
        • Schünemann H.
        • Hill S.
        • Guyatt G.
        • Akl E.A.
        • Ahmed F.
        The GRADE approach and Bradford Hill's criteria for causation.
        J Epidemiol Community Health. 2011; 65: 392-395
        • Akl E.A.
        • Welch V.
        • Pottie K.
        • Eslava-Schmalbach J.
        • Darzi A.
        • Sola I.
        • et al.
        GRADE equity guidelines 2: considering health equity in GRADE guideline development: equity extension of the guideline development checklist.
        J Clin Epidemiol. 2017; 90: 68-75
        • Cuello-Garcia C.A.
        • Morgan R.L.
        • Brozek J.
        • Santesso N.
        • Verbeek J.
        • Thayer K.
        • et al.
        A scoping review and survey provides the rationale, perceptions, and preferences for the integration of randomized and nonrandomized studies in evidence syntheses and GRADE assessments.
        J Clin Epidemiol. 2018; 98: 33-40
        • Morgan R.L.
        • Thayer K.A.
        • Bero L.
        • Bruce N.
        • Falck-Ytter Y.
        • Ghersi D.
        • et al.
        GRADE: assessing the quality of evidence in environmental and occupational health.
        Environ Int. 2016; 92-93: 611-616
        • Pottie K.
        • Welch V.
        • Morton R.
        • Akl E.A.
        • Eslava-Schmalbach J.H.
        • Katikireddi V.
        • et al.
        GRADE equity guidelines 4: considering health equity in GRADE guideline development: evidence to decision process.
        J Clin Epidemiol. 2017; 90: 84-91
        • Schünemann H.J.
        • Cuello C.
        • Akl E.A.
        • Mustafa R.A.
        • Meerpohl J.J.
        • Thayer K.
        • et al.
        GRADE guidelines: 18. How ROBINS-I and other tools to assess risk of bias in nonrandomized studies should be used to rate the certainty of a body of evidence.
        J Clin Epidemiol. 2019; 111: 105-114
        • Welch V.A.
        • Akl E.A.
        • Guyatt G.
        • Pottie K.
        • Eslava-Schmalbach J.
        • Ansari M.T.
        • et al.
        GRADE equity guidelines 1: considering health equity in GRADE guideline development: introduction and rationale.
        J Clin Epidemiol. 2017; 90: 59-67
        • Welch V.A.
        • Akl E.A.
        • Pottie K.
        • Ansari M.T.
        • Briel M.
        • Christensen R.
        • et al.
        GRADE equity guidelines 3: considering health equity in GRADE guideline development: rating the certainty of synthesized evidence.
        J Clin Epidemiol. 2017; 90: 76-83
        • Alexander P.E.
        • Li S.-A.
        • Gionfriddo M.R.
        • Stoltzfus R.J.
        • Neumann I.
        • Brito J.P.
        • et al.
        Senior GRADE methodologists encounter challenges as part of WHO guideline development panels: an inductive content analysis.
        J Clin Epidemiol. 2016; 70: 123-128
        • Burford B.J.
        • Rehfuess E.
        • Schunemann H.J.
        • Akl E.A.
        • Waters E.
        • Armstrong R.
        • et al.
        Assessing evidence in public health: the added value of GRADE.
        J Public Health. 2012; 34: 631-635
        • Gopinathan U.
        • Hoffman S.J.
        Institutionalising an evidence-informed approach to guideline development: progress and challenges at the World Health Organization.
        BMJ Glob Health. 2018; 3: e000716
        • Harder T.
        • Abu Sin M.
        • Bosch-Capblanch X.
        • Bruno C.
        • de Carvalho Gomes H.
        • Duclos P.
        • et al.
        Towards a framework for evaluating and grading evidence in public health.
        Health Policy. 2015; 119: 732-736
        • Sterne J.A.C.
        • Hernan M.A.
        • Reeves B.C.
        • Savovic J.
        • Berkman N.D.
        • Viswanathan M.
        • et al.
        ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions.
        BMJ. 2016; 355: i4919
        • Lewin S.
        • Glenton C.
        • Munthe-Kaas H.
        • Carlsen B.
        • Colvin C.J.
        • Gulmezoglu M.
        • et al.
        Using qualitative evidence in decision making for health and social interventions: an approach to assess confidence in findings from qualitative evidence syntheses (GRADE-CERQual).
        PLoS Med. 2015; 12: e1001895
        • Murad M.H.
        • Mustafa R.A.
        • Schünemann H.J.
        • Sultan S.
        • Santesso N.
        Rating the certainty in evidence in the absence of a single estimate of effect.
        Evid Based Med. 2017; 22: 85-87
        • Juarez S.P.
        • Honkaniemi H.
        • Dunlavy A.C.
        • Aldridge R.W.
        • Barreto M.L.
        • Katikireddi S.V.
        • et al.
        Effects of non-health-targeted policies on migrant health: a systematic review and meta-analysis.
        Lancet Glob Health. 2019; 7: e420-e435
        • National Institute for Health and Care Excellence
        Physical activity and the environment. NICE guideline [NG90].
        (Available at)
        https://www.nice.org.uk/guidance/ng90
        Date: 2018
        Date accessed: February 6, 2020
        • Stone D.
        Policy paradox: The art of political decision making.
        WW Norton, New York1997
        • Lorenc T.
        • Tyner E.F.
        • Petticrew M.
        • Duffy S.
        • Martineau F.P.
        • Phillips G.
        • et al.
        Cultures of evidence across policy sectors: systematic review of qualitative evidence.
        Eur J Public Health. 2014; 24: 1041-1047
        • Griffiths S.M.
        The Sustainable Development Goals: an agenda for us all.
        Perspect Public Health. 2019; 139: 224-225
        • Morton S.
        • Pencheon D.
        • Squires N.
        Sustainable Development Goals (SDGs), and their implementation: a national global framework for health, development and equity needs a systems approach at every level.
        Br Med Bull. 2017; 124: 81-90
        • Rehfuess E.A.
        • Stratil J.M.
        • Scheel I.B.
        • Portela A.
        • Norris S.L.
        • Baltussen R.
        The WHO-INTEGRATE evidence to decision framework version 1.0: integrating WHO norms and values and a complexity perspective.
        BMJ Glob Health. 2019; 4: e000844
        • Rose G.
        Strategy of prevention: lessons from cardiovascular disease.
        BMJ. 1981; 282: 1847
        • Webster J.
        • Waqanivalu T.
        • Arcand J.
        • Trieu K.
        • Cappuccio F.P.
        • Appel L.J.
        • et al.
        Understanding the science that supports population-wide salt reduction programs.
        J Clin Hypertens (Greenwich, Conn). 2017; 19: 569-576
        • Petticrew M.
        • McKee M.
        • Lock K.
        • Green J.
        • Phillips G.
        In search of social equipoise.
        BMJ. 2013; 347: f4016
        • Craig P.
        • Katikireddi S.V.
        • Leyland A.
        • Popham F.
        Natural experiments: an overview of methods, approaches, and contributions to public health intervention research.
        Annu Rev Public Health. 2017; 38: 39-56
        • von Philipsborn P.
        • Stratil J.M.
        • Burns J.
        • Busert L.K.
        • Pfadenhauer L.M.
        • Polus S.
        • et al.
        Environmental interventions to reduce the consumption of sugar-sweetened beverages and their effects on health.
        Cochrane Database Syst Rev. 2019; 6: Cd012292
        • Burns J.
        • Boogaard H.
        • Polus S.
        • Pfadenhauer L.M.
        • Rohwer A.C.
        • van Erp A.M.
        • et al.
        Interventions to reduce ambient particulate matter air pollution and their effect on health.
        Cochrane Database Syst Rev. 2019; 5: Cd010919
        • Morgan R.L.
        • Thayer K.A.
        • Santesso N.
        • Holloway A.C.
        • Blain R.
        • Eftim S.E.
        • et al.
        A risk of bias instrument for non-randomized studies of exposures: a users' guide to its application in the context of GRADE.
        Environ Int. 2019; 122: 168-184
        • Andrews J.C.
        • Schünemann H.J.
        • Oxman A.D.
        • Pottie K.
        • Meerpohl J.J.
        • Coello P.A.
        • et al.
        GRADE guidelines: 15. Going from evidence to recommendation—determinants of a recommendation's direction and strength.
        J Clin Epidemiol. 2013; 66: 726-735
        • Schunemann H.J.
        • Hill S.R.
        • Kakad M.
        • Bellamy R.
        • Uyeki T.M.
        • Hayden F.G.
        • et al.
        WHO rapid advice guidelines for pharmacological management of sporadic human infection with avian influenza A (H5N1) virus.
        Lancet Infect Dis. 2007; 7: 21-31
        • Fischer A.J.
        • Threlfall A.
        • Meah S.
        • Cookson R.
        • Rutter H.
        • Kelly M.P.
        The appraisal of public health interventions: an overview.
        J Public Health. 2013; 35: 488-494
        • Hultcrantz M.
        • Rind D.
        • Akl E.A.
        • Treweek S.
        • Mustafa R.A.
        • Iorio A.
        • et al.
        The GRADE Working Group clarifies the construct of certainty of evidence.
        J Clin Epidemiol. 2017; 87: 4-13
        • Threlfall A.G.
        • Meah S.
        • Fischer A.J.
        • Cookson R.
        • Rutter H.
        • Kelly M.P.
        The appraisal of public health interventions: the use of theory.
        J Public Health. 2015; 37: 166-171
        • Pega F.
        • Liu S.Y.
        • Walter S.
        • Lhachimi S.K.
        Unconditional cash transfers for assistance in humanitarian disasters: effect on use of health services and health outcomes in low- and middle-income countries.
        Cochrane Database Syst Rev. 2015; : CD011247
        • Pega F.
        • Liu S.Y.
        • Walter S.
        • Pabayo R.
        • Saith R.
        • Lhachimi S.K.
        Unconditional cash transfers for reducing poverty and vulnerabilities: effect on use of health services and health outcomes in low- and middle-income countries.
        Cochrane Database Syst Rev. 2017; : CD011135
        • World Health Organization
        Guidelines for malaria vector control.
        (Available at)
        • Choi L.
        • Pryce J.
        • Garner P.
        The combination of indoor residual spraying with insecticide-treated nets versus insecticide-treated nets alone for preventing malaria.
        Cochrane Database Syst Rev. 2017; : CD012688
        • Choi L.
        • Wilson A.
        Larviciding to control malaria.
        Cochrane Database Syst Rev. 2017; : CD012736
        • Gleave K.
        • Lissenden N.
        • Richardson M.
        • Choi L.
        • Ranson H.
        Piperonyl butoxide (PBO) combined with pyrethroids in insecticide-treated nets to prevent malaria in Africa.
        Cochrane Database Syst Rev. 2018; 11: CD012776
        • Maia M.F.
        • Kliner M.
        • Richardson M.
        • Lengeler C.
        • Moore S.J.
        Mosquito repellents for malaria prevention.
        Cochrane Database Syst Rev. 2018; 2: Cd011595
        • Pluess B.
        • Tanser F.C.
        • Lengeler C.
        • Sharp B.L.
        Indoor residual spraying for preventing malaria.
        Cochrane Database Syst Rev. 2010; : Cd006657
        • Pryce J.
        • Choi L.
        • Richardson M.
        • Malone D.
        Insecticide space spraying for preventing malaria transmission.
        Cochrane Database Syst Rev. 2018; 11: Cd012689
        • Pryce J.
        • Richardson M.
        • Lengeler C.
        Insecticide-treated nets for preventing malaria.
        Cochrane Database Syst Rev. 2018; : CD000363
        • Walshe D.P.
        • Garner P.
        • Adeel A.A.
        • Pyke G.H.
        • Burkot T.R.
        Larvivorous fish for preventing malaria transmission.
        Cochrane Database Syst Rev. 2017; : CD008090
        • European Commission
        JRC in brief 2019.
        (Available at)
        https://ec.europa.eu/jrc/en/about/jrc-in-brief
        Date accessed: February 6, 2020
        • European Commission
        European Breast Guidelines 2019.
        (Available at)
        • European Commission
        Methods of the European guidelines for breast cancer screening and diagnosis 2016.
        (Available at)
        • European Commission
        Scope of the European guidelines for breast cancer screening and diagnosis.
        (Available at)
        • Schünemann H.J.
        • Lerda D.
        • Quinn C.
        • Follmann M.
        • Alonso-Coello P.
        • Rossi P.G.
        • et al.
        Breast cancer screening and diagnosis: a synopsis of the European Breast guidelines.
        Ann Intern Med. 2020; 172: 46-56
        • Montgomery P.
        • Movsisyan A.
        • Grant S.P.
        • Macdonald G.
        • Rehfuess E.A.
        Considerations of complexity in rating certainty of evidence in systematic reviews: a primer on using the GRADE approach in global health.
        BMJ Glob Health. 2019; 4: e000848