Advertisement

Guideline uptake is influenced by six implementability domains for creating and communicating guidelines: a realist review

  • Monika Kastner
    Correspondence
    Corresponding author. Tel.: 416-864-6060 x 77367; fax: 416-864-5805.
    Affiliations
    Li Ka Shing Knowledge Institute (LKSKI) of St. Michael's Hospital, Knowledge Translation Program. 209 Victoria Street, Toronto, Ontario, Canada, M5B 1W8
    Search for articles by this author
  • Onil Bhattacharyya
    Affiliations
    Li Ka Shing Knowledge Institute (LKSKI) of St. Michael's Hospital, Knowledge Translation Program. 209 Victoria Street, Toronto, Ontario, Canada, M5B 1W8

    Keenan Research Centre of LKSKI of St. Michael's Hospital, 209 Victoria Street, Toronto, Ontario, Canada, M5B 1W8
    Search for articles by this author
  • Leigh Hayden
    Affiliations
    Li Ka Shing Knowledge Institute (LKSKI) of St. Michael's Hospital, Knowledge Translation Program. 209 Victoria Street, Toronto, Ontario, Canada, M5B 1W8
    Search for articles by this author
  • Julie Makarski
    Affiliations
    Department of Oncology, Escarpment Cancer Research Institute, McMaster University, 699 Concession Street, Hamilton, Ontario, L8V 5C2
    Search for articles by this author
  • Elizabeth Estey
    Affiliations
    Strategic Policy, Planning & Initiatives, Health Services, Region of Peel, 10 Peel Centre Drive, 2nd Floor, Brampton, Ontario, Canada, L6T 4B9
    Search for articles by this author
  • Lisa Durocher
    Affiliations
    Department of Oncology, Escarpment Cancer Research Institute, McMaster University, 699 Concession Street, Hamilton, Ontario, L8V 5C2
    Search for articles by this author
  • Ananda Chatterjee
    Affiliations
    Li Ka Shing Knowledge Institute (LKSKI) of St. Michael's Hospital, Knowledge Translation Program. 209 Victoria Street, Toronto, Ontario, Canada, M5B 1W8
    Search for articles by this author
  • Laure Perrier
    Affiliations
    Li Ka Shing Knowledge Institute (LKSKI) of St. Michael's Hospital, Knowledge Translation Program. 209 Victoria Street, Toronto, Ontario, Canada, M5B 1W8

    Continuing Education and Professional Development, Faculty of Medicine, University of Toronto, 500 University Avenue, Suite 650, Toronto, Ontario, Canada, M5G 1V7
    Search for articles by this author
  • Ian D. Graham
    Affiliations
    Department of Epidemiology and Community Medicine, Ottawa Hospital Research Institute, University of Ottawa, 725 Parkdale Avenue, Ottawa, Ontario, Canada K1Y 4E9
    Search for articles by this author
  • Sharon E. Straus
    Affiliations
    Li Ka Shing Knowledge Institute (LKSKI) of St. Michael's Hospital, Knowledge Translation Program. 209 Victoria Street, Toronto, Ontario, Canada, M5B 1W8

    Keenan Research Centre of LKSKI of St. Michael's Hospital, 209 Victoria Street, Toronto, Ontario, Canada, M5B 1W8

    Faculty of Medicine, University of Toronto, 563 Spadina Crescent, Toronto, Ontario, Canada, M5S 2J7
    Search for articles by this author
  • Merrick Zwarenstein
    Affiliations
    Department of Family Medicine, Schulich School of Medicine and Dentistry, Western University, Clinical Skills Building, London, Ontario, Canada, N6A 5C1
    Search for articles by this author
  • Melissa Brouwers
    Affiliations
    Department of Oncology, Escarpment Cancer Research Institute, McMaster University, 699 Concession Street, Hamilton, Ontario, L8V 5C2
    Search for articles by this author
Open AccessPublished:January 14, 2015DOI:https://doi.org/10.1016/j.jclinepi.2014.12.013

      Abstract

      Objectives

      To identify factors associated with the implementability of clinical practice guidelines (CPGs) and to determine what characteristics improve their uptake.

      Study Design and Setting

      We conducted a realist review, which involved searching multiple sources (eg, databases, experts) to determine what about guideline implementability works, for whom, and under what circumstances. Two sets of reviewers independently screened abstracts and extracted data from 278 included studies. Analysis involved the development of a codebook of definitions, validation of data, and development of hierarchical narratives to explain guideline implementability.

      Results

      We found that guideline implementability is associated with two broad goals in guideline development: (1) creation of guideline content, which involves addressing the domains of stakeholder involvement in CPGs, evidence synthesis, considered judgment (eg, clinical applicability), and implementation feasibility and (2) the effective communication of this content, which involves domains related to fine-tuning the CPG's message (using simple, clear, and persuasive language) and format.

      Conclusion

      Our work represents a comprehensive and interdisciplinary effort toward better understanding, which attributes of guidelines have the potential to improve uptake in clinical practice. We also created codebooks and narratives of key concepts, which can be used to create tools for developing better guidelines to promote better care.

      Keywords

      What is new?

        Key finding

      • We identified six domains of guideline implementability hypothesized to affect uptake of recommendations within two broad categories: (1) the “creation” of guideline content, which involves addressing stakeholder involvement in guidelines, evidence synthesis, considered judgment, and implementation feasibility and (2) the “communication” of this content by fine-tuning the guideline's message and format.

        What this adds to what was known?

      • Building on the work of others, this is the first systematic review to investigate guideline implementability from a comprehensive and multidisciplinary perspective (ie, psychology, management, and human factors engineering).
      • Our analysis moves beyond the medical and implementation science literature (which tend to focus on the creation of content) and incorporates other disciplinary content, which expands understanding of the relevant factors, particularly those related to the communication of content.

        What is the implication and what should change now?

      • Guidelines summarize clinical evidence to inform clinicians' decision making, but how they are developed and written influences how often they are used.
      • Our work represents an important, comprehensive, and interdisciplinary effort toward better understanding of which attributes of guidelines have the potential to improve uptake in clinical practice.
      • We created narratives of key concepts, which can be used to develop tools to build better guidelines and promote better care.

      1. Introduction

      Recent efforts in the clinical practice guideline (CPG) research enterprise have focused on identifying factors that can be targeted to increase the uptake of recommendations to improve patient outcomes and strengthen delivery systems. These efforts have included investigations into extrinsic (changing the practice setting to facilitate recommendation use) and intrinsic approaches (changing the guideline itself) to improving CPGs. Both approaches are needed, but given the costs and context dependence of extrinsic approaches, investigation of intrinsic approaches may lead to solutions at minimal cost and that may be more broadly applicable and feasible. Shiffman et al. [
      • Shiffman R.
      • Dixon J.
      • Brandt C.
      • Essaihi A.
      • Hsiao A.
      • Michel G.
      • et al.
      The Guideline Implementability Appraisal (GLIA): development of an instrument to identify obstacles to guideline implementation.
      ] have referred to “implementability” as the perceived characteristics of guidelines that predict the relative ease of their implementation. Gagliardi et al. [
      • Gagliardi A.
      • Brouwers M.C.
      • Palda V.A.
      • Lemieux-Charles L.
      • Grimshaw J.M.
      How can we improve guideline use? A conceptual framework of implementability.
      ] developed a framework of guideline implementability, which was tested with 20 different CPGs, and found that elements related to guideline format (eg, guidelines that provide summary versions) and content (eg, clinical considerations to individualize recommendations) provided the best opportunities to modify CPGs for improved uptake. Grol et al. [
      • Grol R.
      • Dalhuijsen J.
      • Thomas S.
      • Veld C.
      • Rutten G.
      • Mokkink H.
      Attributes of clinical guidelines that influence use of guidelines in general practice: observational study.
      ] found that guidelines that are compatible with existing norms among the target group for implementation and those that do not demand too much change to existing routines, extra resources, or acquisition of new knowledge and skills were used more. Michie et al. [
      • Michie S.
      • Johnston M.
      Changing clinical behaviour by making guidelines specific.
      ,
      • Michie S.
      • Lester K.
      Words that matter: increasing the implementation of clinical guidelines.
      ] suggest that clarity, specificity of behavioral instructions, and specific plans are important to get physicians to follow guidelines, but these factors have largely been overlooked.
      Together, these contributions have served as an excellent foundation and have enabled a dialog within the health services research community about guideline implementability. However, in developing this concept and understanding it further, the field has mostly focused their efforts on the medical literature. In doing so, it has largely ignored relevant paradigms from other disciplines. For example, in the areas of social, cognitive, and health psychology, there have been decades of research developing theory and models to explain behavior change, persuasion, motivation, and communication styles [
      • Rogers E.
      Diffusion of innovations.
      ,
      • Rey G.
      • Buchwald F.
      The expertise reversal effect: cognitive load and motivational explanations.
      ,
      • Wang A.
      • Dowding T.
      Effects of visual priming on improving web disclosure to investors.
      ].
      There has been much discussion around improving the rigor of guidelines [
      • Tricoci P.
      • Allen J.M.
      • Kramer J.M.
      • Califf R.M.
      • Smith Jr., S.C.
      Scientific evidence underlying the ACC/AHA clinical practice guidelines.
      ], and consequently, there are a number of tools to increase how the evidence supporting guideline recommendations is synthesized, analyzed, and presented. These include AGREE II [
      • Brouwers M.C.
      • Kho M.
      • Browman G.P.
      • Burgers J.S.
      • Cluzeau F.
      • Feder G.
      • et al.
      AGREE II: advancing guideline development, reporting and evaluation in health care.
      ], GRADE [
      • Atkins D.
      • Best D.
      • Briss P.A.
      • Eccles M.
      • Falck-Ytter Y.
      • Flottorp S.
      • et al.
      GRADE Working Group
      Grading quality of evidence and strength of recommendations.
      ], GLIA [
      • Shiffman R.
      • Dixon J.
      • Brandt C.
      • Essaihi A.
      • Hsiao A.
      • Michel G.
      • et al.
      The Guideline Implementability Appraisal (GLIA): development of an instrument to identify obstacles to guideline implementation.
      ], ADAPTE [
      • Fervers B.
      • Burgers J.
      • Haugh M.C.
      • Latreille J.
      • Milka-Cabanne N.
      • Paquet L.
      • et al.
      Adaptation of clinical guidelines: literature review and proposition for a framework and procedure.
      ], and CAN-IMPLEMENT [
      • Harrison M.B.
      • Graham I.
      • van den Hoek J.
      • Dogherty E.J.
      • Carley M.E.
      • Angus V.
      Guideline adaptation and implementation planning: a prospective observational study.
      ]. However, these tools are mostly informed by the medical literature [
      • Shiffman R.
      • Dixon J.
      • Brandt C.
      • Essaihi A.
      • Hsiao A.
      • Michel G.
      • et al.
      The Guideline Implementability Appraisal (GLIA): development of an instrument to identify obstacles to guideline implementation.
      ,
      • Gagliardi A.
      • Brouwers M.C.
      • Palda V.A.
      • Lemieux-Charles L.
      • Grimshaw J.M.
      How can we improve guideline use? A conceptual framework of implementability.
      ] and target methodological and reporting concerns. Currently, no resources take a comprehensive view of all factors relevant to guideline implementability and investigate this from other disciplines focused on changing human behavior, such as psychology, marketing, design, and human factors engineering. To better understand the concept of implementability and the relationship between characteristics of guidelines and their uptake by clinicians (who represent a primary end users of CPGs), our primary objective was to identify factors associated with the implemenatbility of CPGs and recommendations through a comprehensive and multidisciplinary perspective. Our secondary objective was to determine what characteristics are posited to improve uptake by whom and under what circumstances.

      2. Methods

      We conducted a realist review [
      • Pawson R.
      • Greenhalgh T.
      • Harvey G.
      • Walshe K.
      Realist review—a new method of systematic review designed for complex policy interventions.
      ], which is an explicitly theory-driven approach to the synthesis of evidence as it seeks to interrogate the underlying mechanisms of the programs or interventions being studied [
      • Pawson R.
      • Greenhalgh T.
      • Harvey G.
      • Walshe K.
      Realist review—a new method of systematic review designed for complex policy interventions.
      ]. Our protocol is published elsewhere [
      • Kastner M.
      • Estey E.
      • Perrier L.
      • Graham I.D.
      • Grimshaw J.
      • Straus S.E.
      • Zwarenstein M.
      • Bhattacharyya O.
      Understanding the relationship between the perceived characteristics of clinical practice guidelines and their uptake: protocol for a realist review.
      ]. We report our methods and findings according to the RAMESES (Realist And Meta-narrative Evidence Synthesis: Evolving Standards) criteria for the publication standards of realist reviews [
      • Wong G.
      • Greenhalgh T.
      • Westhorp G.
      • Buickingham J.
      • Pawson R.
      RAMESES publication standards: realist synthesis.
      ] and include a flow diagram of our methods for increased clarity (Fig. 1).

      2.1 Search strategy

      Consistent with methodological standards, we used a multiple search strategy that consisted of five iterative stages of searching (see Fig. 2): stage 1: we consulted guideline development and knowledge translation (KT) experts to identify a set of core articles in guideline implementability supplemented with a background search by an information specialist in MEDLINE and EMBASE (up to the year 2010) using the search terms: “implement*,” “clinical practice guidelines,” and “knowledge translation.” Our candidate theories (ie, reasons for poor implementation of guidelines) were identified through consultation with guideline and KT experts: (1) guidelines are not used in part because of specific perceived guideline characteristics affecting uptake by clinicians (eg, too complex or too difficult to follow); (2) there are trade-offs between various guideline attributes that facilitate or hinder uptake (eg, increasing specificity may reduce applicability). Stage 2: we sought the expertise of seven content experts known among our team in guideline development and KT (n = 3), psychology (n = 2), management (n = 1), and human factors engineering (n = 1) to suggest seminal articles or to provide direction to where to look for articles in these disciplines (a method akin to snowball sampling in qualitative research). Stage 3: we expanded our search by looking for the related articles of our core medicine citations identified in stage 1 (n = 76). Reviewers independently selected a subset of priority articles (highly cited or written by known researchers in guideline implementability; n = 22), which were entered into the PubMed “related articles” interface resulting in 1,677 citations. Two reviewers scanned this yield independently using our inclusion criteria of which 131 were deemed potentially relevant. Stage 4: we searched the reference lists of articles from stages 1 and 2 to determine whether we had reached theoretical saturation. Duplicate review of a random subset of citations did not identify new concepts or understandings, so we did not pursue this further. Stage 5: we continuously sought out other potentially relevant articles including the gray literature (eg, unpublished documents and Web sites such as the Guideline International Network). During all stages, we conducted saturation testing iteratively, by asking at regular intervals of searching whether the latest sample of literature has added anything new to our understanding and whether further searching is likely to add new knowledge [
      • Pawson R.
      • Greenhalgh T.
      • Harvey G.
      • Walshe K.
      Realist review—a new method of systematic review designed for complex policy interventions.
      ].

      2.2 Article selection process

      Two sets of reviewers independently screened abstracts to look for our candidate theories and to test for relevance. We included all article types written in English and excluded opinion-driven reports (editorials, commentaries, and letters) unless authored by an individual identified by our team as an “expert” in the field of guideline implementability. Our unit of analysis was not dependent on study design because relevant data could exist in any section of the article (eg, Abstract; Introduction; Discussion), so we did not assess study quality [
      • Pawson R.
      • Greenhalgh T.
      • Harvey G.
      • Walshe K.
      Realist review—a new method of systematic review designed for complex policy interventions.
      ].

      2.3 Data extraction

      We developed a standardized data extraction form that was pilot tested independently by two reviewers using five articles. Data were extracted by two primary reviewers and audited by another two reviewers with disagreements resolved through group consensus on study characteristics, discipline (eg, medicine, psychology), guideline attribute name and definition, attribute operationalization (how the attribute functions within the discipline or context), attribute relationship with guideline uptake, and any potential trade-offs.

      2.4 Data organization

      We identified 1,736 guideline attributes, which were initially sorted by the same name or name root (eg, valid/validity). Two groups of reviewers took the same list of sorted attributes and independently clustered them into logical categories to determine how attributes are related to one another. We then combined groups of similar attributes (including their synonyms and antonyms) that conceptually “fit” within a larger theme and created a label and description for each category. Table 1 describes the operationalization of this process. We compared the categorizations between the two reviewer groups for agreement by documenting “agreed” and “divergent” classifications. This resulted in deriving a common set of 27 categories (and their attributes) grouped into five broad domains that we conceptualized as associated with guideline uptake: language, format, rigor of development, feasibility, and decision making. We considered this conceptualization as our “raw data” for analysis.
      Table 1Process that we used to categorize attributes using the message domain as an example
      GoalStepsExample
      Organize, group, and appropriately label similar or “like” attributes1. Group attributes that are antonyms• Complex/simple
      2. Group attributes that are synonyms• Unclear/confusing
      3. Group attributes with the same root
      • Specific/specificity
      • Validity/valid
      4. Sort database by attribute
      Categorize attributes into logical domains5. Are there commonalities among attributes?

      6. Is there a central theme or focus among groups of attributes?
      The following attributes can be grouped into a category called “Clarity”
      • Unambiguous
      • Precise
      • Specific
      Go through each domain to determine sense and fit of attributes
      • 7.
        Do the attributes belong within the same cluster?
      • 8.
        Can they be collapsed?
      • 9.
        Use attribute definitions to make these decisions
      The following categories can be collapsed:
      • “Complexity” with “Information overload”
      • “Actionability” with “Wording”
      Develop a definition for domains10. Based on their included attributes and definitions, define and label the clusterThe Message domain can be defined as: The clarity, simplicity, and persuasiveness of the guideline language

      2.5 Analysis

      Data analysis involved a three-level process where data were further interrogated and refined with each level of synthesis: level 1: development of a codebook: we developed (in duplicate) a codebook of definitions for each of the five identified domains and attributes to determine recurrent patterns of outcomes (ie, what makes guidelines implementable) and their associated mechanisms and contexts. We included attributes that were deemed “modifiable” by guideline developers (eg, the actionability of recommendations vs. the self-efficacy of the guideline end user); documented its definition, operationalization, and the discipline and context they were studied; and noted any reported positive or negative relationships with guideline uptake and any trade-offs if existed. We subsequently looked at the data for consistency or divergence of outcomes as a function of discipline. The various disciplines tended to focus on unique sets of attributes; the data did not lend to a formal cross-discipline analysis; level 2: validation of data: we determined which attributes were most appropriate for which domain/subdomain to identify the hierarchy of groupings and important relationships. We tested the validity of this organization by administering an online survey to nine experts in KT and guideline development known among our team. They were asked to review the organization and to rename, rearrange, and condense as they saw fit. Accordingly, we collapsed and renamed some attributes/categories/domains (eg, the language domain was renamed message), and we added a sixth domain (stakeholder involvement)—see Table 2; level 3: hierarchical explanatory narratives: we further expanded the knowledge derived from previous levels of analysis by building up an explanation for what intrinsic factors about a guidelines promote (facilitator attributes) or impede (barrier attributes) guideline use and uptake, under what circumstances this happens (an explanation of situations/contexts/settings), the mechanism by which it happens (its operationalization), and the outcomes and consequences of this. We did this by deriving explanations for each of the broad domains and subdomains from a summation of their smaller parts.
      Table 2Final list of attribute categories across six domains of guideline implementability
      Category (N = 16)Major attributesDomain (N = 6)
      Credibility of guideline development groupCredibilityStakeholder involvement
      Disclosure of conflict of interestConflict of interest; transparency; funding sources; editorial independence
      Reporting of what is neededScope; patient preferences; cost and resource requirements; outcomes data; harms and benefitsEvidence synthesis
      Execution of what is neededEvidence-based; valid and reliable; transparent
      Updating of guidelinesUpdating; currency
      Clinical applicabilityClinical relevance; appropriateness of patient population; considered implementationConsidered judgment
      ValuesGuideline developer values; professional/provider values (clinical judgment, clinical freedom); patient values (acceptability, patient preferences);
      Local applicabilityAdaptation; application toolsImplementation feasibility
      Resource constraintsAvailability of resources; economic outcomes
      NoveltyCompatibility; requires new knowledge and skills
      SimpleInformation overload; complexityMessage
      ClearActionability (specificity, ambiguity); effective writing
      PersuasiveFraming; relative advantage
      Multiple versionsEnd users; versions (flat, dynamic); document typeFormat
      ComponentsComponents to include in guidelines (eg, purpose, target audience, methods)
      PresentationLayout of full document (placement, length); structure within sections (match the system to the real world, sequential bundling); information visualization [information display (eg, algorithms, pictures), information context (eg, framing, vividness)]

      3. Results

      Of 2,550 potentially relevant articles identified, 350 articles were screened in full text and 278 articles contributed to the analysis from 170 journals across seven disciplines: medicine (n = 188), psychology (n = 30), management/marketing/business (n = 23), human factors engineering (n = 17), information technology/computer science (n = 10), graphic design (n = 7), and sociology (n = 3). The three most common study designs were narrative reviews (n = 110), qualitative studies (n = 37), and observational studies (n = 32). See Fig. 3 for the flow of article selection.
      Using our hierarchical narrative approach, we built an explanation for each of the six domains (comprising 27 attributes) showing that guideline implementability is associated with two broad goals during the guideline development process: the creation of guideline content (four domains) and the communication of this content (two domains)—see Table 3 and Appendix A at www.jclinepi.com.
      Table 3Summary of hierarchical narratives to describe guideline implementability
      Guideline implementability: To facilitate uptake, the process of guideline development has two broad aims: the creation of content and the communication of that content. The four domains of content creation are stakeholder involvement, evidence synthesis, considered judgment in formulating recommendations, and feasibility. The two domains of content communication relate to fine-tuning the message itself and its format.
      Creation of content: The four interrelated domains of content creation are stakeholder involvement (including credibility of the developers and disclosure of conflicts of interest), evidence synthesis (specifying what evidence is needed and how and when it is synthesized), considered judgment (including clinical applicability and values), and feasibility (local applicability, resource constraints, and novelty). These domains may be considered nonsequentially and iteratively.Communication of content: Communication of guidelines entails fine-tuning both the message of the recommendations (through use of simple, clear, and persuasive language) and their format (through representation in multiple versions, inclusion of specific components, and effective layout and structure).
      Stakeholder involvementEvidence synthesisConsidered judgmentImplementation feasibilityMessageFormat
      The guideline development group should have appropriate composition; its members should have relevant, unbiased expertise and suitable credibility, and potential conflicts of interest should be disclosed. The target population of end users (patients, the public) should be clearly defined, and their views and preferences considered. Ensuring stakeholder involvement during guideline development facilitates uptake.To enhance guideline validity and reproducibility, the necessary evidence must be specified, the method of synthesis clearly defined (ideally evidence-based, valid, reliable, and transparent), and the timing of sequential syntheses appropriate (balancing timeliness of the guideline with stability over time).The guideline development group must supplement evidentiary factors (quality, quantity, and consistency) with considered judgment, making complex trade-offs between the competing benefits and harms, side effects, and risks of various options for managing the disease or condition. They must also consider clinical applicability (whether the guideline responds to variability among patients) and the values and preferences of patients, developers, and care providers (ie, the relative worth or importance of a health state and consequences such as benefits, harms, and costs of a decision).Feasibility reflects local applicability (ie, strategies for adapting recommendations to local conditions), consideration of resource constraints (availability of resources and other economic implications), and the influence of novelty of (or familiarity with) the guidelines, where novelty refers to the degree to which the recommendations propose behaviors considered unconventional by clinicians or patients). Feasible implementation of guidelines allow for flexibility in individual clinical decisions, are in agreement with users' opinions and skills, and are suitable for routine use in intended settings.Guideline messages should use simple, clear, and persuasive language. Simplicity can be achieved by limiting the number of elements, the number of steps within each recommendation, or the number of conditional factors influencing performance (to prevent the quantity of information from exceeding available cognitive capacity). Clarity is enhanced by using specific, unambiguous language and by applying a direct writing style, with active voice, suitable punctuation, short sentences, and bullet lists to convey series of points, and without awkward breaks, abbreviations, hyphenation, redundancy, or unnecessary jargon. Finally, the guideline messages should be clinically convincing and should be framed in terms of potential gains, to convey their relative advantage over any previous approach.Guidelines may be formatted in multiple versions (eg, research-based, information-gathering, analytical tool; briefer guide for clinical education; short version for point-of-care clinical use; lay-language version for patients). Formatting involves determining which components to include in various versions (eg, scope and purpose, target audience, guideline development panel, update plan, and implementation considerations), their presentation (eg, proper placement of visual elements and document length), and structure (ie, matching the system to the real world and bundling); and the overall visualization of information.

      3.1 Creation of content in guidelines

      This (see Table 4 and Appendix B at www.jclinepi.com) category includes four domains: (1) stakeholder involvement: findings showed that the credibility of a guideline affects its likelihood of uptake. The key determinants of credibility are the range of stakeholders involved [
      • Solberg L.I.
      • Brekke M.L.
      • Fazio C.J.
      • et al.
      Lessons from experienced guideline implementers: attend to many factors and use multiple strategies.
      ], disclosure of any conflicts of interest, and funding sources [
      • Muhlhauser I.
      From authority recommendations to fact-sheets—a future for guidelines.
      , ,
      • Rosenfeld R.M.
      • Shiffman R.
      Clinical practice guideline development manual: a quality driven approach for translating evidence into action.
      ,
      American Academy of Pediatrics Steering Committee on Quality Ia, Management
      Classifying recommendations for clinical practice guidelines.
      ,
      • Horvath A.R.
      • Kis E.
      • Dobos E.
      Guidelines for the use of biomarkers: principles, processes and practical consideration.
      ]. The literature suggests that addressing stakeholder involvement entails ensuring the appropriate composition and relevant and unbiased expertise of the guideline development group, documenting the views and preferences of the target population (patients, public), and defining target end users [
      • Brouwers M.C.
      • Kho M.
      • Browman G.P.
      • Burgers J.S.
      • Cluzeau F.
      • Feder G.
      • et al.
      AGREE II: advancing guideline development, reporting and evaluation in health care.
      ,
      • Grimshaw J.M.
      • Russell I.T.
      Achieving health gain through clinical guidelines II: ensuring guidelines change medical practice.
      ]; (2) evidence synthesis: synthesizing the appropriate evidence is a complex task for guideline developers but appears to be the most developed aspect of guideline development in the literature, primarily originating from medicine. Rigorous evidence synthesis is emphasized across medicine, nursing, and health policy, and the approaches reflect a common understanding of how to conduct this process. The literature reflects the attributes deemed necessary to enhance guideline validity and reproducibility and comprises the consistent reporting of the elements that need to be included in guidelines [
      • Rosenfeld R.M.
      • Shiffman R.
      Clinical practice guideline development manual: a quality driven approach for translating evidence into action.
      ,
      • Wolf M.
      • Bower D.J.
      • Marbella A.M.
      • Casanova J.E.
      US family physicians' experiences with practice guidelines.
      ,
      • Hayward R.S.A.
      • W M.
      • Tunis S.R.
      • Bass E.B.
      • Guyatt G.
      Users' guide to the medical literature: VII. How to use clinical practice guidelines. Are the recommendations valid?.
      ,
      • McAlister F.A.
      • van Diepen S.
      • Padwal R.S.
      • Johnson J.A.
      • Majumdar S.R.
      How evidence-based are the recommendations in evidence-based guidelines?.
      ], the execution of these elements [
      • McAlister F.A.
      • van Diepen S.
      • Padwal R.S.
      • Johnson J.A.
      • Majumdar S.R.
      How evidence-based are the recommendations in evidence-based guidelines?.
      ,
      • Grimshaw J.M.
      • Russell I.T.
      Effect of clinical guidelines on medical practice: a systematic review of rigorous evaluations.
      ,

      Institute of Medicine (IOM). Clinical practice guidelines we can trust. Washington, DC: The National Academies Press. Brief report available at: http://www.iom.edu/Reports/2011/Clinical-Practice-Guidelines-We-Can-Trust.aspx. Accessed on September 7, 2014.

      ], and the currency of guidelines [
      • Chou R.
      Using evidence in pain practice. Part I: assessing quality of systematic reviews and clinical practice guidelines.
      ,
      • Watine J.
      • Friedberg B.
      • Nagy E.
      • Onody R.
      • Oosterhuis W.
      • Bunting P.S.
      • Charet J.C.
      • et al.
      Conflict between guideline methodologic quality and recommendation validity: a potential problem for practitioners.
      ]; (3) considered judgment: the literature suggests that guideline developers need to supplement evidence-based formulation of recommendations with considered judgment. This involves the consideration of the evidence in light of other factors such the complex trade-offs between competing benefits and harms and risks of different options for managing the disease or condition, clinical applicability and contexts, the values and preferences of those for whom the recommendations are intended (providers, patients, and developers), organizational needs, and costs [
      • Brouwers M.C.
      • Kho M.
      • Browman G.P.
      • Burgers J.S.
      • Cluzeau F.
      • Feder G.
      • et al.
      AGREE II: advancing guideline development, reporting and evaluation in health care.
      ,
      • Grimshaw J.M.
      • Russell I.T.
      Achieving health gain through clinical guidelines II: ensuring guidelines change medical practice.
      ,
      • Oosterhuis W.P.
      • Bruns D.
      • Watine J.
      • Sandberg S.
      • Horvath A.R.
      Evidence-based guidelines in laboratory medicine: principles and methods.
      ,
      • Verkerk K.
      • Van Veenendaal H.
      • Severens J.L.
      • Hendriks E.J.M.
      • Burgers J.S.
      Considered judgment in evidence-based guideline development.
      ,
      • Bousquet J.
      • Schunemann H.J.
      • Zuberbier T.
      • Bachert C.
      • Baena-Cagnani C.E.
      • Bousquet P.J.
      • et al.
      Development and implementation of guidelines in allergic rhinitis—an ARIA-GA2LEN paper.
      ,
      • Schunemann H.J.
      • Fretheim A.
      • Oxman A.
      Improving the use of research evidence in guideline development: 10. Integrating values and consumer involvement.
      ,
      • Fervers B.
      • Carretier J.
      • Bataillard A.
      Clinical practice guidelines.
      ]; (4) implementation feasibility: feasibility involves the local applicability of guidelines [
      • Keeley P.W.
      Clinical guidelines.
      ], the consideration of resource constraints to make them more implementable [
      • Brouwers M.C.
      • Kho M.
      • Browman G.P.
      • Burgers J.S.
      • Cluzeau F.
      • Feder G.
      • et al.
      AGREE II: advancing guideline development, reporting and evaluation in health care.
      ,
      • Schunemann H.J.
      • Oxman A.
      • Brozek J.
      • Glasziou P.
      • Jaeschke R.
      • Vist G.E.
      • et al.
      Grading quality of evidence and strength of recommendations for diagnostic tests and strategies.
      ,
      • Redelmeier D.A.
      • Ferris L.E.
      • Tu J.V.
      • Hux J.E.
      • Schull M.J.
      Problems for clinical judgement: introducing cognitive psychology as one more basic science.
      ], and the influence of the degree of novelty or familiarity of the interventions [
      • Shiffman R.
      • Dixon J.
      • Brandt C.
      • Essaihi A.
      • Hsiao A.
      • Michel G.
      • et al.
      The Guideline Implementability Appraisal (GLIA): development of an instrument to identify obstacles to guideline implementation.
      ,
      • Rosenfeld R.M.
      • Shiffman R.
      Clinical practice guideline development manual: a quality driven approach for translating evidence into action.
      ]. There is sometimes a tension between feasibility and impact. A recommendation may be highly feasible but unlikely to make much of a difference (it supports the status quo) [
      • Dahm P.Y.L.
      • Gallucci M.
      • Simone G.
      • Schunemann H.J.
      How to use a clinical practice guideline.
      ]. In contrast, a recommendation may require far-reaching changes and have enormous potential impact but have low feasibility [
      • Brouwers M.C.
      • Kho M.
      • Browman G.P.
      • Burgers J.S.
      • Cluzeau F.
      • Feder G.
      • et al.
      AGREE II: advancing guideline development, reporting and evaluation in health care.
      ,
      • Grimshaw J.M.
      • Russell I.T.
      Effect of clinical guidelines on medical practice: a systematic review of rigorous evaluations.
      ].
      Table 4Facilitators and barriers for domains of guideline implementability
      Please see Appendix at www.jclinepi.com for more detailed list of facilitators, barriers, and suggested actions for guideline developers with references.
      Domain and examples of facilitators and barriersActions for guideline developers
      Stakeholder involvement
      • Facilitators:
        • Wide range of stakeholders
        • Disclosure of conflicts of interest and funding sources
      • Barriers:
        • Industry contributions
        • Recommendations based on expert opinion alone
      • Select unbiased participants with relevant expertise
      • Declare competing interests and funding sources
      • Document views and preferences of target population
      Evidence synthesis
      • Facilitators:
        • Consistent reporting of elements (eg, phases of illness, resource requirements, outcomes data, harms and benefits)
        • Identification of recommendations based on expert judgment or consensus
        • Maintenance of currency in relation to new evidence
      • Barrier:
        • Excessive frequency of revision
      • Report type and quality of evidence
      • Report methods of synthesis
      • Set timelines for guideline review
      Considered judgment
      • Facilitators:
        • Perceived clinical relevance and appropriateness for patient population
        • Indicators of relative strength of recommendations
        • Definition of values that influenced recommendations
      • Barriers:
        • Lack of fit between clinicians' experiences and recommendations
        • Lack of applicability to all or “typical” patients.
        • Use of low-quality, weak, or conflicting evidence
      • State population(s) to which statements apply
      • Disclose appropriateness or applicability of
      • Clearly state value judgments
      • Document process for managing dissent
      Implementation feasibility
      • Facilitators:
        • Limit on recommendations that require large investment of time or resources
        • Availability of data regarding resource requirements and cost effectiveness
        • Minimization of change required for users and systems
      • Barriers:
        • Requirement for new knowledge or skills
        • Unconventional changes
        • Inconsistency of changes with existing values, needs, and experiences of adopters
      • Formulate recommendations in terms of measurable criteria and targets for quality improvement
      • Identify costs and resource requirements
      • Specify competencies, training, and technical specifications required
      • Include economic data
      Message
      • Facilitators:
        • Actionability
        • Crisp and persuasive messages
        • Effective use of language
      • Barriers:
        • Underlying evidence that is contradictory or complex
        • Use of evidence that is still evolving or evidence not commonly observed in practice
        • Ambiguity and vagueness
        • Poor framing of the guideline
      • Use conditional statements
      • Use specific, concrete statements
      • Justify any deliberate vagueness
      • Use short sentences with proper punctuation
      • Keep related information together
      • Frame recommendations in terms of gains
      • Focus on errors of omission (not doing the right thing) rather than errors of commission (doing the wrong thing)
      Format
      • Facilitators:
        • Use of multiple formats or alternate versions
        • Inclusion of components known to be important to implementation
        • Appropriate placement of graphics and text
        • Appropriate structure (high-level categorization of recommendations)
        • Information visualization to shift cognitive load to perceptual system through graphics and animation (eg, matching system to the real world, bundling of related recommendations)
        • Use of words for procedural information, logical conditions, and abstract concepts
        • Use of images for spatial structures, location, and detail
      • Tailor guidelines to intended end users
      • Highlight key evidence-based features having the most significance for patient care
      • Highlight key recommendations with links to more extensive explications
      • Choose the most appropriate graphic according to the type of information to be conveyed
      • Use color appropriately
      a Please see Appendix at www.jclinepi.com for more detailed list of facilitators, barriers, and suggested actions for guideline developers with references.

      3.2 The communication of content to guideline end users

      This (Table 4 and Appendix B at www.jclinepi.com) category includes our final two domains: (5) message: studies suggest that to optimize the messaging of a guideline and in turn its uptake, the language used in recommendations has to be simple, clear, and persuasive. Literature spanned across all disciplines, with the psychology literature focusing on how to reduce cognitive load and increase understanding and retention and the management literature focusing on crafting convincing and salient arguments. It is also suggested that the level of complexity of guidelines and recommendations is inversely proportional to its adoption [
      • Scott S.D.
      • Plotnikoff R.C.
      • Karunamuni N.
      • Bize R.
      • Rodgers W.
      Factors influencing the adoption of an innovation: an examination of the uptake of the Canadian Heart Health Kit (HHK).
      ,
      • Tornatzky L.G.
      • Klein K.J.
      Innovation characteristics and innovation adoption-implementation: a meta-analysis of findings.
      ,
      • Thompson R.L.
      • Higgins C.A.
      • Howell J.M.
      Personal computing: toward a conceptual model of utilization.
      ,
      • Ball K.
      Surgical smoke evacuation guidelines: compliance among perioperative nurses.
      ,
      • Cabana M.D.
      • Flores G.
      The role of clinical practice guidelines in enhancing quality and reducing racial/ethnic disparities in paediatrics.
      ,
      • Rashidian A.
      • Eccles M.P.
      • Russell I.
      Falling on stony ground? A qualitative study of implementation of clinical guidelines' prescribing recommendations in primary care.
      ] and compliance [
      • Gurses A.P.
      • Marsteller J.A.
      • Ozok A.A.
      • Xiao Y.
      • Owens S.
      • Pronovost P.J.
      Using an interdisciplinary approach to identify factors that affect clinicians' compliance with evidence-based guidelines.
      ,
      • Shaneyfelt T.M.
      • Mayo-Smith M.F.
      • Rothwangl J.
      Are guidelines following guidelines? The methodological quality of clinical practice guidelines in the peer-reviewed medical literature.
      ]. The bodies of literature did not contradict but rather focus on different aspects of messaging; (6) format: a number of formatting aspects of the guideline were identified from the literature, which allows the various elements of guidelines to become more explicit [
      • Redelmeier D.A.
      • Ferris L.E.
      • Tu J.V.
      • Hux J.E.
      • Schull M.J.
      Problems for clinical judgement: introducing cognitive psychology as one more basic science.
      ], promoting their use in practice [
      • Gagliardi A.
      • Brouwers M.C.
      • Palda V.A.
      • Lemieux-Charles L.
      • Grimshaw J.M.
      How can we improve guideline use? A conceptual framework of implementability.
      ,
      • Carlsen B.G.C.
      • Pope C.
      Thou shalt versus thou shalt not: a meta-synthesis of GPs' attitudes to clinical practice guidelines.
      ,
      • Tong A.
      Clinical guidelines: can they be effective?.
      ,
      • Grimshaw J.M.
      • Russell I.T.
      Effect of clinical guidelines on medical practice: a systematic review of rigorous evaluations.
      ,
      • Stone T.T.
      • Schweikhart S.B.
      • Mantese A.
      • Sonnad S.S.
      Guideline attribute and implementation preferences among physicians in multiple health systems.
      ,
      • Harris J.S.
      Development, use, and evaluation of clinical practice guidelines.
      ]. These include having: (1) multiple versions of guidelines. This refers to the evolution of guidelines, which typically progress from a research-based tool to a short version for clinical use and to a lay-language version for patients [
      • Harris J.S.
      Development, use, and evaluation of clinical practice guidelines.
      ]; (2) specific components within guidelines (eg, purpose, methods, recommendations); and (3) the presentation (appearance) of guidelines, which involves layout, structure, and information visualization. Some of the common design principles for scientific communication have an empirical foundation, but many are derived from best practices and user preferences. Because most formatting principles are based on cognitive processes, they are likely to be generalizable across disciplines and contexts.

      4. Discussion

      Our synthesis of 278 articles from a multidisciplinary body of literature identified six domains of guideline implementability that affects uptake of recommendations within two broad categories (the creation and communication of guideline content). Building on the work of Shiffman and Gagliardi, we investigated guideline implementability from a broad perspective. Shiffman et al. [
      • Shiffman R.
      • Dixon J.
      • Brandt C.
      • Essaihi A.
      • Hsiao A.
      • Michel G.
      • et al.
      The Guideline Implementability Appraisal (GLIA): development of an instrument to identify obstacles to guideline implementation.
      ] developed GLIA by searching the literature for guideline attributes in five key reports describing impact of factors on successful implementation and by examining three instruments for the appraisal of guideline quality. Gagliardi et al. [
      • Gagliardi A.
      • Brouwers M.C.
      • Palda V.A.
      • Lemieux-Charles L.
      • Grimshaw J.M.
      How can we improve guideline use? A conceptual framework of implementability.
      ] focused on 18 health care studies and created an implementability framework comprised 22 elements organized into eight domains (adaptability, usability, validity, applicability, communicability, accommodation, implementation, and evaluation). We drew on a wide range of disciplines to further enhance this work and involved appropriate experts in the review process. Our analysis moves beyond the medical and implementation science literature (which tend to focus on aspects of the creation of content) and incorporates other disciplinary content, which expands understanding related to the communication of content. Literature from human factors engineering adds information and direction on how developers can structure guidelines to meet the specific needs of end users and mirror their work processes and approach to care. The business/management literature adds insights and direction regarding the importance of persuasive and clear messaging, whereas the design literature outlines specific design principles, which are intended to improve the usability and attractiveness of products. Literature from cognitive psychology alerts to the limitations of information processing and provides some explicit strategies for developers to ease cognitive load.
      We achieved rigor in our review through transparent documentation of the methodological approach [
      • Kastner M.
      • Estey E.
      • Perrier L.
      • Graham I.D.
      • Grimshaw J.
      • Straus S.E.
      • Zwarenstein M.
      • Bhattacharyya O.
      Understanding the relationship between the perceived characteristics of clinical practice guidelines and their uptake: protocol for a realist review.
      ] as well as for our search strategy, study selection, analysis, and interpretation, including any deviations from the original method. We also provided a detailed flow diagram for our study methods (Fig. 1) and article selection process (Fig. 2). We also adopted a consensus-based approach to clarify difficult concepts, where smaller groups of investigators discussed their assumptions related to the meaning of a concept, sought clarification from discipline-specific experts if needed, and then reached consensus within the broader study team on the applicability of such concepts in the context of guideline implementability.
      Our review suggests a few areas for further study. Guideline end users commonly complain about their length, and we know that simplifying language and content can improve information processing and retention and improve perceived ease of use (and thus potentially intention to use), so simplifying guidelines appears to be a worthy endeavor. However, we do not know the threshold of simplification that could result in misinterpretation of guideline recommendations. Determining what type of content ought to be simplified and in what ways requires additional investigation to provide developers with guidance regarding a balanced approach to creating and communicating guideline content. In addition, several issues of how to operationalize the concept of considered judgment remain unclear. An approach to help developers facilitate the process is needed. Without practical procedures, it is likely that this step will be conducted in a cursory fashion or not at all.
      Our work has some limitations. We chose a realist approach to develop theories about what attributes of guidelines contribute to their implementability, for whom, and under what circumstances. However, the literature was rarely explicit about context and circumstances of guideline development and uptake, so it became difficult to develop and interrogate such theories and to explore context–mechanism–outcome relationships. Additionally, we had anticipated exploring the similarities and differences between how the literature understood guideline implementability but found few contradictions in the literature. Studies tended to focus on the problems inherent in guidelines from different perspectives: usability and usefulness (human factors engineering), information processing and retention (cognitive psychology), document construction (design), consumer relations (management), and clinical relevance and importance (medicine). As an alternative, we developed a conceptual framework to synthesize the data, highlighting contextual issues through the domain narratives, positing relationships rather than proving them. Although we sought the expertise of content experts in psychology, management, and human factors engineering, we may have missed other relevant articles in these disciplines as we did not systematically search these beyond our expert-identified and snowball sampling techniques. To offset this risk, we performed a “related articles” search of “core” articles. Additionally, literature from medicine may be overrepresented as the bulk of our core articles was identified from this discipline. However, given that guidelines are most heavily discussed in the health/medicine literature, we thought this was an acceptable risk of bias. There may be relevant concepts from other disciplines we did not include, and there are areas that are not well covered within the disciplines we studied. However, we incorporated a rigorous, systematic, and transparent methodology and consulted experts at several key points during the analysis as a check for external validity.

      5. Conclusions and recommendations

      Guidelines summarize clinical evidence to inform clinicians' decision making, but how they are developed and written influences how they are used. Our work represents a comprehensive and interdisciplinary effort toward better understanding, which attributes of guidelines have the potential to improve uptake in clinical practice. However, each of these will need to be empirically tested to identify the best “package” of attributes. We also created narratives of key concepts, which can be used to develop tools to determine their impact on building better guidelines aimed at increasing their uptake and promoting better care.

      Acknowledgments

      The authors thank the following individuals for providing content expertise: Jamie Brehaut (psychology); Jeremy Grimshaw (knowledge translation); Stephen Hanna, Holger Schunemann, Francoise Cluzeau (guideline development); Dilip Soman (management); Mark Chignell (human factors engineering); Hyun Joo Lee (design).
      Ethical approval: No ethics approval was required for this study.

      References

        • Shiffman R.
        • Dixon J.
        • Brandt C.
        • Essaihi A.
        • Hsiao A.
        • Michel G.
        • et al.
        The Guideline Implementability Appraisal (GLIA): development of an instrument to identify obstacles to guideline implementation.
        BMC Med Inform Decis Mak. 2005; 5: 23
        • Gagliardi A.
        • Brouwers M.C.
        • Palda V.A.
        • Lemieux-Charles L.
        • Grimshaw J.M.
        How can we improve guideline use? A conceptual framework of implementability.
        Implement Sci. 2011; 6: 26
        • Grol R.
        • Dalhuijsen J.
        • Thomas S.
        • Veld C.
        • Rutten G.
        • Mokkink H.
        Attributes of clinical guidelines that influence use of guidelines in general practice: observational study.
        BMJ. 1998; 317: 858-861
        • Michie S.
        • Johnston M.
        Changing clinical behaviour by making guidelines specific.
        BMJ. 02/07 2004; 328: 343-345
        • Michie S.
        • Lester K.
        Words that matter: increasing the implementation of clinical guidelines.
        Qual Saf Health Care. 2005; 14: 367-370
        • Rogers E.
        Diffusion of innovations.
        Free Press, New York1995
        • Rey G.
        • Buchwald F.
        The expertise reversal effect: cognitive load and motivational explanations.
        J Exp Psychol Appl. 2011; 17: 33-48
        • Wang A.
        • Dowding T.
        Effects of visual priming on improving web disclosure to investors.
        J Behav Finance. 2010; 11: 11-20
        • Tricoci P.
        • Allen J.M.
        • Kramer J.M.
        • Califf R.M.
        • Smith Jr., S.C.
        Scientific evidence underlying the ACC/AHA clinical practice guidelines.
        JAMA. 2009; 301: 831-841
        • Brouwers M.C.
        • Kho M.
        • Browman G.P.
        • Burgers J.S.
        • Cluzeau F.
        • Feder G.
        • et al.
        AGREE II: advancing guideline development, reporting and evaluation in health care.
        CMAJ. 2010; 182: E839-E842
        • Atkins D.
        • Best D.
        • Briss P.A.
        • Eccles M.
        • Falck-Ytter Y.
        • Flottorp S.
        • et al.
        • GRADE Working Group
        Grading quality of evidence and strength of recommendations.
        BMJ. 2004; 328: 1490
        • Fervers B.
        • Burgers J.
        • Haugh M.C.
        • Latreille J.
        • Milka-Cabanne N.
        • Paquet L.
        • et al.
        Adaptation of clinical guidelines: literature review and proposition for a framework and procedure.
        Int J Qual Health Care. 2006; 18: 167-176
        • Harrison M.B.
        • Graham I.
        • van den Hoek J.
        • Dogherty E.J.
        • Carley M.E.
        • Angus V.
        Guideline adaptation and implementation planning: a prospective observational study.
        Implement Sci. 2013; 8: 49
        • Pawson R.
        • Greenhalgh T.
        • Harvey G.
        • Walshe K.
        Realist review—a new method of systematic review designed for complex policy interventions.
        J Health Serv Res Policy. 2005; 19: S21-S34
        • Kastner M.
        • Estey E.
        • Perrier L.
        • Graham I.D.
        • Grimshaw J.
        • Straus S.E.
        • Zwarenstein M.
        • Bhattacharyya O.
        Understanding the relationship between the perceived characteristics of clinical practice guidelines and their uptake: protocol for a realist review.
        Implement Sci. 2011; 6: 69
        • Wong G.
        • Greenhalgh T.
        • Westhorp G.
        • Buickingham J.
        • Pawson R.
        RAMESES publication standards: realist synthesis.
        BMC Med. 2013; 11: 21
        • Solberg L.I.
        • Brekke M.L.
        • Fazio C.J.
        • et al.
        Lessons from experienced guideline implementers: attend to many factors and use multiple strategies.
        Jt Comm J Qual improv. 2000; 26: 171-188
        • Muhlhauser I.
        From authority recommendations to fact-sheets—a future for guidelines.
        Diabetologia. 2010; 53: 2285-2288
      1. The ADAPTE Process: resource toolkit for guideline adaptation. Version 2.0. 2009 (Available at) (Accessed on September 7, 2014)
        • Rosenfeld R.M.
        • Shiffman R.
        Clinical practice guideline development manual: a quality driven approach for translating evidence into action.
        Otolaryngol Head Neck Surg. 2009; 140: S1-S43
        • American Academy of Pediatrics Steering Committee on Quality Ia, Management
        Classifying recommendations for clinical practice guidelines.
        Pediatrics. 2004; 114: 874-877
        • Horvath A.R.
        • Kis E.
        • Dobos E.
        Guidelines for the use of biomarkers: principles, processes and practical consideration.
        Scand J Clin Lab Invest Suppl. 2010; 70: 109-116
        • Grimshaw J.M.
        • Russell I.T.
        Achieving health gain through clinical guidelines II: ensuring guidelines change medical practice.
        Qual Health Care. 1994; 3: 45-52
        • Wolf M.
        • Bower D.J.
        • Marbella A.M.
        • Casanova J.E.
        US family physicians' experiences with practice guidelines.
        Fam Med. 1998; 30: 117-121
        • Hayward R.S.A.
        • W M.
        • Tunis S.R.
        • Bass E.B.
        • Guyatt G.
        Users' guide to the medical literature: VII. How to use clinical practice guidelines. Are the recommendations valid?.
        JAMA. 1995; 274: 570-574
        • McAlister F.A.
        • van Diepen S.
        • Padwal R.S.
        • Johnson J.A.
        • Majumdar S.R.
        How evidence-based are the recommendations in evidence-based guidelines?.
        PLoS Med. 2007; 4: e250
        • Grimshaw J.M.
        • Russell I.T.
        Effect of clinical guidelines on medical practice: a systematic review of rigorous evaluations.
        Lancet. 1993; 342: 1317-1322
      2. Institute of Medicine (IOM). Clinical practice guidelines we can trust. Washington, DC: The National Academies Press. Brief report available at: http://www.iom.edu/Reports/2011/Clinical-Practice-Guidelines-We-Can-Trust.aspx. Accessed on September 7, 2014.

        • Chou R.
        Using evidence in pain practice. Part I: assessing quality of systematic reviews and clinical practice guidelines.
        Pain Med. 2008; 9: 518-530
        • Watine J.
        • Friedberg B.
        • Nagy E.
        • Onody R.
        • Oosterhuis W.
        • Bunting P.S.
        • Charet J.C.
        • et al.
        Conflict between guideline methodologic quality and recommendation validity: a potential problem for practitioners.
        Clin Chem. 2006; 52: 65-72
        • Oosterhuis W.P.
        • Bruns D.
        • Watine J.
        • Sandberg S.
        • Horvath A.R.
        Evidence-based guidelines in laboratory medicine: principles and methods.
        Clin Chem. 2004; 50: 806-818
        • Verkerk K.
        • Van Veenendaal H.
        • Severens J.L.
        • Hendriks E.J.M.
        • Burgers J.S.
        Considered judgment in evidence-based guideline development.
        Int J Qual Health Care. 2006; 18: 365-369
        • Bousquet J.
        • Schunemann H.J.
        • Zuberbier T.
        • Bachert C.
        • Baena-Cagnani C.E.
        • Bousquet P.J.
        • et al.
        Development and implementation of guidelines in allergic rhinitis—an ARIA-GA2LEN paper.
        Allergy. 2010; 65: 1212-1221
        • Schunemann H.J.
        • Fretheim A.
        • Oxman A.
        Improving the use of research evidence in guideline development: 10. Integrating values and consumer involvement.
        Health Res Policy Syst. 2006; 4: 22
        • Fervers B.
        • Carretier J.
        • Bataillard A.
        Clinical practice guidelines.
        J Visc Surg. 2010; 147: e341-e349
        • Keeley P.W.
        Clinical guidelines.
        Palliat Med. 2003; 17: 368-374
        • Schunemann H.J.
        • Oxman A.
        • Brozek J.
        • Glasziou P.
        • Jaeschke R.
        • Vist G.E.
        • et al.
        Grading quality of evidence and strength of recommendations for diagnostic tests and strategies.
        BMJ. 2008; 336: 1106-1110
        • Redelmeier D.A.
        • Ferris L.E.
        • Tu J.V.
        • Hux J.E.
        • Schull M.J.
        Problems for clinical judgement: introducing cognitive psychology as one more basic science.
        CMAJ. 2001; 164: 358-360
        • Dahm P.Y.L.
        • Gallucci M.
        • Simone G.
        • Schunemann H.J.
        How to use a clinical practice guideline.
        J Urol. 2009; 181: 472-479
        • Scott S.D.
        • Plotnikoff R.C.
        • Karunamuni N.
        • Bize R.
        • Rodgers W.
        Factors influencing the adoption of an innovation: an examination of the uptake of the Canadian Heart Health Kit (HHK).
        Implement Sci. 2008; 3: 41
        • Tornatzky L.G.
        • Klein K.J.
        Innovation characteristics and innovation adoption-implementation: a meta-analysis of findings.
        IEEE Trans Eng Management. 1982; EM-29: 28-43
        • Thompson R.L.
        • Higgins C.A.
        • Howell J.M.
        Personal computing: toward a conceptual model of utilization.
        MIS Q. 1991; 15: 125-143
        • Ball K.
        Surgical smoke evacuation guidelines: compliance among perioperative nurses.
        AORN J. 2010; 92: e1-e23
        • Cabana M.D.
        • Flores G.
        The role of clinical practice guidelines in enhancing quality and reducing racial/ethnic disparities in paediatrics.
        Paediatric Respir Rev. 2002; 3: 52-58
        • Rashidian A.
        • Eccles M.P.
        • Russell I.
        Falling on stony ground? A qualitative study of implementation of clinical guidelines' prescribing recommendations in primary care.
        Health Policy. 2008; 85: 148-161
        • Gurses A.P.
        • Marsteller J.A.
        • Ozok A.A.
        • Xiao Y.
        • Owens S.
        • Pronovost P.J.
        Using an interdisciplinary approach to identify factors that affect clinicians' compliance with evidence-based guidelines.
        Crit Care Med. 2010; 38: S282-S291
        • Shaneyfelt T.M.
        • Mayo-Smith M.F.
        • Rothwangl J.
        Are guidelines following guidelines? The methodological quality of clinical practice guidelines in the peer-reviewed medical literature.
        JAMA. 1999; 281: 1900-1905
        • Carlsen B.G.C.
        • Pope C.
        Thou shalt versus thou shalt not: a meta-synthesis of GPs' attitudes to clinical practice guidelines.
        Br J Gen Pract. 2007; 57: 971-978
        • Tong A.
        Clinical guidelines: can they be effective?.
        Nurs Times. 2001; 97: III-IV
        • Grimshaw J.M.
        • Russell I.T.
        Effect of clinical guidelines on medical practice: a systematic review of rigorous evaluations.
        Obstetrical Gynecol Surv. 1994; 49: 469
        • Stone T.T.
        • Schweikhart S.B.
        • Mantese A.
        • Sonnad S.S.
        Guideline attribute and implementation preferences among physicians in multiple health systems.
        Qual Manag Health Care. 2005; 14: 177-187
        • Harris J.S.
        Development, use, and evaluation of clinical practice guidelines.
        J Occup Environ Med. 1997; 39: 23-34