If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, CanadaMichael G DeGroote Cochrane Canada and McMaster GRADE Centres, McMaster University, Hamilton, Ontario, Canada
Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, CanadaMichael G DeGroote Cochrane Canada and McMaster GRADE Centres, McMaster University, Hamilton, Ontario, Canada
Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, CanadaDepartment of Internal Medicine, American University of Beirut, Beirut, Lebanon
Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, CanadaMichael G DeGroote Cochrane Canada and McMaster GRADE Centres, McMaster University, Hamilton, Ontario, Canada
Department of Clinical Epidemiology, Biostatistics and Bioinformatics, Amsterdam UMC, University of Amsterdam, Amsterdam Public Health, Amsterdam, Netherlands
Institute for Evidence in Medicine, Medical Center and Faculty of Medicine, University of Freiburg, Freiburg, GermanyCochrane Germany, Cochrane Germany Foundation, Freiburg, Germany
Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, CanadaMichael G DeGroote Cochrane Canada and McMaster GRADE Centres, McMaster University, Hamilton, Ontario, Canada
Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, CanadaMichael G DeGroote Cochrane Canada and McMaster GRADE Centres, McMaster University, Hamilton, Ontario, CanadaInstitute for Evidence in Medicine, Medical Center and Faculty of Medicine, University of Freiburg, Freiburg, GermanyDepartment of Medicine, McMaster University, Hamilton, Canada
Health guidelines are a key knowledge translation tool produced and used by numerous stakeholders worldwide. Effective participation in guideline development groups or development groups is crucial for guideline success, yet little guidance exists for members of these groups. In this study, we present the Guideline Participant Tool (GPT) to support effective participation in guideline groups, in particular those using the Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) approach.
Study Design and Setting
We used a mixed methods and iterative approach to develop a tool to support guideline participation. We used the findings of a published systematic review to develop an initial list of items for considerations for guideline participants. Then, we refined this list through key informant interviews with guideline chairs, sponsors, and participants. Finally, we validated the GPT in three guideline groups with 26 guideline group members.
Results
The initial list of items based on 37 articles from the existing systematic review included 15 themes and 61 items for a draft tool. Ten key informant interviews helped us refine the list to include the following themes: selection of participants, guideline group process, and tool format. 26 respondents completed the validation survey from three guideline groups. Refinement of the tool ultimately generated a GPT with 33 items for participant consideration before, during, and in follow-up to guideline group meetings.
Conclusion
The GPT contains helpful guidance for all guideline participants, particularly those without previous guideline experience. Future research should further explore the need for additional tools to support guideline participants and identify and develop strategies for improving guideline members’ participation in guideline groups. This work will be incorporated into INGUIDE.org guideline training and credentialing efforts by the Guidelines International Network and McMaster University.
Effective participation and group process is important to the functioning of guidelines and the development of impactful and evidence-based recommendations but tools to inform guideline development group members and panels are sparse.
•
This study presents the results of a mixed methods and iterative study to develop the Guideline Participant Tool (GPT) to support effective participation in guideline panels.
What is the implication and what should change now?
•
Guideline group members can be informed about their role and processes to optimize their participation in guideline development by using this tool.
•
The guideline participant tool gives suggestions to guideline participants for preparation, conduct during guideline group meetings, and in follow-up to guideline group meetings and can be used to improve the guideline process.
•
It informs the work performed to develop the GIN-McMaster Guideline Development Checklist and focuses on suggestions for the participants themselves.
•
We aim to include the findings in the new INGUIDE guideline development certification and credentialing program.
1. Background
Health guidelines are one of the key knowledge translation (KT) tools to implement evidence into practice [
]. The Grading of Recommendations, Assessment, Development, and Evaluation approach (GRADE) is widely used by organizations that produce guidelines to support this aspect of KT [
]. For example, the World Health Organization (WHO) has adopted GRADE for the structuring of guideline development to ensure optimal quality and transparency in development of its guidelines [
]. Guideline producing organizations run guideline group meetings regularly; it is estimated that over 700 GRADE guidelines have been developed (BiGG Database http://sites.bvsalud.org/bigg/biblio/, January 2020). Research to evaluate and optimize the guideline development process and meeting efficacy, in particular with regards to applying GRADE in guideline panels or development groups, is therefore of high importance.
We previously developed the Guidelines International Network (GIN)-McMaster guideline development tool and checklist for guideline group chairs (https://heigrade.mcmaster.ca/guideline-development/) to support optimal planning, creation, and implementation of trustworthy guidelines [
]. This work focused on key tasks and criteria that guideline developers should use for the organization and management of these projects. The checklist has been used in many guideline efforts [
Centro Nazionale per l’Eccellenza Clinica, la Qualità e la Sicurezza delle Cure - Istituto Superiore di Sanita. Manuale metodologico per la produzione di linee guida di pratica clinica. https://snlg.iss.it/wp-content/uploads/2019/04/MM_v1.3.2_apr_2019.pdf. Accessed November 30, 2019.
]. Although more guidance for guideline development is available, the contribution and participation of guideline group members has become more demanding.
At the same time, appropriate participation by guideline group members in guideline groups is important for effective functioning and development of recommendations [
]. That includes orienting and training members who often have no prior experience with guidelines or group processes in general. In working with numerous guideline groups from professional organizations, the WHO, the European Commission, and ministries of health the need to orient guideline participants to the tasks and process involved in guideline development has become apparent [
]. Therefore, the objective of this study was to develop a tool to support effective and appropriate participation on guideline groups.
2. Methods
We developed the Guideline Participant Tool (GPT) and a description of guideline participant roles using a mixed methods iterative approach. We developed a study protocol and obtained research ethics approval from the McMaster-Hamilton Integrated Research Ethics Board (#1128).
First, we used the findings of a systematic review of guideline handbooks to identify recommendations for guideline participants. The methods for this review are described elsewhere [
]. In the second step, we interviewed individuals who participated in previous guideline development projects to ascertain their feedback on the tool. We conducted interviews in blocks of 3 to 4 participants to balance multiple perspectives when refining the tool. After reaching saturation of feedback on the tool, we used the GPT prospectively in three guideline groups and sought feedback from guideline group chairs, members, and organizers. Four sequential iterations of the GPT were thus developed. We describe each of these steps in the subsequent paragraphs below.
2.1 Guideline manual review
We used our review of guideline manuals to develop the GIN-McMaster guideline development checklist [
]. We identified reports and guidelines related to guideline methodology, chairing, and participation in guideline groups. We consulted guideline experts (HJS, EAA, JM, and JB) to identify new guideline manuals and documents published since the initial search for development of this tool. The search and selection process is presented in Fig. 1. We uploaded the guideline documents to NVIVO (v11) and reviewed these for items relevant to participants in the guideline development process. We encoded these items and generated themes for analysis. Three authors (TP, EA, and HJS) reviewed the comprehensive list of items and refined it to create a first iteration of the GPT for use in the first set of interviews.
Fig. 1Search and selection process for initial literature review [
We conducted qualitative interviews using a convenience sample of the past guideline group members with whom we had recently participated in international guidelines. We complimented this sample with additional prospective guideline group members through a review of guideline group websites. We sought a diverse range of perspectives based on geography, disciplines/professions, gender, and duration of involvement in the guideline process.
We invited guideline group members by e-mail to participate in a 30-60 minute semistructured interview through video conferencing or in-person. If no response was received, we sent a follow-up e-mail 2 weeks later. Interviewees received the latest iteration of the GPT and list of participant roles, and we requested electronic feedback on the document in advance of the interview.
The semistructured interviews used probing questions about the following: experiences participating in guideline committees, additions and omissions from the tool, identification of the most important elements of the tool, and its potential use in practice. We tailored the interviews as per interviewee involvements in guideline development and the feedback they provided on the GPT in advance of the interviews. One author (TP) conducted all interviews, using an interview guide available in Appendix 3, and a professional transcriptionist transcribed all audio files. We verified and deidentified transcription files.
The lead authors (TP, EAA, and HJS) met after each set of interviews to discuss the feedback and the next iteration of the tool. We sought to improve the content, organization, flow, and clarity of the tool with each new iteration. We then invited interviewees to comment on the revisions to the tool based on their interview. We judged saturation to be reached when additional interviews suggested minimal further changes to the GPT. All interviewees were again invited to comment electronically on the final iteration of the tool at the conclusion of all interviews.
Interview transcripts were imported to NVIVO (v11) qualitative data analysis software. Three researchers (TP, MJ, and TB) independently reviewed and coded the transcripts using a closed loop iterative process. We also conducted independent thematic analyses of interview transcripts, then met to collate, discuss, and finalize the analysis as a group. The thematic analysis also informed edits to the GPT and participant role descriptions.
2.3 External validation survey
The final iteration of the tool was used to illicit validate the tool using an online survey in SurveyMonkey (San Mateo, US). Feedback was sought from three guideline groups: the American Society of Hematologists (ASH), the European Commission Initiative on Breast Cancer (ECIBC), and the WHO’s Guideline on tuberculosis infection prevention and control. Respondents included guideline group chairs, guideline group participants, and individuals from guideline sponsor organizations. The full external validation survey is presented in Appendix 4. We excluded any responses that suggested not having reviewed the tool from the analysis. We used feedback from this survey to further refine the tool to arrive at a final version of the GPT presented in this article.
3. Results
3.1 Guideline manual review results
We reviewed 37 documents and articles in-depth suggesting guidance for guideline participants categorized into the following themes:
i.
Conflict/declaration of interests, bias, confidentiality, and nondisclosure;
ii.
Decision-making processes;
iii.
Evaluation of guideline groups;
iv.
Facilitation tips, managing group conflict, and working with problematic participants;
v.
Follow-up from guideline group meetings;
vi.
Group processes recommendations;
vii.
Knowledge of participants in guideline groups;
viii.
Membership and participant qualities for guideline groups;
ix.
Preparations for guideline group meetings;
x.
Presentations to guideline groups;
xi.
Research and writing methods for participants;
xii.
Respect to fellow participants in the guideline group;
xiii.
Responsibilities and tasks of guideline group members;
xiv.
Timeline of participation in guideline groups; and
xv.
Training for prospective guideline group participants.
Review and refinement of this list resulted in 61 items preliminarily incorporated into the GPT.
3.2 Interview results
All individuals we invited to participate in interviews agreed to do so. Interview participant characteristics are presented in Table 1. Interviews ranged from 26 to 48 minutes in duration. Interviewees were from five continents with experience in health guidelines from the subnational to international level. Four interviewees reported less than 2 years of experience in guideline development process. Eight of the interviewees were either content or methodological experts or both. One interviewee belonged to a guideline sponsor organization. One interviewee had served as a patient representative in a guideline group member.
No prior experience Experience with 1 guideline More than 1 guideline
2 (20%) 1 (10%) 7 (70%)
Thematic analysis of the interview transcripts identified numerous changes to improve the GPT; these were iteratively incorporated. Feedback emerging from the analysis resulted in the following four themes: (1) Experiences in guidelines regarding participant behavior, (2) the selection of participants, (3) considerations on the guideline development process, and (4) format of the tool. We organized feedback from these themes to inform refinements to the GPT.
Given the multiple technical and professional competencies that guideline groups may require, new members (serving to train them in the guideline process) may have different informational needs than longstanding members, who may only need a reminder of expectations. Interviewees articulated that all members, in particular technical experts must communicate concisely, effectively, and collaboratively toward the production of the guidelines. Members should be aware of their roles, the tasks to be completed, and the timelines within which the guidelines are expected to be completed.
3.2.2 Selection of participants
Interviewees consistently reported that successful guideline development process required selecting the appropriate members. Most interviewees identified “technical experts” as a subgroup of guideline participants who, through their work on the evidence synthesis have significant influence on the guideline process because of their content expertise. They suggested that technical experts should communicate clearly with the guideline group who may not all have the same expertise.
3.2.3 Guideline group process
Interviewees noted that effective guideline group process is impacted by the quality of organization and support from guideline sponsor organizations and effective chairing. From experience, interviewees asked for guideline meetings to be structured by an agenda and clear expectations to ensure that discussions are held in a timely manner and activities are kept on target. Along the same line, interviewees highlighted the importance of having an early set of clear PICO questions, systematic reviews, and timelines for the technical experts conducting evidence syntheses, so that sufficient time is allowed for the review of evidence in advance of guideline meetings.
Interviewees also noted from their experience that differences cultures and languages can have a major impact on group process and that respect and consideration for this should be given by all guideline participants.
3.2.4 Format of the GPT
While the purpose of the GPT to orient guideline group members was clear among the interviewees, there were varied opinions on how the tool should be structured and shared with guideline group members. We attempted to balance these perspectives in refining the tool. For example, some interviewees suggested shortening the tool to one page, whereas others thought it should be longer and more comprehensive. Some suggested subdividing the list to customize it either for the particular guideline or for a particular participant's role. However, some preferred providing the whole tool to each participant to understand other participant role. We initially structured this list as a checklist, but changed this to a tool based on interviewee feedback, because interviewees felt the items were suggestions and considerations, not checklist items to be completed.
3.3 The tool
The final version of the GPT includes guidance for guideline group members containing 33 items for consideration before, during, and in follow-up to guideline group meetings (Appendix 1). We also developed a description of participant roles including cochairs, consumer representatives, content experts, end users, methodologists, other stakeholders, technical experts, support persons, and observers (Appendix 2).
3.4 Validation survey results
Twenty-six eligible guideline group members from the ASH, ECIBC, and WHO completed the survey. Twenty-four participants (92%) had received at least some formal training in health research methodology, epidemiology, and/or biostatistics. Twenty-one (81%) had been involved in guideline development in the past in varying capacities.
Fig. 2 reveals that participants tended to feel that they understood the purpose of the GPT (69% of survey respondents agreed that they were clear on the purpose of the tool). Seventy-three percent of survey respondents agreed that the tool helped with orienting them to the guideline process. In particular, they found the list with names of participants in the guideline development group valuable for understanding guideline members’ roles. In general, participants felt that the length of the tool was appropriate and that they had ample time to review the tool before meetings.
Fig. 2Descriptive results of validation survey questions for GPT.
Most participants found that the tool is most useful before guideline group meetings explaining what to expect at each phase. Participants thought that the tool was useful beforehand as a reference for orienting themselves to the structure of meetings, understanding the guideline development process, and what might be required of them. Respondents agreed that the tool serves as a reference for them to stay on track with the required tasks and to support structuring the process of guideline development.
4. Discussion
In this study, we developed a tool that supports guideline development by orienting various guideline group members to the tasks at hand and processes involved in guideline development, in particular when applying GRADE. The tool includes a comprehensive list of roles, responsibilities, and guidance for participation. This work will inform training and certification modules of the GIN-McMaster guideline credentialing and certification program to improve guideline quality (INGUIDE.org).
4.1 Strengths
The strengths of this study include the systematic process and stakeholder engagement through interviews and validation in real guideline group members. The iterative refinement presents another strength to arrive at a better final tool that has received input from many stakeholders across different settings and organizations. The combination of guideline developers with experience in chairing numerous guideline groups and guideline group members involved in development of the tool is a further area of strength. Given its successful validation with 3 guideline guideline groups, we believe this tool can be used by broad types of organizations involved in guideline development to train participants and to improve guideline group function.
4.2 Limitations
While we identified several limitations to the tool through the course of this study, we have modified the GPT to address these. There were areas of disagreement between respondents through interviews and in the validation survey (e.g., the tool being too long or too short). We tried to balance perspectives. While 69% of survey respondents were clear on the purpose of the tool in the validation survey, an important minorty was not. Therefore, we propose that implementation requires careful introduction of the purpose and intended use of the tool at the outset of a guideline process.
4.3 Implications for practice
The GPT presents guidance that guideline sponsoring organizations can use for the orientation of guideline group members, an area that requires further development. Participants for whom much of the advice in this tool would be most applicable, for example, more disruptive participants, may also be those individuals less likely to read and review the tool in the first place. We suggest that guideline sponsors ask for the review and understanding of the tool as a prerequisite to participating in a guideline group and that chairs refer to the GPT at various stages in the guideline group process. Given ongoing efforts by GIN and McMaster University to create a guideline development certification and credentialing program (INGUIDE.org), we believe use of the GPT together with the checklist for guideline group and development group chairs (https://heigrade.mcmaster.ca/guideline-development/) (Appendix 5) will support such efforts and is feasible and realistic.
4.4 Implications for research
We intend to further evaluate the GPT prospectively as it is more deeply incorporated into the guideline development process. We will prospectively seek feedback with each use of the tool to evaluate and create updated versions and we have built in a link to the validation survey in the tool itself. The current version of the tool is at: https://heigrade.mcmaster.ca/guideline-development/guideline-participants.
5. Conclusions
We have used a mixed-methods and iterative process to develop the GPT and participant role descriptions that will be used by multiple guideline groups developing GRADE guidelines. The GPT and participant roles will be incorporated into INGUIDE.org guideline participant training and credentialing, and we will continue to seek feedback and improve the tool going forward. We hope our efforts will assist with guideline participant orientation and guideline group process to ultimately create more effective health guidelines.
Centro Nazionale per l’Eccellenza Clinica, la Qualità e la Sicurezza delle Cure - Istituto Superiore di Sanita. Manuale metodologico per la produzione di linee guida di pratica clinica. https://snlg.iss.it/wp-content/uploads/2019/04/MM_v1.3.2_apr_2019.pdf. Accessed November 30, 2019.
31100-8&id=gr1.jpg){kind=link}
31100-8&id=gr2.jpg){kind=link}