If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Corresponding author. Evidence Synthesis Group, Population Health Sciences Institute, Faculty of Medical Sciences, Newcastle University, Newcastle upon Tyne, UK.
Cochrane Incontinence, Population Health Sciences Institute, Faculty of Medical Sciences, Newcastle University, Newcastle upon Tyne, UKEvidence Synthesis Group, Population Health Sciences Institute, Faculty of Medical Sciences, Newcastle University, Newcastle upon Tyne, UK
Evidence Synthesis Group, Population Health Sciences Institute, Faculty of Medical Sciences, Newcastle University, Newcastle upon Tyne, UKNIHR Innovation Observatory, Newcastle University, Newcastle upon Tyne, UK
Cochrane Incontinence, Population Health Sciences Institute, Faculty of Medical Sciences, Newcastle University, Newcastle upon Tyne, UKEvidence Synthesis Group, Population Health Sciences Institute, Faculty of Medical Sciences, Newcastle University, Newcastle upon Tyne, UK
Evidence Synthesis Group, Population Health Sciences Institute, Faculty of Medical Sciences, Newcastle University, Newcastle upon Tyne, UKNIHR Innovation Observatory, Newcastle University, Newcastle upon Tyne, UK
The project aimed to rapidly identify priority topic uncertainties as a first step to identify future systematic review questions of pertinence to key international fecal incontinence (FI) stakeholders (patients, carers, health care professionals, policy makers and voluntary, community, or social enterprise representatives). The paper’s aim is to share our methods, experience, and learning with other groups planning to deliver a rapid priority setting exercise.
Study Design and Setting
An evidence gap map incorporated three evidence streams: emerging evidence identified through horizon scanning; existing evidence identified through systematic searches of bibliographic databases; and FI stakeholder insights collected through an international survey. The evidence gap map was presented during an online workshop with stakeholders, where they shared their expertize to expand, refine, and rank topic uncertainties using ideation techniques, focus group discussions, consensus techniques, and online polling.
Results
The multistep methods used to deliver this priority setting exercise resulted in identification of broad priority topic uncertainties. The methods appear to have high acceptability and engagement with participants but await full evaluation.
Conclusion
This project successfully followed robust methodology, building upon frameworks from published priority setting and evidence gap mapping projects while incorporating strong patient and public involvement components.
Incorporating horizon scanning into priority-setting exercises and displaying results alongside existing evidence and stakeholder insights in an evidence map helps to facilitate knowledge exchange with key stakeholders and identify gaps in research.
What this adds to what is known?
•
An existing body of literature explores and develops methods to conduct priority-setting exercises. This article aims to provide detailed information about the steps and tools used to incorporate different types of evidence into an evidence map to facilitate stakeholder participation. We hope this article serves as a blueprint for others who plan to run priority-setting exercises.
What is the implication?
•
We encourage other groups planning to conduct a priority-setting exercise to include an evidence map to ensure a comprehensive assessment of the evidence base and facilitate stakeholder knowledge exchange.
1. Introduction
Cochrane is an international network that produces high-level evidence to support health care decision-making. It strives for relevance by undertaking a systematic approach to selecting research questions for its systematic reviews. This article describes the novel rapid priority setting exercise that Cochrane incontinence commissioned. The project acted as the first step identifying priority topic uncertainties for future systematic review questions relevant to key international fecal incontinence (FI) stakeholders (patients, carers, health care professionals, policy makers and voluntary, community, or social enterprise representatives).
Since the James Lind Alliance was established in 2004 [
], an emphasis on formalized approaches for priority setting has emerged, particularly the inclusion of end-user feedback to increase the likelihood of producing reviews that positively affect health outcomes. To date, there is no gold standard priority setting method; many published articles vary in scope and breadth [
]. A study which assessed the presence and effectiveness of priority setting systems used for Cochrane reviews found heterogeneity in the methods and processes leading the authors to recommend increasing inclusivity among all relevant stakeholders and being transparent in the methods used to improve quality [
Rapid methods are increasingly undertaken to ensure timely identification and delivery of health research priorities, and there are examples of their successful use [
]. However, there is no universally accepted rapid approach and methods that may include single screening of topic uncertainties, audio transcription of qualitative components with single coding rather than verbatim transcription and double data coding [
FI is defined as the involuntary passage of fecal material through the anal canal, is estimated to affect between 2% and 17% of community dwelling individuals and can significantly impact a person's health and quality of life [
]. To our knowledge, there are two priority setting exercises completed, whereby the authors identified priority areas neither was rapid nor open to all patients [
The primary aim of this project was to engage with key international stakeholders to rapidly identify topic uncertainties, rank them and gather broad insight on those deemed most important, see Box 1 for the research objectives.
List of the research objectives for this priority setting exercise.
1.
To understand the current demand of existing reviews in the Cochrane incontinence portfolio.
2.
To examine the volume and range of systematic review publications.
3.
To apply horizon scanning techniques to identify emerging evidence (pipeline (early stage) or ongoing trials) of drugs, health technologies, medical devices, or diagnostics.
4.
To identify topic uncertainties of international FI stakeholders.
5.
To synthesize findings from objectives one to four in an evidence gap map to identify gaps and unmet needs.
6.
To engage in knowledge exchange, use ideation and consensus techniques to identify further topic uncertainties and rank those identified.
7.
To explore the highest-ranked priority topic uncertainties in more depth through facilitated group discussion.
In this paper, we focus on providing information about methods used throughout the rapid priority setting exercise to provide a blueprint for others undertaking similar exercises. Detailed information about the results and horizon scanning methods to identify emerging evidence of drugs, health technologies, medical devices, or diagnostics are published elsewhere [
A steering committee advised on the scope of the priority setting exercise (comprising; Cochrane incontinence editorial staff, clinical specialists, systematic review, priority setting, and patient and public involvement methodologists).
We adapted the five-stage framework developed by Batchelor et al. [
The Eczema Priority Setting Partnership: a collaboration between patients, carers, clinicians and researchers to identify and prioritize important research questions for the treatment of eczema.
], multistep process, described as: 1-2) generation and prioritization of topics, 3) mapping of topics to existing evidence, and 4) reflection of the relative importance with stakeholders. Then we used Anstee's et al. [
Abrams P. Cardozo L. Wagg A. Wein A. Incontinence: 6th International Consultation on Incontinence. ICI-ICS. International Continence Society,
Bristol, UK2017
] and mapping our evidence and key international FI stakeholder insights against them. We incorporated rapid review methodology while undertaking stages 2 b, 3 b, and 4 b [
The Eczema Priority Setting Partnership: a collaboration between patients, carers, clinicians and researchers to identify and prioritize important research questions for the treatment of eczema.
2.1 Stage 1a–initiation–identifying key international FI stakeholders
To ensure transparency, our intention to conduct a priority setting exercise and the protocol were published on the Cochrane incontinence website.
Patient and public involvement took place at two time points. Involvement was at an influencing level as described within the Authors and Consumers Together Impacting on eVidencE Framework continuum of involvement [
]. To enable the best engagement levels, successful awareness, and recruitment, we contacted 27 organizations and other individuals (Supplementary File 1). We strived to involve people who have FI or care for someone with FI, nurses, doctors, and allied health care professionals across geographic and economic settings by personally inviting stakeholders working in diverse areas to participate. Figure 4 illustrates the flow of engagement and participation at each stage.
2.2 Stage 1b scoping
Preparatory scoping to assess the breadth of literature was undertaken to plan sufficient staffing and resources.
2.3 Stage 2a consultation of stakeholder insights - survey
This stage identified broad topic uncertainties by asking adult respondents “What fecal incontinence topic(s) or question(s) would you like Cochrane incontinence to prioritize over the next 5 years?”. To ensure wide geographic coverage this survey was disseminated online via the Qualtrics platform on 17th August 2020 (Supplementary File 2) to 27 organizations identified during stage 1a (Supplementary File 1).
Respondents were provided with a project summary and then asked for explicit consent to participate. The survey text was adapted from a Cochrane skin prioritization project and reviewed for readability [
]. Participants were asked to submit five priority topic areas and to place them in order of importance. The survey was open for 2 weeks, and once responses were collected (68 responses at closing), they were reviewed by two members of the research team and mapped to treatment domains along with other evidence streams in a process described in stage 3a.
2.4 Stage 2b identifying existing evidence
An information specialist identified existing systematic reviews by searching eight electronic databases in July 2020 [
]. Two reviewers screened the titles and abstracts in a nonblinded manner, a third reviewer resolved conflicts. All systematic reviews meeting four of five minimum quality criteria used by the Database of Abstracts of Reviews of Effects were included in the study [
In addition, to assess the most utilized reviews and to highlight where Cochrane reviews already exist, we assessed usage of 13 Cochrane systematic reviews that investigate FI treatment and management by reviewing internally available accession scores from Wiley, similar usage data can be collected from Web of Science [
2.5 Stage 3a collation of key international FI stakeholder insights - survey
Two researchers reviewed responses collected from the online survey. Responses pertaining to topic uncertainties were charted against preidentified domains, checked for accuracy, and then mapped using EPPI-Reviewer Web and EPPI-Mapper [
Evidence for Policy and Practice Information and Co-ordinating Centre, Social Science Research Unity at the Institute of Education, University of London,
London, UK2021
2.6 Stage 3b horizon scanning for emerging evidence
To complement stage 2b and provide a comprehensive overview of the research landscape we undertook a horizon scan to identify emerging evidence (pipeline (early stage) or ongoing trials) of drugs, health technologies, medical devices or diagnostics by performing a search across clinical trial registries using the National Institute for Health and Care Research Innovation Observatory's ScanMedicine database in August 2020 [
]. Two independent reviewers checked records identified for eligibility.
2.7 Stage 4 combining stakeholder insights with evidence in a gap map
Information from the three streams of evidence: existing evidence (systematic reviews); emerging evidence (pipeline and ongoing clinical trials) and stakeholder insights (survey), were presented visually using an interactive evidence gap map (Fig. 2) and can be found on the Cochrane Incontinence website. The map was created using Evidence for Policy and Practice Information (EPPI)-reviewer Web and EPPI-mapper [
Evidence for Policy and Practice Information and Co-ordinating Centre, Social Science Research Unity at the Institute of Education, University of London,
London, UK2021
Abrams P. Cardozo L. Wagg A. Wein A. Incontinence: 6th International Consultation on Incontinence. ICI-ICS. International Continence Society,
Bristol, UK2017
]. The size of the circles in the evidence gap map are proportional to the amount of information available and details of each study can be viewed by clicking on a circle in the online interactive version.
High quality, systematic reviews already exist, and new trials would not add value to the existing evidence base.
•
Systematic reviews exist, and there are new ongoing trials.
•
No systematic reviews exist, but there are new ongoing trials.
•
Survey participants' submitted topic uncertainties indicating they considered the topic important.
2.8 Stage 5: key international FI stakeholder insights workshop
The workshop, held on 17th of November 2020, aimed to facilitate knowledge exchange between stakeholders and the research team about the current evidence to clarify, refine, and rank topic uncertainties.
We used an open snowballing recruitment strategy publishing an expression of interest form on international networks (Supplementary File 1). In addition, embedded within the online stakeholder survey was a question asking respondents if they would be interested in taking part in the workshop, and 57 individuals indicated their interest. Both forms asked what perspective they were answering from (individual patients, patient group, clinician, or health care professional) and where participants indicated ‘yes’, the information was used to ensure people with diverse perspectives were invited.
Due to the COVID-19 pandemic, the intended face-to-face workshop was held as a 3.5 hour online event using the platform Zoom. Participants were formally invited by email 3 weeks before the scheduled workshop, with 21 individuals from seven countries attending (Fig. 4). The invitation included a description of the minimum IT requirements and a question enquiring if accessibility arrangements were required. A ‘technology drop-in session’ was run so that people could test connectivity and explore the functionality of software beforehand.
Before the workshop eligible participants were provided with:
•
Briefing note that described; the aims and outcomes of the day, educational material explaining what a systematic review is, participation requirements, and contact details.
•
The final schedule for the workshop.
•
Housekeeping rules and a screenshot of the basic Zoom meeting screen flagging how to mute/unmute your microphone, switch video on/off, ‘raise hand’/react, send a chat message, etc.
•
Prereading material–narrative discussion of the priority topic uncertainties identified in the survey, mapped to the existing and emerging evidence and an electronic file of the evidence gap map (Supplementary File 3).
The schedule of the workshop was adapted from that described in of Synnot et al. [
Selecting, refining and identifying priority Cochrane Reviews in health communication and participation in partnership with consumers and other stakeholders.
2.8.1 Review of priority topic uncertainties and evidence to date
Introductions and ice breaker activities were undertaken to establish a climate of trust between attendees, promote good communication and active listening.
To facilitate knowledge exchange, participants were shown the top level of the evidence gap map followed by an overview of the usage of reviews published by Cochrane incontinence. To orientate participants with the research landscape, they were given a focused presentation of the 20 commonly indicated topic uncertainties in the survey, using a “traffic light system” describing types of evidence relevant to each topic uncertainty. Opportunities were given at regular intervals for participants to ask questions about the information being presented.
2.8.2 Clarification, expansion, and generation of further topic uncertainties
‘Padlet’, online software mimicking a board of “post-it notes”, was used to capture brainstorming ideation. Topic uncertainties identified up to this point could be amended or added to based on workshop attendee's updated knowledge of available research evidence, an example of Padlet is shown in Fig. 3.
Fig. 3A screenshot of Padlet in use during the workshop.
Mentimeter, anonymous interactive polling software, was used to rank topic uncertainties. Topic uncertainties were taken from the Padlet “post-it notes” then uploaded alongside those previously discussed to Mentimeter. Workshop participants were each allocated 100 points and told to split their points between as many or as few topic uncertainties as they liked. The topic uncertainties were then ranked based upon all participants' votes.
2.8.4 Breakout session - explore and refine
The six highest-ranked priority topic uncertainties were explored in more depth through small group discussions. Each of the three breakout groups consisted of two facilitators and 5-7 participants were carefully selected to include three to four health care professionals and three to four patients, carers, or individuals from relevant third sector organizations. Each group discussed two of the six priority topic uncertainties. Participants consented to audio recording via Zoom, and one facilitator made handwritten notes and monitored the Zoom chat function. Although facilitators were present in each group, facilitation was undertaken neutrally. The sessions followed an unstructured format due to the successful group dynamic and the rich conversational exchanges that emerged. Intermittent loose prompts were given to guide the discussion back to the relevant topic or to explore participants' comments. This unstructured, neutral facilitation approach deviated from the protocol, which stipulated use of the Systematic Reviews on Health Policy and Systems Research tool and a directive facilitation approach [
Following the breakout sessions, feedback from each group was given to the main room, and the participants were invited to discuss any points raised, expanding, or challenging as they saw fit. Participants were informed of the next steps of the project before the workshop close.
2.9 Rapid qualitative analysis of breakout room discussion
The audio recordings for individual breakout groups were transcribed independently by three researchers (FP, NOC, and KT) using rapid intelligent verbatim transcription [
]. Following transcription, one researcher coded the data iteratively in Excel, abstracting and organizing concepts into broad themes, seeking cross-cutting commonalities between all three groups. The identified codes and themes were reviewed and agreed upon iteratively by two other researchers.
3. Reflections
To evaluate the acceptability of our methods, we disseminated an anonymous online evaluation, eight out of 21 participants responded with a further seven sending feedback via e-mail. While acknowledging the response rate to the survey was poor, we will discuss feedback below, along with the research team and steering committee's observations.
3.1 Evidence synthesis
Our methodological approach struck a reasoned balance between rigor and efficiency or rapidity. Our searches were wide-ranging and included an extensive database search with no date restrictions. In addition, screening and charting were undertaken in blinded duplicate or checked in full by a second unblinded reviewer. However, we did restrict the mapping of systematic reviews and papers to those published in English or easily translated by Google Translate (https://translate.google.co.uk/). Consequently, 12 articles were not included.
To give a rapid, representative overview of the research landscape, we only included and mapped systematic reviews of interventions and the primary studies included within them. Consequently, it is possible this project did not capture primary studies that may have been published outside of the included systematic reviews development timescale or inclusion criteria. Publication bias may affect the primary studies being captured. In line with mapping review methods, we did not assess the quality of included studies. We also completed our horizon scan with a reduced number of sources in order to expedite the process. As medical devices, digital, and diagnostics do not follow the same regulatory process as medicines, it is likely innovative products of this type were missed.
Other published literature on priority setting for systematic reviews have reported using ‘horizon scanning’ [
]. Specifically, our search source, ScanMedicine, facilitates wider coverage of emerging evidence by identifying early signals of new and active areas in the field, compared to other databases, such as, MEDLINE and Embase, which predominantly cover full text publications. Using ScanMedicine proved to be a time- efficient method. Additionally, and perhaps most importantly, few priority setting exercises seem to incorporate and compare both existing and emerging evidence.
3.2 Patient and public involvement and recruitment
Our recruitment strategy succeeded in ensuring diversity of geography and perspective among s the participants. Twenty-one people were recruited from seven countries, two from low-middle income countries. Eleven participants were clinicians, and health care professionals or policy makers and ten were patients, carers, or voluntary community and social enterprise representatives providing equal representation between the two major stakeholder groupings. Moving the workshop online due to the COVID-19 pandemic meant it was possible to attract a wider variety of international FI stakeholders, 53% of whom were from outside the UK; this gave rise to a valuable discussion of the differences in health care systems and models of care [
]. This engagement of patients, carers, and health care professionals from diverse sectors, geographical regions, disciplines, and demographics facilitated an engaging discussion and a rich synthesis of the priority topic uncertainties.
However, our principles for equity, diversity, and inclusion in public involvement and engagement could have been applied to this work in a stronger manner. We only successfully recruited two individuals from low-middle income countries; therefore, it is unlikely we truly captured the views of people from disadvantaged and low- and–middle-income countries. In discussions about the impact of cultural differences, some of the focus group members felt they had limited knowledge. However, the final workshop was likely far more diverse from an international viewpoint than it would have been if the original face-to-face workshop had taken place. We strongly advocate that future exercises adopt the equity lens to ensure inclusion of priority topics from people in different sociodemographic groups, specifically those in disadvantaged populations [
Overall, the project team found consolidating a large body of evidence from multiple sources in a visual tool made it easier to facilitate knowledge exchange with the participants. In addition, it also increased transparency between the project team and stakeholders when referring to the evidence. The evaluative feedback indicated participants, too, found it a valuable inclusion ‘the tool was very informative and helpful’ and several clinicians openly referred to it during discussion. However, one participant found it difficult to navigate and would have liked more granular instructions included in the prereading, indicating there may be a difference in understanding and usability of the evidence gap maps between different stakeholders which we plan to evaluate in future projects.
Reviewing other priority setting exercises for systematic reviews, leads us to believe organizing the information gathered into an evidence gap map was a novel method [
]. We would encourage future groups to evaluate its utility with a wide range of stakeholders. To maximize its usefulness, we advise sending practical instructional guidance to participants before the workshop.
•
Prereading material
All materials were reviewed internally before dissemination, however, not by patient representatives. Doing so would have allowed us to evaluate usability and understanding of the evidence gap map. We included a Cochrane UK developed video (‘What are systematic reviews?’) to explain what a systematic review is, and the feedback demonstrated this was useful to one participant noting they got the most out of “the explanation process of systematic reviews video."
•
Technology and software
100% of the people who completed the evaluation found Zoom easy to use, 63% found Padlet easy to use, and 88% found Mentimeter easy to use.
Padlet and Mentimeter were selected for use because, to our knowledge, at the time of the workshop, they were the only software that could be instantly accessed without creating a log-in and we wanted to reduce the burden of participation.
Our evaluation form did not specifically measure the usefulness of the ‘tech drop-in session’ but in the evaluation of a recent workshop run as part of the innovation observatory's ‘imagine series’ [
] which used the same ‘tech drop in sessions’ a participant flagged “This was my first attempt at Zoom so I am flushed with triumph! Apart from hearing all the interesting contributions today, the fact that I was enabled with your help to participate in a Zoom meeting at all means I can now use Zoom technology to help open up prospects for improving my daily living”. This step should be implemented as good practice and helps widen participation.
3.4 Stakeholder satisfaction
All workshop participants who completed the evaluation reported being satisfied with their experience, several people noted the helpfulness of the prereading materials and felt the researchers had successfully tailored the sessions to a mixed audience. Several participants would have liked to have heard a more comprehensive summary of findings during the final wrap-up stage of the workshop with details of more specific conclusions rather than a general discussion. They suggested using the note-taking whiteboard system throughout the breakout session to help refine and explore topic-specific points in more detail.
3.5 The priority topic uncertainties
The topics arising from the workshop were education, impact, and burden of living with FI, psychological support, health care service improvements, stigma and treatment, and management [
]. The immediate project team comprised methodologists who had no prior expectations about specific topics. However, clinical members of the steering committee did not express surprise at the findings. Particularly the desire for patients to receive psychological support and the need to address the stigma of having FI as this mirrored their clinical experience and published literature [
]. The topics were broadly in keeping with areas highlighted previously; however it is likely the inclusion of patients, carers, nurses, and other allied health professionals gave a greater voice to nonclinical interventions, which have historically been lower in prioritized lists. For example, the priorities identified previously by Whitehead et al. [
], were predominantly guided by hospital physicians (although patient advocates were also asked) who identified the importance of randomized clinical trials for the treatment of FI. Continence nursing professionals, however, highlighted the need to focus on continence literacy and communication, and the dissemination of evidence-based management interventions, key findings of the priority setting exercise detailed here [
]. Future priority setting exercises should explicitly ask workshop attendees whether the topics identified were as expected to help better understand the added benefit of combining the multiple evidence synthesis approaches suggested here.
The information collected during this priority setting exercise gave us a broad understanding of the areas which international FI stakeholders felt were important. Recurring themes suggests we had hit saturation. Due to the broad topics identified, the next step for this project is to host a multistakeholder session to define a range of research questions to fulfill the priority topic uncertainties.
4. Conclusion
Throughout the planning and conduct, we strived to follow the best practices described in the literature, such as inclusivity, transparency, and rigor [
]. The recruitment strategy and online delivery successfully attracted many international FI stakeholders and this would not have been the case had the meeting taken place face-to-face. Upon evaluation, the tools used before and during the workshop seem appropriate and appear to have high acceptability to participants. The incorporation of rapid methods, such as single screening and coding, demonstrated possibilities to save time.
The identification of emerging technologies, including devices and medicines, through horizon scanning for pipeline (early stage) or ongoing trials existing trial reports, high-level systematic review evidence and topic uncertainties submitted by survey respondents in an evidence gap map facilitated detailed comparisons, this was particularly novel and not something we identified in other published priority setting exercises. The evidence gap map was a useful component to visualize the research landscape and stakeholder insight (survey) and display our findings comprehensively and transparently to encourage participation.
The information collected during the priority setting exercise gave us a broad understanding of the areas which key international FI stakeholders felt were important. As noted above, a second phase is planned to explore the topics in more detail and achieve a consensus on deliverable research and innovation.
Acknowledgments
We would like to thank all those who participated in the workshop and completed the online survey–your help was invaluable. We would also like to thank KSR, who kindly gave permission for the use of their database, KSR Evidence, during a free trial period.
The Eczema Priority Setting Partnership: a collaboration between patients, carers, clinicians and researchers to identify and prioritize important research questions for the treatment of eczema.
Abrams P. Cardozo L. Wagg A. Wein A. Incontinence: 6th International Consultation on Incontinence. ICI-ICS. International Continence Society,
Bristol, UK2017
Evidence for Policy and Practice Information and Co-ordinating Centre, Social Science Research Unity at the Institute of Education, University of London,
London, UK2021
Selecting, refining and identifying priority Cochrane Reviews in health communication and participation in partnership with consumers and other stakeholders.
In line with the guidance for authors, which states the highlights are optional due to being out of office, for multiple team members we will not proceed with the inclusion of highlights at this time but hope to do so for future endeavors.
Declaration of interests: The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Conflicts of interest: KT, NO'C, SG, SJ, SW, FP: None.
This study was supported by the National Institute for Health Research (NIHR), UK, via a collaboration between Cochrane Incontinence [NIHR129464 Cochrane Incontinence Group] and the Innovation Observatory [NIHRIO/project reference HSRIC-2016-10009].
00308-0&id=gr1.jpg){kind=link}
00308-0&id=gr2.jpg){kind=link}
00308-0&id=gr3.jpg){kind=link}
00308-0&id=gr4.jpg){kind=link}