Advertisement

Cochrane Qualitative and Implementation Methods Group guidance series—paper 2: methods for question formulation, searching, and protocol development for qualitative evidence synthesis

Published:December 14, 2017DOI:https://doi.org/10.1016/j.jclinepi.2017.10.023

      Abstract

      This paper updates previous Cochrane guidance on question formulation, searching, and protocol development, reflecting recent developments in methods for conducting qualitative evidence syntheses to inform Cochrane intervention reviews. Examples are used to illustrate how decisions about boundaries for a review are formed via an iterative process of constructing lines of inquiry and mapping the available information to ascertain whether evidence exists to answer questions related to effectiveness, implementation, feasibility, appropriateness, economic evidence, and equity. The process of question formulation allows reviewers to situate the topic in relation to how it informs and explains effectiveness, using the criterion of meaningfulness, appropriateness, feasibility, and implementation. Questions related to complex questions and interventions can be structured by drawing on an increasingly wide range of question frameworks. Logic models and theoretical frameworks are useful tools for conceptually mapping the literature to illustrate the complexity of the phenomenon of interest. Furthermore, protocol development may require iterative question formulation and searching. Consequently, the final protocol may function as a guide rather than a prescriptive route map, particularly in qualitative reviews that ask more exploratory and open-ended questions.

      Keywords

      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'

      Subscribe:

      Subscribe to Journal of Clinical Epidemiology
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect

      References

        • Harris J.
        Chapter 2: using qualitative research to develop robust effectiveness questions and protocols for Cochrane systematic reviews.
        in: Noyes J. Booth A. Hannes K. Harden A. Harris J. Lewin S. Supplementary guidance for Inclusion of Qualitative Research in Cochrane Systematic Reviews of Interventions. Cochrane Qualitative & Implementation Methods Group, 2011 (Version 1. Available at http://methods.cochrane.org/qi/supplemental-handbook-guidance. Accessed August 2011)
        • Booth A.
        Chapter 3: searching for studies.
        in: Noyes J. Booth A. Hannes K. Harden A. Harris J. Lewin S. Supplementary guidance for Inclusion of Qualitative Research in Cochrane Systematic Reviews of Interventions. Cochrane Collaboration Qualitative Methods Group, 2011 (Version 1. Available at http://methods.cochrane.org/qi/supplemental-handbook-guidance. Accessed August 2011)
        • Cochrane Qualitative & Implementation Methods Group
        Chapter 1: preparing a protocol for Cochrane intervention reviews including qualitative research.
        in: Noyes J. Booth A. Hannes K. Harden A. Harris J. Lewin S. Supplementary guidance for Inclusion of Qualitative Research in Cochrane Systematic Reviews of Interventions. Cochrane Qualitative & Implementation Methods Group, 2011 (Version 1. Available at http://methods.cochrane.org/qi/supplemental-handbook-guidance. Accessed August 2011)
        • Hessels L.K.
        • Van Lente H.
        Re-thinking new knowledge production: a literature review and a research agenda.
        Res Pol. 2008; 37: 740-760
        • Lavis J.
        • Davies H.
        • Oxman A.
        • Denis J.-L.
        • Golden-Biddle K.
        • Ferlie E.
        Towards systematic reviews that inform healthcare management and policy-making.
        J Health Services Res Policy. 2005; 10: 35-48
      1. Coemans,S., Wang Q., Hannes, K. 2015 The ultimate irony of developing an a priori review protocol for studying the literature on arts-based research methods in the area of community-based research. 3rd Conference on Arts Based Research and Artistic Research, Porto, Portugal, 28-29 January 2015.

        • Leeuw F.L.
        Reconstructing program theories: methods available and problems to be solved.
        Am J Eval. 2003; 24: 5-20
        • Kingdon J.W.
        Agendas, Alternatives, and Public Policies.
        Little, Brown, Boston1984
        • Stone D.A.
        Causal stories and the formation of policy agendas.
        Polit Sci Q. 1989; 104: 281-300
        • Peerally M.F.
        • Carr S.
        • Waring J.
        • Dixon-Woods M.
        The problem with root cause analysis.
        BMJ Qual Saf. 2017; 26: 417-422
        • Rwashana A.S.
        • Nakubulwa S.
        • Nakakeeto-Kijjambu M.
        • Adam T.
        Advancing the application of systems thinking in health: understanding the dynamics of neonatal mortality in Uganda.
        Health Res Policy Syst. 2014; 12: 36
        • SURE Collaboration, 2011
        SURE Guides for preparing and using evidence-based policy briefs: 3. Clarifying the problem.
        (Version 2.1 [updated November 2011]) The SURE Collaboration, 2011 (Available at) (Accessed July 27, 2017)
        • Booth A.
        • Noyes J.
        • Flemming K.
        • Gerhardus A.
        • Wahlster P.
        • van der Wilt G.J.
        • et al.
        Guidance on choosing qualitative evidence synthesis methods for use in health technology assessments of complex interventions.
        2016 ([Online]. Available at) (Accessed July 27, 2017)
        • Campbell M.
        • Egan M.
        • Lorenc T.
        • Bond L.
        • Popham F.
        • Fenton C.
        • et al.
        Considering methodological options for reviews of theory: illustrated by a review of theories linking income and health.
        Syst Rev. 2014; 3: 1-11
        • Campbell N.C.
        • Murray E.
        • Darbyshire J.
        • Emery J.
        • Farmer A.
        • Griffiths F.
        • et al.
        Designing and evaluating complex interventions to improve health care.
        BMJ. 2007; 334: 455-459
        • Rohwer A.
        • Booth A.
        • Pfadenhauer L.
        • Brereton L.
        • Gerhardus A.
        • Mozygemba K.
        • et al.
        Guidance on the use of logic models in health technology assessments of complex interventions.
        2016 ([Online]. Available at) (Accessed July 27, 2017)
        • Petticrew M.
        Time to rethink the systematic review catechism? Moving from ‘what works’ to ‘what happens’.
        Syst Rev. 2015; 4: 1
        • Hannes K.
        Building a case for mixed method reviews.
        in: Richards D.A. Hallberg I.R. Complex interventions in health: an overview of research methods. Taylor & Francis, Abingdon2015
        • Marshall M.N.
        Bridging the ivory towers and the swampy lowlands; increasing the impact of health services research on quality improvement.
        Int J Qual Health Care. 2014; 26: 1-5
        • Heyvaert M.
        • Hannes K.
        • Onghena P.
        Using mixed methods research synthesis for literature reviews.
        Sage, Thousand Oaks, CA2016
        • Oliver K.
        • Rees R.
        • Brady L.M.
        • Kavanagh J.
        • Oliver S.
        • Thomas J.
        Broadening public participation in systematic reviews: a case example involving young people in two configurative reviews.
        Res Synth Methods. 2015; 6: 206-217
        • Rycroft-Malone J.
        • McCormack B.
        • Hutchinson A.M.
        • DeCorby K.
        • Bucknall T.K.
        • Kent B.
        • et al.
        Realist synthesis: illustrating the method for implementation research.
        Implementation Sci. 2012; 7: 33
        • Harris J.
        • Croot L.
        • Thompson J.
        • Springett J.
        How stakeholder participation can contribute to systematic reviews of complex interventions.
        J Epidemiol Community Health. 2016; 70: 207
        • Baxter S.
        • Killoran A.
        • Kelly M.P.
        • Goyder E.
        Synthesizing diverse evidence: the use of primary qualitative data analysis methods and logic models in public health reviews.
        Public Health. 2010; 124: 99-106
        • Garside R.
        • Pearson M.
        • Moxham T.
        What influences the uptake of information to prevent skin cancer? A systematic review and synthesis of qualitative research.
        Health Educ Res. 2010; 25: 162-182
        • Anderson L.M.
        • Petticrew M.
        • Rehfuess E.
        • Armstrong R.
        • Ueffing E.
        • Baker P.
        • et al.
        Using logic models to capture complexity in systematic reviews.
        Res Synth Methods. 2011; 2: 33-42
        • Allmark P.
        • Baxter S.
        • Goyder E.
        • Guillaume L.
        • Crofton-Martin G.
        Assessing the health benefits of advice services: using research evidence and logic model methods to explore complex pathways.
        Health Social Care Community. 2013; 21: 59-68
        • Baxter S.K.
        • Blank L.
        • Woods H.B.
        • Payne N.
        • Rimmer M.
        • Goyder E.
        Using logic model methods in systematic review synthesis: describing complex pathways in referral management interventions.
        BMC Med Res Methodol. 2014; 14: 1
        • De Buck E.
        • Hannes K.
        • Van Remoortel H.
        • Govender T.
        • Vande Veegaete A.
        • Musekiwa A.
        • et al.
        Protocol: approaches to promote sanitation and handwashing behaviour change in low- and middle income countries: a mixed method systematic review. Campbell systematic reviews.
        2016 (Available at)
        • Mosler H.J.
        A systematic approach to behaviour change interventions for the water and sanitation sector in developing countries: a conceptual model, a review, and a guideline.
        Int J Environ Health Res. 2012; 22: 431-449
        • O'Neill J.
        • Tabish H.
        • Welch V.
        • Petticrew M.
        • Pottie K.
        • Clarke M.
        • et al.
        Applying an equity lens to interventions: using PROGRESS ensures consideration of socially stratifying factors to illuminate inequities in health.
        J Clin Epidemiol. 2014; 67: 56-64
        • Cargo M.
        • Stankov I.
        • Thomas J.
        • Saini M.
        • Rogers P.
        • Mayo-Wilson E.
        • et al.
        Development, inter-rater reliability and feasibility of a checklist to assess implementation (Ch-IMP) in systematic reviews: the case of provider-based prevention and treatment programs targeting children and youth.
        BMC Med Res Methodol. 2015; 15: 73
        • Rogers P.J.
        Using programme theory to evaluate complicated and complex aspects of interventions.
        Evaluation. 2008; 2008: 29-48
      2. Wong G, Greenhalgh T, Westhorp G, Pawson R, Development of methodological guidance, publication standards and training materials for realist and meta-narrative reviews: the RAMESES (Realist And Meta-narrative Evidence Syntheses – Evolving Standards) project. Health Services and Delivery Research, No. 2.30, 2014.

        • Rycroft-Malone J.
        • Kitson A.
        • Harvey G.
        • McCormack B.
        • Seers K.
        • Titchen A.
        • et al.
        Ingredients for change: revisiting a conceptual framework.
        Qual Saf Health Care. 2002; 11: 174-180
        • Lewin S.
        • Glenton C.
        • Munthe-Kaas H.
        • Carlsen B.
        • Colvin C.J.
        • Gülmezoglu M.
        • et al.
        Using qualitative evidence in decision making for health and social interventions: an approach to assess confidence in findings from qualitative evidence syntheses (GRADE-CERQual).
        PLoS Med. 2015; 12: e1001895
        • Pfadenhauer L.M.
        • Mozygemba K.
        • Gerhardus A.
        • Hofmann B.
        • Booth A.
        • Lysdahl K.B.
        • et al.
        Context and implementation: a concept analysis towards conceptual maturity.
        Z für Evidenz, Fortbildung Qualität im Gesundheitswesen. 2015; 109: 103-114
        • Hannes K.
        • Harden A.
        Multi-context versus context-specific qualitative evidence syntheses: combining the best of both.
        Res Synth Methods. 2011; 2: 271-278
        • Pearson A.
        • Wiechula R.
        • Court A.
        • Lockwood C.
        The JBI model of evidence-based healthcare.
        Int J Evidence-Based Healthc. 2005; 3: 207-215
        • Welch V.A.
        • Petticrew M.
        • O'Neill J.
        • Waters E.
        • Armstrong R.
        • Bhutta Z.A.
        • et al.
        Health equity: evidence synthesis and knowledge translation methods.
        Systematic reviews. 2013; 2: 1
        • Rashidian A.
        • Shakibazadeh E.
        • Karimi-Shahanjarini A.
        • Glenton C.
        • Noyes J.
        • Lewin S.
        • et al.
        Barriers and facilitators to the implementation of doctor-nurse substitution strategies in primary care: qualitative evidence synthesis.
        Cochrane Database Syst Rev. 2013; https://doi.org/10.1002/14651858.CD010412
        • Booth A.
        • Sutton A.
        • Papaioannou D.
        Systematic Approaches to a Successful Literature Review.
        2nd ed. Sage, London2016
        • Glenton C.
        • Lewin S.
        • Scheel I.B.
        Still too little qualitative research to shed light on results from reviews of effectiveness trials: a case study of a Cochrane review on the use of lay health workers.
        Implement Sci. 2011; 6: 53
        • Noyes J.
        • Hendry M.
        • Lewin S.
        • Glenton C.
        • Chandler J.
        • Rashidian A.
        Qualitative “trial-sibling” studies and“ unrelated” qualitative studies contributed to complex intervention reviews.
        J Clin Epidemiol. 2016; 74: 133-143
        • Noyes J.
        • Hendry M.
        • Booth A.
        • Chandler J.
        • Lewin S.
        • Glenton C.
        • et al.
        Current use and Cochrane guidance on selection of social theories for systematic reviews of complex interventions.
        J Clin Epidemiol. 2016; 75: 78-92
        • Lorenc T.
        • Pearson M.
        • Jamal F.
        • Cooper C.
        • Garside R.
        The role of systematic reviews of qualitative evidence in evaluating interventions: a case study.
        Res Synth Methods. 2012; 3: 1-10
      3. Oliver S, Kavanagh J, Caird J, Lorenc T, Oliver K, Harden A. et al. Health promotion, inequalities and young people’s health: a systematic review of research. 2008. London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London.

        • Pfadenhauer L.
        • Rohwer A.
        • Burns J.
        • Booth A.
        • Lysdahl K.B.
        • Hofmann B.
        • et al.
        Guidance for the assessment of context and implementation in health technology assessments (HTA) and systematic reviews of complex interventions: the context and implementation of complex interventions (CICI) framework [online].
        (Available at) (Accessed July 27, 2017)
        • Stansfield C.
        • Brunton G.
        • Rees R.
        Search wide, dig deep: literature searching for qualitative research. An analysis of the publication formats and information sources used for four systematic reviews in public health.
        Res Synth Methods. 2014; 5: 142-151
        • Papaioannou D.
        • Sutton A.
        • Carroll C.
        • Booth A.
        • Wong R.
        Literature searching for social science systematic reviews: consideration of a range of search techniques.
        Health Inf Libraries J. 2010; 27: 114-122
        • Hessels L.K.
        • Van Lente H.
        • Smits R.
        In search of relevance: the changing contract between science and society.
        Sci Public Policy. 2009; 36: 387-401
        • Booth A.
        • Harris J.
        • Croot E.
        • Springett J.
        • Campbell F.
        • Wilkins E.
        Towards a methodology for cluster searching to provide conceptual and contextual “richness” for systematic reviews of complex interventions: case study (CLUSTER).
        BMC Med Res Methodol. 2013; 13: 118
        • Lorenc T.
        • Clayton S.
        • Neary D.
        • Whitehead M.
        • Petticrew M.
        • Thomson H.
        • et al.
        Crime, fear of crime, environment, and mental health and wellbeing: mapping review of theories and causal pathways.
        Health & Place. 2012; 18: 757-765
        • Booth A.
        • Carroll C.
        Systematic searching for theory to inform systematic reviews: is it feasible? Is it desirable?.
        Health Inf Libraries J. 2015; 32: 220-235
        • Pound P.
        • Campbell R.
        Locating and applying sociological theories of risk-taking to develop public health interventions for adolescents.
        Health Sociol Rev. 2015; 24: 64-80
        • Finfgeld-Connett D.
        • Johnson E.D.
        Literature search strategies for conducting knowledge-building and theory-generating qualitative systematic reviews.
        J Adv Nurs. 2013; 69: 194-204
        • Barroso J.
        • Gollop C.J.
        • Sandelowski M.
        • Meynell J.
        • Pearce P.F.
        • Collins L.J.
        The challenges of searching for and retrieving qualitative studies.
        West J Nurs Res. 2003; 25: 153-178
        • Booth A.
        Searching for qualitative research for inclusion in systematic reviews: a structured methodological review.
        Syst Rev. 2016; 5: 74
        • Flemming K.
        • Briggs M.
        Electronic searching to locate qualitative research: evaluation of three strategies.
        J Adv Nurs. 2007; 57: 95-100
        • Gorecki C.A.
        • Brown J.M.
        • Briggs M.
        • Nixon J.
        Evaluation of five search strategies in retrieving qualitative patient-reported electronic data on the impact of pressure ulcers on quality of life.
        J Adv Nurs. 2010; 66: 645-652
        • Pearson M.
        • Moxham T.
        • Ashton K.
        Effectiveness of search strategies for qualitative research about barriers and facilitators of program delivery.
        Eval Health professions. 2011; 34: 297-308
        • Tong A.
        • Flemming K.
        • McInnes E.
        • Oliver S.
        • Craig J.
        Enhancing transparency in reporting the synthesis of qualitative research: ENTREQ.
        BMC Med Res Methodol. 2012; 12: 1
        • Moher D.
        • Liberati A.
        • Tetzlaff J.
        • Altman D.G.
        • PRISMA Group
        Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement.
        Int J Surg. 2010; 8: 336-341
        • Booth A.
        “Brimful of STARLITE”: toward standards for reporting literature searches.
        J Med Libr Assoc. 2006; 94: 421-429
        • Shaw R.L.
        Conducting literature reviews.
        in: Forrester M.A. Doing qualitative research in psychology: A practical guide. Sage, London2010: 39-56
        • Denyer D.
        • Tranfield D.
        Producing a systematic review.
        in: Buchanan D.A. Bryman A. The SAGE handbook of organizational research methods. SAGE Publications Ltd, London2009
        • Major C.H.
        • Savin-Baden M.
        Designing the synthesis.
        in: Major C.H. Savin-Baden M. An introduction to qualitative research synthesis: managing the information explosion in social science research. Routledge, London2010: 43-55
        • Stern C.
        • Jordan Z.
        • McArthur A.
        Developing the review question and inclusion criteria.
        Am J Nurs. 2014; 114: 53-56
        • Petticrew M.
        • Roberts H.
        How to find the studies: the literature search.
        in: Petticrew M. Roberts H. Systematic reviews in the social sciences: A practical guide. Blackwell Publishing, Oxford2006
        • Booth A.
        Clear and present questions: formulating questions for evidence based practice.
        Libr Hi Tech. 2006; 24: 355-368
        • Cooke A.
        • Smith D.
        • Booth A.
        Beyond PICO: the SPIDER tool for qualitative evidence synthesis.
        Qual Health Res. 2012; 22: 1435-1443
        • Jordan J.
        • Rose L.
        • Dainty K.N.
        • Noyes J.
        • Clarke S.
        • Blackwood B.
        Factors that impact on the use of mechanical ventilation weaning protocols in critically ill adults and children: a qualitative evidence-synthesis.
        Cochrane Database Syst Rev. 2015; https://doi.org/10.1002/14651858.CD009851.pub2
        • Stansfield C.
        • Kavanagh J.
        • Rees R.
        • Gomersall A.
        • Thomas J.
        The selection of search sources influences the findings of a systematic review of people's views: a case study in public health.
        BMC Med Res Methodol. 2012; 12: 1
        • Ravitch S.M.
        • Riggan M.
        Reason & rigor: How conceptual frameworks guide research.
        Sage Publications, Thousand Oaks, CA2016
        • Husk K.
        • Lovell R.
        • Cooper C.
        • Garside R.
        Participation in environmental enhancement and conservation activities for health and well-being in adults (Protocol).
        Cochrane Database Syst Rev. 2013; https://doi.org/10.1002/14651858.CD010351
        • Hurley M.
        • Dickson K.
        • Walsh N.
        • Hauari H.
        • Grant R.
        • Cumming J.
        • et al.
        Exercise interventions and patient beliefs for people with chronic hip and knee pain: a mixed methods review (Protocol).
        Cochrane Database Syst Rev. 2013; : CD010842
        • Bohren M.A.
        • Munthe-Kaas H.
        • Berger B.O.
        • Allanson E.E.
        • Tunçalp Ö.
        Perceptions and experiences of labour companionship: a qualitative evidence synthesis.
        Cochrane Database Syst Rev. 2016; https://doi.org/10.1002/14651858.CD012449
        • SURE Collaboration, 2011
        SURE Guides for preparing and using evidence-based policy briefs: 5. Identifying and addressing barriers to implementing policy options.
        (Version 2.1 [updated November 2011]) The SURE Collaboration, 2011 (Available at http://www.who.int/evidence/sure/guides)
        • Guyatt G.H.
        • Oxman A.D.
        • Vist G.E.
        • Kunz R.
        • Falck-Ytter Y.
        • Alonso-Coello P.
        • et al.
        GRADE: an emerging consensus on rating quality of evidence and strength of recommendations.
        BMJ. 2008; 336: 924-926
        • Downe S.
        • Finlayson K.
        • Tunçalp Ö.
        • Gülmezoglu A.M.
        Factors that influence the uptake of routine antenatal services by pregnant women: a qualitative evidence synthesis (Protocol).
        Cochrane Database Syst Rev. 2016; : CD012392