Advertisement

Cochrane Qualitative and Implementation Methods Group guidance series—paper 4: methods for assessing evidence on intervention implementation

Published:December 06, 2017DOI:https://doi.org/10.1016/j.jclinepi.2017.11.028

      Abstract

      Objectives

      This article provides reviewers with guidance on methods for identifying and processing evidence to understand intervention implementation.

      Study Design and Setting

      Strategies, tools, and methods are applied to the systematic review process to illustrate how process and implementation can be addressed using quantitative, qualitative, and other sources of evidence (i.e., descriptive textual and nonempirical).

      Results

      Reviewers can take steps to navigate the heterogeneity and level of uncertainty present in the concepts, measures, and methods used to assess implementation. Activities can be undertaken in advance of a Cochrane quantitative review to develop program theory and logic models that situate implementation in the causal chain. Four search strategies are offered to retrieve process and implementation evidence. Recommendations are made for addressing rigor or risk of bias in process evaluation or implementation evidence. Strategies are recommended for locating and extracting data from primary studies. The basic logic is presented to assist reviewers to make initial review-level judgments about implementation failure and theory failure.

      Conclusion

      Although strategies, tools, and methods can assist reviewers to address process and implementation using quantitative, qualitative, and other forms of evidence, few exemplar reviews exist. There is a need for further methodological development and trialing of proposed approaches.

      Keywords

      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'

      Subscribe:

      Subscribe to Journal of Clinical Epidemiology
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect

      References

        • Pfadenhauer L.M.
        • Mozygemba K.
        • Gerhardus A.
        • Hofmann B.
        • Booth A.
        • Lysdahl K.B.
        • et al.
        Context and implementation: a concept analysis towards conceptual maturity.
        Z Evid Fortbild Qual Gesundhwes. 2015; 109: 103-114
        • Rossi P.H.
        • Lipsey M.W.
        • Freeman H.E.
        Evaluation: a systematic approach.
        7th ed. Sage Publications, Thousand Oaks2004
        • Patton M.Q.
        Qualitative research and evaluation methods: integrating theory and practice. 4th ed.
        Sage, Los Angeles2015
        • Harris J.L.
        • Booth A.
        • Cargo M.
        • Hannes K.
        • Harden A.
        • Flemming K.
        • et al.
        Cochrane Qualitative and Implementation Methods Group guidance series—paper 2: methods for question formulation, searching and protocol development for qualitative evidence synthesis.
        J Clin Epidemiol. 2018; 97: 39-48
        • Noyes J.
        • Booth A.
        • Flemming K.
        • Garside R.
        • Harden A.
        • Lewin S.
        • et al.
        Cochrane Qualitative and Implementation Methods Group Guidance series—paper 3: methods for assessing methodological limitations, data extraction and synthesis, and confidence in synthesized qualitative findings.
        J Clin Epidemiol. 2018; 97: 49-58
        • Harden A.
        • Thomas J.
        • Cargo M.
        • Harris J.
        • Pantoja T.
        • Flemming K.
        • Booth A.
        • et al.
        Cochrane Qualitative and Implementation Methods Group guidance series—paper 5: methods for integrating qualitative and implementation evidence within intervention effectiveness reviews.
        J Clin Epidemiol. 2018; 97: 70-78
        • Higgins J.P.
        • Altman D.G.
        • Gotzsche P.C.
        • Juni P.
        • Moher D.
        • Oxman A.D.
        • et al.
        The Cochrane Collaboration's tool for assessing risk of bias in randomised trials.
        BMJ. 2011; 343: d5928
        • Dane A.V.
        • Schneider B.H.
        Program integrity in primary and early secondary prevention: are implementation effects out of control?.
        Clin Psychol Rev. 1998; 18: 23-45
        • Weiss C.H.
        Evaluation research: methods of assessing program effectiveness.
        Prentice-Hall Inc, Englewood Cliffs, New Jersey1972
        • Moore G.
        • Audrey S.
        • Barker M.
        • Lyndal B.
        • Bonell C.
        • Hardeman W.
        • et al.
        Process evaluation of complex interventions.
        Medical Research Council (MRC) Guidance, UK2015
        • Milat A.
        • Newson R.
        • King L.
        • Rissel C.
        • Wolfenden L.
        • Bauman A.
        • et al.
        A guide to scaling up population health interventions.
        Public Health Res Pract. 2016; 26: e2611604
        • Wilson S.J.
        • Lipsey M.W.
        The effects of school-based social information processing interventions on aggressive behavior: part I: Universal programs.
        Campbell Systematic Reviews, 2006 (Available at https://www.campbellcollaboration.org/library/school-based-interventions-aggressive-behaviour-part1.html. Accessed June 2016)
        • Thorpe K.E.
        • Zwarenstein M.
        • Oxman A.D.
        • Treweek S.
        • Furberg C.D.
        • Altman D.G.
        • et al.
        A pragmatic-explanatory continuum indicator summary (PRECIS): a tool to help trial designers.
        J Clin Epidemiol. 2009; 62: 464-475
        • Schillinger D.
        An introduction to effectiveness, dissemination and implementation research. A resource manual for communtiy-engaged research.
        University of California San Francisco Clinical and Translational Science Institute (CTSI) Community Engagement Program, 2010 (Available at http://ctsi.ucsf.edu/files/CE/edi_introguide.pdf. Accessed February 2016)
        • Noyes J.
        • Hendry M.
        • Lewin S.
        • Glenton C.
        • Chandler J.
        • Rashidian A.
        Qualitative “trial-sibling” studies and “unrelated” qualitative studies contributed to complex intervention reviews.
        J Clin Epidemiol. 2016; 74: 133-143
        • Moore G.
        • Audrey S.
        • Barker M.
        • Bond L.
        • Bonell C.
        • Cooper C.
        • et al.
        Process evaluation in complex public health intervention studies: the need for guidance.
        J Epidemiol Community Health. 2013; 68: 101-102
        • Carroll C.
        • Patterson M.
        • Wood S.
        • Booth A.
        • Rick J.
        • Balain S.
        A conceptual framework for implementation fidelity.
        Implementation Sci. 2007; 2: 40
      1. Steckler A. Linnan L. Process evaluation for public health interventions and research. Jossey-Bass, San Francisco2002
        • Armstrong R.
        • Waters E.
        • Moore L.
        • Riggs E.
        • Cuervo L.G.
        • Lumbiganon P.
        • et al.
        Improving the reporting of public health intervention research: advancing TREND and CONSORT.
        J Public Health. 2008; 30: 103-109
        • Creswell J.W.
        • Klassen A.C.
        • Plano Clark V.L.
        • Smith K.C.
        Best practices for mixed methods research in the health sciences.
        National Institutes of Health, Office of Behavioral and Social Sciences Research, 2011 (Available at https://obssr.od.nih.gov/training/mixed-methods-research/. Accessed March 2016)
        • Chen H.-T.
        Practical program evaluation. Assessing and improving planning, implementation and effectiveness.
        Sage Publications, Thousand Oaks, CA2005
        • Farrington D.
        • Ttofi M.
        School-based programs to reduce bullying and victimization.
        Campbell Systematic Reviews, 2009 (Available at https://www.campbellcollaboration.org/library/school-based-programmes-to-reduce-bullying-victimisation.html. Accessed April 2016)
        • Grant M.J.
        • Booth A.
        A typology of reviews: an analysis of 14 review types and associated methodologies.
        Health Info Libr J. 2009; 26: 91-108
        • Funnell S.C.
        • Rogers P.J.
        Purposeful program theory: effective use of theories of change and logic models.
        San Francisco, CA, John Wiley/Jossey-Bass2011
        • Hoffmann T.
        • Glasziou P.
        • Boutron I.
        • Milne R.
        • Perera R.
        • Moher D.
        Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide.
        BMJ. 2014; 348: g1687
        • Anderson L.M.
        • Petticrew M.
        • Rehfuess E.
        • Armstrong R.
        • Ueffing E.
        • Baker P.
        • et al.
        Using logic models to capture complexity in systematic reviews.
        Res Synth Methods. 2011; 2: 33-42
        • Barnett-Page E.
        • Thomas J.
        Methods for the synthesis of qualitative research: a critical review.
        BMC Med Res Methodol. 2009; 9: 59
        • Walker L.O.
        • Avant K.C.
        Strategies for theory construction in nursing.
        Pearson Prentice Hall, Upper Saddle River, NJ2005
        • Carroll C.
        • Booth A.
        • Leaviss J.
        • Rick J.
        “Best fit” framework synthesis: refining the method.
        BMC Med Res Methodol. 2013; 13: 37
        • Booth A.
        • Noyes J.
        • Flemming K.
        • Gerhardus A.
        • Wahlster P.
        • Van der Wilt G.J.
        • et al.
        Guidance on choosing qualitative evidence synthesis methods for use in health technology assessments of complex interventions.
        ([Online])2016 (Available at) (Accessed October 2016)
        • Noyes J.
        • Pearson A.
        • Hannes K.
        Booth a on behalf of the Cochrane Qualitative Research Methods Group. Qualitative research and cochrane reviews.
        in: Higgins J. Green S. Cochrane handbook for the systematic review of interventions: cochrane book series. John Wiley and Sons Ltd, UK2008: 571-592
        • Oliver K.
        • Rees R.
        • Brady L.M.
        • Kavanagh J.
        • Oliver S.
        • Thomas J.
        Broadening public participation in systematic reviews: a case example involving young people in two configurative reviews.
        Res Synth Methods. 2015; 6: 206-217
        • Jamal F.
        • Langford R.
        • Daniels P.
        • Thomas J.
        • Harden A.
        • Bonell C.
        Consulting with young people to inform systematic reviews: an example from a review on the effects of schools on health.
        Health Expect. 2014; 18: 3225-3235
        • Lewin S.
        • Glenton C.
        • Oxman A.D.
        Use of qualitative methods alongside randomised controlled trials of complex healthcare interventions: methodological study.
        BMJ. 2009; 339: b3496
        • Adams R.J.
        • Huff A.S.
        Shades of grey: guidelines for working with the grey literature in systematic reviews for management and organizational studies.
        Int J Management Rev. 2017; 19: 432-454
        • Harden A.
        • Oakley A.
        • Weston R.
        A review of the effectiveness and appropriateness of peer-delivered health promotion for young people.
        London, UK, Institute of Education, University of London1999
        • Lefebvre C.M.E.
        • Glanville J.
        Chapter 6: searching for studies.
        (updated March 2011)in: Higgins J. Green S. Cochrane handbook for systematic reviews of interventions version 510. The Cochrane Collaboration, 2011 (Available at) (Accessed January 2016)
        • Cooper Robbins S.C.
        • Ward K.
        • Skinner S.R.
        School-based vaccination: a systematic review of process evaluations.
        Vaccine. 2011; 29: 9588-9599
        • Booth A.
        • Harris J.
        • Croot E.
        • Springett J.
        • Campbell F.
        • Wilkins E.
        Towards a methodology for cluster searching to provide conceptual and contextual “richness” for systematic reviews of complex interventions: case study (CLUSTER).
        BMC Med Res Methodol. 2013; 13: 118
        • Bonell C.
        • Farah J.
        • Harden A.
        • Wells H.
        • Parry W.
        • Fletcher A.
        • et al.
        Systematic review of the effects of schools and school environment interventions on health: evidence mapping and synthesis.
        Public Health Res. 2013; 1
        • Lewin S.
        • Hendry M.
        • Chandler J.
        • Oxman A.D.
        • Michie S.
        • Shepperd S.
        • et al.
        Assessing the complexity of interventions within systematic reviews: development, content and use of a new tool (iCAT_SR).
        BMC Med Res Methodol. 2017; 17: 76
        • Cargo M.
        • Stankov I.
        • Thomas J.
        • Saini M.
        • Rogers P.
        • Mayo-Wilson E.
        • et al.
        Development, inter-rater reliability and feasibility of a checklist to assess implementation (Ch-IMP) in systematic reviews: the case of provider-based prevention and treatment programs targeting children and youth.
        BMC Med Res Methodol. 2015; 15: 73
        • Salmon J.
        • Ball K.
        • Crawford D.
        • Booth M.
        • Telford A.
        • Hume C.
        • et al.
        Reducing sedentary behaviour and increasing physical activity among 10-year-old children: overview and process evaluation of the 'Switch-Play' intervention.
        Health Promotion Int. 2005; 20: 7-17
        • Sutcliffe K.
        • Thomas J.
        • Stokes G.
        • Hinds K.
        • Bangpan M.
        Intervention Component Analysis (ICA): a pragmatic approach for identifying the critical features of complex interventions.
        Syst Rev. 2015; 4: 1-13
        • Montgomery P.
        • Underhill K.
        • Gardner F.
        • Operario D.
        • Mayo-Wilson E.
        The Oxford Implementation Index: a new tool for incorporating implementation data into systematic reviews and meta-analyses.
        J Clin Epidemiol. 2013; 66: 874-882
        • Armstrong R.W.E.
        • Jackson N.
        • Oliver S.
        • Popay J.
        • Shepherd J.
        • Petticrew M.
        • et al.
        Guidelines for systematic reviews of health promotion and public health interventions. Version 2.
        Melbourne University, Australia2007
        • Cargo M.
        Tools for critically appraising process and implementation in systematic reviews.
        (Paper presented at The Joanna Briggs Institute Biennial International Convention Ripples to Revolution: From Bench to Bedside; Adelaide, Australia)2009
        • Rees R.
        • Oliver K.
        • Woodman J.
        • Thomas J.
        Children's views about obesity, body size, shape and weight. A systematic review: Social Science Research Unit.
        London, UK, Institute of Education, University of London2009
        • Shepherd J.
        • Kavanagh J.
        • Picot J.
        • Cooper K.
        • Harden A.
        • Barnett-Page E.
        • et al.
        The effectiveness and cost-effectiveness of behavioural interventions for the prevention of sexually transmitted infections in young people aged 13-19: a systematic review and economic evaluation.
        Health Technology Assess. 2010; 14 ([iii-iv]): 1-206
        • Health Care Practice Research and Development Unit
        Evaluation tool for qualitative research.
        2009 (Available at) (Accessed March 2016)
        • Hannes K.
        • Lockwood C.
        • Pearson A.
        A comparative analysis of three online appraisal instruments' ability to assess validity in qualitative research.
        Qual Health Res. 2010; 20: 1736-1743
        • Lewin S.
        • Glenton C.
        • Munthe-Kaas H.
        • Carlsen B.
        • Colvin C.J.
        • Gulmezoglu M.
        • et al.
        Using qualitative evidence in decision making for health and social interventions: an approach to assess confidence in findings from qualitative evidence syntheses (GRADE-CERQual).
        PLoS Med. 2015; 12: e1001895
        • Effective Public Health Practice Project Quality Assessment Tool
        Quality assessment tool for quantitative studies.
        2009 (Available at http://www.ephpp.ca/PDF/Quality%20Assessment%20Tool_2010_2.pdf. Accessed January 2016)
        • Petrosino A.
        • Turpin-Petrosino C.
        • Hollis-Peel M.E.
        • Lavenberg J.G.
        ‘Scared Straight’ and other juvenile awareness programs for preventing juvenile delinquency.
        Cochrane Database Syst Rev. 2013; (CD002796)