Advertisement

Cochrane Qualitative and Implementation Methods Group guidance series—paper 5: methods for integrating qualitative and implementation evidence within intervention effectiveness reviews

Published:December 11, 2017DOI:https://doi.org/10.1016/j.jclinepi.2017.11.029

      Abstract

      The Cochrane Qualitative and Implementation Methods Group develops and publishes guidance on the synthesis of qualitative and mixed-method evidence from process evaluations. Despite a proliferation of methods for the synthesis of qualitative research, less attention has focused on how to integrate these syntheses within intervention effectiveness reviews. In this article, we report updated guidance from the group on approaches, methods, and tools, which can be used to integrate the findings from quantitative studies evaluating intervention effectiveness with those from qualitative studies and process evaluations. We draw on conceptual analyses of mixed methods systematic review designs and the range of methods and tools that have been used in published reviews that have successfully integrated different types of evidence. We outline five key methods and tools as devices for integration which vary in terms of the levels at which integration takes place; the specialist skills and expertise required within the review team; and their appropriateness in the context of limited evidence. In situations where the requirement is the integration of qualitative and process evidence within intervention effectiveness reviews, we recommend the use of a sequential approach. Here, evidence from each tradition is synthesized separately using methods consistent with each tradition before integration takes place using a common framework. Reviews which integrate qualitative and process evaluation evidence alongside quantitative evidence on intervention effectiveness in a systematic way are rare. This guidance aims to support review teams to achieve integration and we encourage further development through reflection and formal testing.

      Keywords

      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'

      Subscribe:

      Subscribe to Journal of Clinical Epidemiology
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect

      References

        • Barnett-Page E.
        • Thomas J.
        Methods for the synthesis of qualitative research: a critical review.
        BMC Med Res Methodol. 2009; 9: 59
        • Hannes K.
        • Lockwood C.
        Synthesizing qualitative research: choosing the right approach.
        John Wiley & Sons, Ltd, Chichester, UK2011
        • Sandelowski M.
        • Barroso J.
        Handbook for synthesizing qualitative research.
        Springer, New York2007
        • Glenton C.
        • Colvin C.J.
        • Carlsen B.
        • Swartz A.
        • Lewin S.
        • Noyes J.
        • et al.
        Barriers and facilitators to the implementation of lay health worker programmes to improve access to maternal and child health: qualitative evidence synthesis.
        Cochrane Database Syst Rev. 2013; : CD010414
        • Hannes K.
        • Booth A.
        • Harris J.
        • Noyes J.
        Celebrating methodological challenges and changes: reflecting on the emergence and importance of the role of qualitative evidence in Cochrane reviews.
        Systematic Reviews. 2013; 2: 84
        • Petticrew M.
        • Rehfuess E.
        • Noyes J.
        • Higgins J.
        • Mayhew A.
        • Panjota T.
        • et al.
        Synthesizing evidence on complex interventions: how meta-analytical, qualitative, and mixed-method approaches can contribute.
        J Clin Epidemiol. 2013; 66: 1230-1243
        • Sutcliffe K.
        • Thomas J.
        • Stokes G.
        • Hinds K.
        • Bangpan M.
        Intervention Component Analysis (ICA): a pragmatic approach for identifying the critical features of complex interventions.
        Syst Rev. 2015; 4: 140
        • Saini M.
        • Shlonsky A.
        Systematic synthesis of qualitative research: a pocket guide for social work research methods.
        Oxford University Press, Oxford2012
        • Yaffe J.
        • Montgomery P.
        • Hopewell S.
        • Shepard L.D.
        Empty reviews: a description and consideration of Cochrane systematic reviews with no included studies.
        PLoS One. 2012; 7: e36626
        • Johnson R.
        • Onwuegbuzie A.
        Mixed methods research: a research paradigm whose time has come?.
        Educ Res. 2004; 33: 14-26
        • Moore G.
        • Audrey S.
        • Barker M.
        • Bond L.
        • Bonell C.
        • Hardeman W.
        • et al.
        Process evaluation of complex interventions: Medical Research Council guidance.
        BMJ. 2015; 350: h1258
        • Kneale D.
        • Thomas J.
        • Harris K.
        Developing and optimising the use of logic models in systematic reviews: exploring practice and good practice in the use of programme theory in reviews.
        PLoS One. 2015; 10: 11
        • Funnell S.C.
        • Rogers P.J.
        Purposeful program theory: effective use of theories of change and logic models.
        John Wiley/Jossey-Bass, New Jersey2011
        • Rossi P.H.
        • Lipsey M.W.
        • Freeman H.E.
        Evaluation. A systematic approach.
        7th ed. Sage Publications, Thousand Oaks2004
        • Suchman E.
        Evaluative research.
        Russell Sage Foundation, New York1967
        • Weiss C.H.
        Evaluation research. Methods of assessing program effectiveness.
        Prentice-Hall, Englewood Cliffs, NJ1972
        • Chen H.-T.
        Practical program evaluation. Assessing and improving planning, implementation and effectiveness.
        Sage Publications, Thousand Oaks, CA2005 (2005)
        • Booth A.
        • Noyes J.
        • Flemming K.
        • Gerhardus A.
        • Wahlster P.
        • Van Der Wilt G.J.
        • et al.
        Guidance on choosing qualitative evidence synthesis methods for use in health technology assessments of complex interventions.
        2016 ([Online]. Available at) (Accessed January 28, 2018)
        • Dixon-Woods M.
        • Agarwal S.
        • Jones D.
        • Young B.
        • Sutton A.
        Synthesising qualitative and quantitative evidence: a review of possible methods.
        J Health Serv Res Policy. 2005; 10: 45-53
        • Harden A.
        • Thomas J.
        Mixed methods and systematic reviews: examples and emerging issues.
        in: Tashakkori A. Teddlie C. Mixed Methods Handbook. 2nd ed. Sage Publications, New York2010
        • Heyvaert M.
        • Hannes K.
        • Onghena P.
        Using mixed methods research synthesis for literature reviews.
        Sage Publications, New York2016
        • Pluye P.
        • Hong Q.N.
        Combining the power of stories and numbers: mixed methods research and mixed methods reviews.
        Annu Rev Public Health. 2014; 35: 29-45
        • Sandelowski M.
        • Voils C.
        • Barroso J.
        Defining and designing mixed research synthesis studies.
        Res Schools. 2006; 13: 29-40
        • Thomas J.
        • Harden A.
        • Oakley A.
        • Oliver S.
        • Sutcliffe K.
        • Rees R.
        • et al.
        Integrating qualitative research with trials in systematic reviews: an example review from public health shows how integration is possible and some potential benefits.
        BMJ. 2004; 328: 1010-1012
        • Pawson R.
        Evidence-based policy: a realist perspective.
        Sage, London2006
        • Wong G.
        • Greenhalgh T.
        • Westhorp G.
        • Buckingham J.
        • Pawson R.
        RAMESES publication standards: realist syntheses.
        J Adv Nurs. 2013; 69: 1005-1022
        • Ragin C.
        Redesigning social inquiry: fuzzy sets and beyond.
        University of Chicago Press, London2008
        • Thomas J.
        • O'Mara-Eves A.
        • Brunton G.
        Using Qualitative Comparative Analysis (QCA) in systematic reviews of complex interventions: a worked example.
        Syst Rev. 2014; 3: 67
        • Tricco A.C.
        • Antony J.
        • Soobiah C.
        • Kastner M.
        • MacDonald H.
        • Cogo E.
        • et al.
        Knowledge synthesis methods for integrating qualitative and quantitative data: a scoping review reveals poor operationalisation of the methodological steps,.
        J ClinicalEpidemiology. 2016; 73: 29-35
        • Noyes J.
        • Lewin S.
        Chapter 6: Supplemental guidance on selecting a method of qualitative evidence synthesis, and integrating qualitative evidence with Cochrane intervention reviews.
        (updated August 2011)in: Noyes J. Booth A. Hannes K. Harden A. Harris J. Lewin S. Supplementary Guidance for Inclusion of Qualitative Research in Cochrane Systematic Reviews of Interventions. Cochrane Collaboration Qualitative Methods Group, 2011 (Available at) (Accessed January 28, 2018)
        • Heyvaert M.
        • Maes B.
        • Onghena P.
        Mixed methods research synthesis: definition, framework, and potential.
        Qual Quant. 2013; 47: 659-676
        • Candy B.
        • King M.
        • Jones L.
        • Oliver S.
        Using qualitative synthesis to explore heterogeneity of complex interventions.
        BMC Med Res Methodol. 2011; 11: 124
        • Harden A.
        • Brunton G.
        • Fletcher A.
        • Oakley A.
        Teenage pregnancy and social disadvantage: a systematic review integrating trials and qualitative studies.
        BMJ. 2009; 339: b4254v
        • Shepherd J.
        • Harden A.
        • Barnett-Page E.
        • Kavanagh J.
        • Picot J.
        • Frampton G.
        • et al.
        Using process data to understand outcomes in sexual health promotion: an example from a review of school-based programmes to prevent sexually transmitted infections.
        Health Educ Res. 2014; 29: 566-582
        • Sutcliffe K.
        • Richardson M.
        • Rees R.
        • Melendez-Torres G.J.
        • Stansfield C.
        • Thomas J.
        What are the critical features of successful Tier 2 weight management programmes?: A systematic review to identify the programme characteristics, and combinations of characteristics, that are associated with successful weight loss.
        EPPI-Centre, Social Science Research Unit, UCL Institute of Education, University College London, London2016
        • Candy B.
        • King M.
        • Jones L.
        • Oliver S.
        Using qualitative evidence on patients' views to help understand variation in effectiveness of complex interventions: a qualitative comparative analysis.
        Trials. 2013; 14: 179
        • Flemming K.
        Synthesis of quantitative and qualitative research: an example using Critical Interpretive Synthesis.
        J Adv Nurs. 2010; 66: 201-217
        • Greenhalgh T.
        • Kristjannson E.
        • Robinson V.
        Realist review to understand the efficacy of school feeding programmes.
        BMJ. 2007; 335: 858-861
        • Roberts K.
        • Dixon-Woods M.
        • Fitzpatrick R.
        • Abrams K.
        • Jones D.
        Factors affecting uptake of childhood immunisation: a Bayesian synthesis of qualitative and quantitative evidence.
        Lancet. 2002; 360: 1596-1599
        • Voils C.
        • Hassselblad V.
        • Crandell J.
        • Chang Y.
        • Lee E.
        • Sandelowski M.
        A Bayesian method for the synthesis of evidence from qualitative and quantitative reports: the example of antiretroviral medication adherence.
        J Health Serv Res Policy. 2009; 14: 226-233
        • Wong G.
        • Greenhalgh T.
        • Westhorp G.
        • Buckingham J.
        • Pawson R.
        RAMESES publication standards: meta-narrative reviews.
        J Adv Nurs. 2013; 69: 987-1004
        • Aslam R.W.
        • Hendry M.
        • Carter B.
        • Noyes J.
        • Rycroft Malone J.
        • Booth A.
        • et al.
        Interventions for preventing unintended repeat pregnancies among adolescents (Protocol).
        Cochrane Database Syst Rev. 2015; 1: CD011477
        • Harris K.M.
        • Kneale D.
        • Lasserson T.J.
        • McDonald V.M.
        • Grigg J.
        • Thomas J.
        School-based self management interventions for asthma in children and adolescents: a mixed methods systematic review (Protocol).
        Cochrane Database Syst Rev. 2015; : CD011651
        • Anderson L.M.
        • Petticrew M.
        • Rehfuess E.
        • Armstrong R.
        • Ueffing E.
        • Baker P.
        • et al.
        Using logic models to capture complexity in systematic reviews.
        Res Synth Methods. 2011; 2: 33-42
        • Rohwer A.
        • Booth A.
        • Pfadenhauer L.
        • Brereton L.
        • Gerhardus A.
        • Mozygemba K.
        • et al.
        Guidance on the use of logic models in health technology assessments of complex interventions.
        2016 (Available at) (Accessed January 28, 2018)
        • Allmark P.
        • Baxter S.
        • Goyder E.
        • Guillaume L.
        • Crofton-Martin G.
        Assessing the health benefits of advice services: using research evidence and logic model methods to explore complex pathways.
        Health Soc Care Community. 2013; 21: 59-68
        • Baxter S.K.
        • Blank L.
        • Woods H.B.
        • Payne N.
        • Rimmer M.
        • Goyder E.
        Using logic model methods in systematic review synthesis: describing complex pathways in referral management interventions.
        BMC Med Res Methodol. 2014; 14: 1
        • Leeuw F.L.
        Reconstructing programme theories: methods available and problems to be solved.
        Am J Eval. 2003; 24: 5-20
        • Pearson M.
        • Hunt H.
        • Cooper C.
        • Shepperd S.
        • Pawson R.
        • Anderson R.
        Providing effective and preferred care closer to home: a realist review of intermediate care.
        Health Soc Care Community. 2015; 23: 577-593
        • Weiss C.
        Evaluation: Methods for studying programs and policies.
        Prentice Hall, Englewood Cliffs, NJ1998
        • Anderson L.M.
        • Oliver S.R.
        • Michie S.
        • Rehfuess E.
        • Noyes J.
        • Shemilt I.
        Investigating complexity in systematic reviews of interventions by using a spectrum of methods.
        J Clin Epidemiol. 2013; 66: 1223-1229
        • Oliver K.
        • Rees R.
        • Brady L.M.
        • Kavanagh J.
        • Oliver S.
        • Thomas J.
        Broadening public participation in systematic reviews: a case example involving young people in two configurative reviews.
        Res Synth Methods. 2015; 6: 206-217
        • Harris J.
        • Springett J.
        • Croot L.
        • Booth A.
        • Campbell F.
        • Thompson J.
        • et al.
        Can community-based peer support promote health literacy and reduce inequalities? A realist review.
        Public Health Research. 2015; 3 (Accessed January 28, 2018)
        • Booth A.
        • Carroll C.
        How to build up the actionable knowledge base: the role of ‘best fit’ framework synthesis for studies of improvement in healthcare.
        BMJ Qual Saf. 2015; 24: 700-708
      1. Hartmann-Boyce J, Johns D, Aveyard P, Onakpoya I, Jebb S, Phillips D, et al. The clinical effectiveness of long-term weight management schemes for adults (Review 1a), Update review for National Institute for Health & Care Excellence guidance on the managing overweight and obese adults. Oxford: University of Oxford; 2013.

        • van Grootel L.
        • van Wesel F.
        • O'Mara-Eves A.
        • Thomas J.
        • Hox J.
        • Boeije H.
        Using the realist perspective to link theory from qualitative evidence synthesis to quantitative studies: Broadening the matrix approach.
        Res Synth Methods. 2017; 8: 303-311