Advertisement

Crowdsourcing trainees in a living systematic review provided valuable experiential learning opportunities: a mixed-methods study

      Highlights

      • Trainees from a wide range of backgrounds reported very positive overall experiences from participating in a living systematic review (LSR).
      • LSRs may provide an ongoing opportunity for experiential learning in evidence-based medicine education.
      • Engaging trainees in LSRs may help ensure review sustainability, while providing valuable learning opportunities that can be tailored to trainees' skills and interests.

      Abstract

      Objectives

      To understand trainee experiences of participating in a living systematic review (LSR) for rheumatoid arthritis and the potential benefits in terms of experiential evidence-based medicine (EBM) education.

      Study Design and Setting

      We conducted a mixed-methods study with trainees who participated in the LSR and who were recruited broadly from training programs in two countries. Trainees received task-specific training and completed one or more tasks in the review: assessing article eligibility, data extraction, and quality assessment. Trainees completed a survey followed by a one-on-one interview. Data were triangulated to produce broad themes.

      Results

      Twenty one trainees, most of whom had a little prior experience with systematic reviews, reported a positive overall experience. Key benefits included learning opportunities, task segmentation (ability to focus on a single task, as opposed to an entire review), working in a supportive environment, international collaboration, and incentives such as authorship or acknowledgment. Trainees reported improvement in their competency as a Scholar, Collaborator, Leader, and Medical Expert. Challenges included communication and technical difficulties and appropriate matching of tasks to trainee skillsets.

      Conclusion

      Participating in an LSR provided benefits to a wide range of trainees and may provide an opportunity for experiential EBM training, while helping LSR sustainability.

      Keywords

      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'

      Subscribe:

      Subscribe to Journal of Clinical Epidemiology
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect

      References

        • Elliott J.H.
        • Synnot A.
        • Turner T.
        • Simmonds M.
        • Akl E.A.
        • McDonald S.
        • et al.
        Living systematic review: 1. Introduction-the why, what, when, and how.
        J Clin Epidemiol. 2017; 91: 23-30
        • Boutron I.
        • Chaimani A.
        • Meerpohl J.J.
        • Hrobjartsson A.
        • Devane D.
        • Rada G.
        • et al.
        The COVID-NMA project: building an evidence ecosystem for the COVID-19 pandemic.
        Ann Intern Med. 2020; 173: 1015-1017
        • Millard T.
        • Synnot A.
        • Elliott J.
        • Green S.
        • McDonald S.
        • Turner T.
        Feasibility and acceptability of living systematic reviews: results from a mixed-methods evaluation.
        Syst Rev. 2019; 8: 325
        • Marshall I.J.
        • Wallace B.C.
        Toward systematic review automation: a practical guide to using machine learning tools in research synthesis.
        Syst Rev. 2019; 8: 163
        • Thomas J.
        • Noel-Storr A.
        • Marshall I.
        • Wallace B.
        • McDonald S.
        • Mavergames C.
        • et al.
        Living systematic reviews: 2. Combining human and machine effort.
        J Clin Epidemiol. 2017; 91: 31-37
        • Wallace B.C.
        • Noel-Storr A.
        • Marshall I.J.
        • Cohen A.M.
        • Smalheiser N.R.
        • Thomas J.
        Identifying reports of randomized controlled trials (RCTs) via a hybrid machine learning and crowdsourcing approach.
        J Am Med Inform Assoc. 2017; 24: 1165-1168
        • Turner T.
        • Green S.
        • Tovey D.
        • McDonald S.
        • Soares-Weiser K.
        • Pestridge C.
        • et al.
        Producing cochrane systematic reviews-a qualitative study of current approaches and opportunities for innovation and improvement.
        Syst Rev. 2017; 6: 147
        • Lam W.W.
        • Fielding R.
        • Johnston J.M.
        • Tin K.Y.
        • Leung G.M.
        Identifying barriers to the adoption of evidence-based medicine practice in clinical clerks: a longitudinal focus group study.
        Med Educ. 2004; 38: 987-997
        • Meats E.
        • Heneghan C.
        • Crilly M.
        • Glasziou P.
        Evidence-based medicine teaching in UK medical schools.
        Med Teach. 2009; 31: 332-337
        • Maggio L.A.
        • ten Cate O.
        • Chen H.C.
        • Irby D.M.
        • O'Brien B.C.
        Challenges to learning evidence-based medicine and educational approaches to meet these challenges: a qualitative study of selected EBM curricula in U.S. and Canadian medical schools.
        Acad Med. 2016; 91: 101-106
        • Hadvani T.
        • Dutta A.
        • Choy E.
        • Kumar S.
        • Molleda C.
        • Parikh V.
        • et al.
        Effectiveness of modalities to teach evidence based medicine to pediatric clerkship students: a randomized controlled trial.
        Acad Pediatr. 2021; 21: 375-383
        • Ilic D.
        • Forbes K.
        Undergraduate medical student perceptions and use of Evidence Based Medicine: a qualitative study.
        BMC Med Educ. 2010; 10: 58
        • Del Mar C.
        • Glasziou P.
        • Mayer D.
        Teaching evidence based medicine.
        BMJ. 2004; 329: 989-990
        • Khan K.S.
        • Coomarasamy A.
        A hierarchy of effective teaching and learning to acquire competence in evidenced-based medicine.
        BMC Med Educ. 2006; 6: 59
        • Hazlewood G.
        • Whittle S.
        • Kamso M.
        • Akl E.
        • Wells G.
        • Tugwell P.
        • et al.
        Disease-modifying anti-rheumatic drugs for rheumatoid arthritis: a systematic review and network meta-analysis.
        Cochrane Database Syst Rev. 2020; 3: CD013562
        • Almutairi K.B.
        • Nossent J.C.
        • Preen D.B.
        • Keen H.I.
        • Inderjeeth C.A.
        The prevalence of rheumatoid arthritis: a systematic review of population-based studies.
        J Rheumatol. 2021; 48: 669-676
        • Safiri S.
        • Kolahi A.A.
        • Hoy D.
        • Smith E.
        • Bettampadi D.
        • Mansournia M.A.
        • et al.
        Global, regional and national burden of rheumatoid arthritis 1990-2017: a systematic analysis of the Global Burden of Disease study 2017.
        Ann Rheum Dis. 2019; 78: 1463-1471
        • Canadian Institute for Health Information (CIHI)
        Prescribed Drug Spending in Canada, 2016: A Focus on Public Drug Programs 2016.
        (Available at)
        • Noel-Storr A.
        • Dooley G.
        • Elliott J.
        • Steele E.
        • Shemilt I.
        • Mavergames C.
        • et al.
        An evaluation of Cochrane Crowd found that crowdsourcing produced accurate results in identifying randomized trials.
        J Clin Epidemiol. 2021; 133: 130-139
        • Thomas J.
        • McDonald S.
        • Noel-Storr A.
        • Shemilt I.
        • Elliott J.
        • Mavergames C.
        • et al.
        Machine learning reduced workload with minimal risk of missing studies: development and evaluation of a randomized controlled trial classifier for Cochrane Reviews.
        J Clin Epidemiol. 2021; 133: 140-151
        • Noel-Storr A.
        Working with a new kind of team: harnessing the wisdom of the crowd in trial identification.
        EFSA J. 2019; 17: e170715
        • Veritas Health Innovation
        Covidence 2021.
        (Available at)
        www.covidence.org
        Date accessed: March 1, 2022
        • Frank J.R.
        • Snell L.
        • Sherbino J.
        • Royal College of Physicians and Surgeons of Canada
        CanMEDS 2015: physician competency framework.
        Royal College of Physicians and Surgeons of Canada, Ottawa, Ontario, Canada2015
        • Braun V.
        • Clarke V.
        Using thematic analysis in psychology.
        Qual Res Psychol. 2006; 3: 77-101
        • Khalil H.
        • Ameen D.
        • Zarnegar A.
        Tools to support the automation of systematic reviews: a scoping review.
        J Clin Epidemiol. 2022; 144: 22-42
        • Jonnalagadda S.R.
        • Goyal P.
        • Huffman M.D.
        Automating data extraction in systematic reviews: a systematic review.
        Syst Rev. 2015; 4: 78