Crowdsourcing trainees in a living systematic review provided valuable experiential learning opportunities: a mixed-methods study


      • Trainees from a wide range of backgrounds reported very positive overall experiences from participating in a living systematic review (LSR).
      • LSRs may provide an ongoing opportunity for experiential learning in evidence-based medicine education.
      • Engaging trainees in LSRs may help ensure review sustainability, while providing valuable learning opportunities that can be tailored to trainees' skills and interests.



      To understand trainee experiences of participating in a living systematic review (LSR) for rheumatoid arthritis and the potential benefits in terms of experiential evidence-based medicine (EBM) education.

      Study Design and Setting

      We conducted a mixed-methods study with trainees who participated in the LSR and who were recruited broadly from training programs in two countries. Trainees received task-specific training and completed one or more tasks in the review: assessing article eligibility, data extraction, and quality assessment. Trainees completed a survey followed by a one-on-one interview. Data were triangulated to produce broad themes.


      Twenty one trainees, most of whom had a little prior experience with systematic reviews, reported a positive overall experience. Key benefits included learning opportunities, task segmentation (ability to focus on a single task, as opposed to an entire review), working in a supportive environment, international collaboration, and incentives such as authorship or acknowledgment. Trainees reported improvement in their competency as a Scholar, Collaborator, Leader, and Medical Expert. Challenges included communication and technical difficulties and appropriate matching of tasks to trainee skillsets.


      Participating in an LSR provided benefits to a wide range of trainees and may provide an opportunity for experiential EBM training, while helping LSR sustainability.


      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to Journal of Clinical Epidemiology
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Elliott J.H.
        • Synnot A.
        • Turner T.
        • Simmonds M.
        • Akl E.A.
        • McDonald S.
        • et al.
        Living systematic review: 1. Introduction-the why, what, when, and how.
        J Clin Epidemiol. 2017; 91: 23-30
        • Boutron I.
        • Chaimani A.
        • Meerpohl J.J.
        • Hrobjartsson A.
        • Devane D.
        • Rada G.
        • et al.
        The COVID-NMA project: building an evidence ecosystem for the COVID-19 pandemic.
        Ann Intern Med. 2020; 173: 1015-1017
        • Millard T.
        • Synnot A.
        • Elliott J.
        • Green S.
        • McDonald S.
        • Turner T.
        Feasibility and acceptability of living systematic reviews: results from a mixed-methods evaluation.
        Syst Rev. 2019; 8: 325
        • Marshall I.J.
        • Wallace B.C.
        Toward systematic review automation: a practical guide to using machine learning tools in research synthesis.
        Syst Rev. 2019; 8: 163
        • Thomas J.
        • Noel-Storr A.
        • Marshall I.
        • Wallace B.
        • McDonald S.
        • Mavergames C.
        • et al.
        Living systematic reviews: 2. Combining human and machine effort.
        J Clin Epidemiol. 2017; 91: 31-37
        • Wallace B.C.
        • Noel-Storr A.
        • Marshall I.J.
        • Cohen A.M.
        • Smalheiser N.R.
        • Thomas J.
        Identifying reports of randomized controlled trials (RCTs) via a hybrid machine learning and crowdsourcing approach.
        J Am Med Inform Assoc. 2017; 24: 1165-1168
        • Turner T.
        • Green S.
        • Tovey D.
        • McDonald S.
        • Soares-Weiser K.
        • Pestridge C.
        • et al.
        Producing cochrane systematic reviews-a qualitative study of current approaches and opportunities for innovation and improvement.
        Syst Rev. 2017; 6: 147
        • Lam W.W.
        • Fielding R.
        • Johnston J.M.
        • Tin K.Y.
        • Leung G.M.
        Identifying barriers to the adoption of evidence-based medicine practice in clinical clerks: a longitudinal focus group study.
        Med Educ. 2004; 38: 987-997
        • Meats E.
        • Heneghan C.
        • Crilly M.
        • Glasziou P.
        Evidence-based medicine teaching in UK medical schools.
        Med Teach. 2009; 31: 332-337
        • Maggio L.A.
        • ten Cate O.
        • Chen H.C.
        • Irby D.M.
        • O'Brien B.C.
        Challenges to learning evidence-based medicine and educational approaches to meet these challenges: a qualitative study of selected EBM curricula in U.S. and Canadian medical schools.
        Acad Med. 2016; 91: 101-106
        • Hadvani T.
        • Dutta A.
        • Choy E.
        • Kumar S.
        • Molleda C.
        • Parikh V.
        • et al.
        Effectiveness of modalities to teach evidence based medicine to pediatric clerkship students: a randomized controlled trial.
        Acad Pediatr. 2021; 21: 375-383
        • Ilic D.
        • Forbes K.
        Undergraduate medical student perceptions and use of Evidence Based Medicine: a qualitative study.
        BMC Med Educ. 2010; 10: 58
        • Del Mar C.
        • Glasziou P.
        • Mayer D.
        Teaching evidence based medicine.
        BMJ. 2004; 329: 989-990
        • Khan K.S.
        • Coomarasamy A.
        A hierarchy of effective teaching and learning to acquire competence in evidenced-based medicine.
        BMC Med Educ. 2006; 6: 59
        • Hazlewood G.
        • Whittle S.
        • Kamso M.
        • Akl E.
        • Wells G.
        • Tugwell P.
        • et al.
        Disease-modifying anti-rheumatic drugs for rheumatoid arthritis: a systematic review and network meta-analysis.
        Cochrane Database Syst Rev. 2020; 3: CD013562
        • Almutairi K.B.
        • Nossent J.C.
        • Preen D.B.
        • Keen H.I.
        • Inderjeeth C.A.
        The prevalence of rheumatoid arthritis: a systematic review of population-based studies.
        J Rheumatol. 2021; 48: 669-676
        • Safiri S.
        • Kolahi A.A.
        • Hoy D.
        • Smith E.
        • Bettampadi D.
        • Mansournia M.A.
        • et al.
        Global, regional and national burden of rheumatoid arthritis 1990-2017: a systematic analysis of the Global Burden of Disease study 2017.
        Ann Rheum Dis. 2019; 78: 1463-1471
        • Canadian Institute for Health Information (CIHI)
        Prescribed Drug Spending in Canada, 2016: A Focus on Public Drug Programs 2016.
        (Available at)
        • Noel-Storr A.
        • Dooley G.
        • Elliott J.
        • Steele E.
        • Shemilt I.
        • Mavergames C.
        • et al.
        An evaluation of Cochrane Crowd found that crowdsourcing produced accurate results in identifying randomized trials.
        J Clin Epidemiol. 2021; 133: 130-139
        • Thomas J.
        • McDonald S.
        • Noel-Storr A.
        • Shemilt I.
        • Elliott J.
        • Mavergames C.
        • et al.
        Machine learning reduced workload with minimal risk of missing studies: development and evaluation of a randomized controlled trial classifier for Cochrane Reviews.
        J Clin Epidemiol. 2021; 133: 140-151
        • Noel-Storr A.
        Working with a new kind of team: harnessing the wisdom of the crowd in trial identification.
        EFSA J. 2019; 17: e170715
        • Veritas Health Innovation
        Covidence 2021.
        (Available at)
        Date accessed: March 1, 2022
        • Frank J.R.
        • Snell L.
        • Sherbino J.
        • Royal College of Physicians and Surgeons of Canada
        CanMEDS 2015: physician competency framework.
        Royal College of Physicians and Surgeons of Canada, Ottawa, Ontario, Canada2015
        • Braun V.
        • Clarke V.
        Using thematic analysis in psychology.
        Qual Res Psychol. 2006; 3: 77-101
        • Khalil H.
        • Ameen D.
        • Zarnegar A.
        Tools to support the automation of systematic reviews: a scoping review.
        J Clin Epidemiol. 2022; 144: 22-42
        • Jonnalagadda S.R.
        • Goyal P.
        • Huffman M.D.
        Automating data extraction in systematic reviews: a systematic review.
        Syst Rev. 2015; 4: 78