Reviews| Volume 137, P45-57, September 2021

Download started.


A review of pragmatic trials found a high degree of diversity in design and scope, deficiencies in reporting and trial registry data, and poor indexing



      We established a large database of trials to serve as a resource for future methodological and ethical analyses. Here, we use meta-data to describe the broad landscape of pragmatic trials including research areas, identification as pragmatic, quality of trial registry data and enrolment.

      Study Design and Setting

      Trials were identified by a validated search filter and included if a primary report of a health-related randomized trial published January 2014-April 2019. Data were collated from MEDLINE, Web of Science,, and full text.


      4337 eligible trials were identified from 13,065 records, of which 1988 were registered in Research areas were diverse, with the most common being general and internal medicine; public, environmental and occupational health; and health care sciences and services. The term “pragmatic” was seldom used in titles or abstracts. Several domains in had questionable data quality. We estimated that one-fifth of trials under-accrued by at least 15%.


      There is a need to improve reporting of pragmatic trials and quality of trial registry data. Under accrual remains a challenge in pragmatic RCTs despite calls for more streamlined recruitment approaches. The diversity of pragmatic trials should be reflected in future ethical analyses.


      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to Journal of Clinical Epidemiology
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Schwartz D
        • Lellouch J.
        Explanatory and pragmatic attitudes in therapeutical trials.
        J Chronic Dis. 1967; 20: 637-648
        • Karanicolas PJ
        • Montori VM
        • Devereaux PJ
        • Schünemann H
        • Guyatt GH.
        A new mechanistic-practical framework for designing and interpreting randomized trials.
        J Clin Epidemiol. 2009; 62: 479-484
        • Tunis S
        • Stryer DB
        • Clancy CM.
        Practical clinical trials: increasing the value of clinical research for decision making in clinical and health policy.
        JAMA. 2003; 290: 1624-1632
        • Dal-Ré R
        • Janiaud P
        • Ioannidis JPA.
        Real-world evidence: How pragmatic are randomized controlled trials labeled as pragmatic?.
        BMC medicine. 2018; 16
        • Treweek S
        • Zwarenstein M.
        Making trials matter: pragmatic and explanatory trials and the problem of applicability.
        Trials. 2009; 10: 37
        • Zwarenstein M
        • Treweek S.
        What kind of randomized trials do we need?.
        CMAJ. 2009; 180: 998-1000
      1. Patient-Centred Outcomes Research Institute (PCORI): PCORI Funding Announcement: Pragmatic Clinical Studies To Evaluate Patient-Centered Outcomes. Available at: Accessed 23 May 2018.

        • Patsopoulos NA.
        A pragmatic view on pragmatic trials.
        Dialogues Clin Neurosci. 2011; 13: 217-224
        • Loudon K
        • Treweek S
        • Sullivan F
        • Donnan P
        • Thorpe KE
        • Zwarenstein M.
        The PRECIS-2 tool: designing trials that are fit for purpose.
        BMJ. 2015; 350: h2147
        • Zwarenstein M
        • Treweek S
        • Gagnier JJ
        • Altman DG
        • Tunis S
        • Haynes B
        • et al.
        Improving the reporting of pragmatic trials: an extension of the CONSORT statement.
        BMJ. 2008; 337: a2390
        • Goldstein CE
        • Weijer C
        • Brehaut JC
        • Fergusson DA
        • Grimshaw JM
        • Horn AR
        • et al.
        Ethics issues in pragmatic randomized controlled trials: a review of the recent literature identifies gaps in argumentation.
        BMC Med Ethics. 2018; : 14
        • Nicholls SG
        • Carroll K
        • Zwarenstein M
        • Brehaut JC
        • Weijer C
        • Hey SP
        • et al.
        The ethical challenges raised in the design and conduct of pragmatic trials: an interview study with key stakeholders.
        Trials. 2019; 20: 765
        • Zuidgeest MGP
        • Goetz I
        • Groenwold RHH
        • Irving E
        • van Thiel G
        • Grobbee DE
        • et al.
        Series: Pragmatic trials and real world evidence: paper 1. introduction.
        J Clin Epidemiol. 2017; 88: 7-13
        • Janiaud P
        • Dal-Ré R
        • Ioannidis JPA.
        Assessment of pragmatism in recently published randomized clinical trials.
        JAMA Intern Med. 2018; 178: 1278-1280
        • Ramsberg J
        • Platt R.
        Opportunities and barriers for pragmatic embedded trials: triumphs and tribulations.
        Learn Health Syst. 2017;
        • Murray EJ
        • Caniglia EC
        • Swanson SA
        • Hernández-Díaz S
        • Hernán MA.
        Patients and investigators prefer measures of absolute risk in subgroups for pragmatic randomized trials.
        J Clin Epidemiol. 2018; 103: 10-21
        • Varas-Doval R
        • Saez-Benito L
        • Gastelurrutia MA
        • Benrimoj SI
        • Garcia-Cardenas V
        • Martinez-Martinez F.
        Systematic review of pragmatic randomised control trials assessing the effectiveness of professional pharmacy services in community pharmacies.
        BMC Health Serv Res. 2021; 21: 156
        • Taljaard M
        • Weijer C
        • Grimshaw JM
        • Ali A
        • Brehaut JC
        • Campbell MK
        • et al.
        Developing a framework for the ethical design and conduct of pragmatic trials in healthcare: a mixed methods research protocol.
        Trials. 2018; 19: 525
        • Taljaard M
        • Goldstein CE
        • Giraudeau B
        • Nicholls SG
        • Carroll K
        • Hey SP
        • et al.
        Cluster over individual randomization: are study design choices appropriately justified? Review of a random sample of trials.
        Clin Trials. 2020; 17: 253-263
        • Taljaard M
        • Hemming K
        • Shah L
        • Giraudeau B
        • Grimshaw JM
        • Weijer C.
        Inadequacy of ethical conduct and reporting of stepped wedge cluster randomized trials: Results from a systematic review.
        Clin Trials. 2017; 14: 333-341
        • Weijer C
        • Goldstein CE
        • Taljaard M.
        TwiC or treat? Are trials within cohorts ethically defensible?.
        Clinical Trials. 2018; 15: 21-24
        • Kim SY
        • Flory J
        • Relton C.
        Ethics and practice of Trials within Cohorts: an emerging pragmatic trial design.
        Clin Trials. 2018; 15: 9-13
        • Taljaard M
        • McDonald S
        • Nicholls SG
        • Carroll K
        • Hey SP
        • Grimshaw JM
        • et al.
        A search filter to identify pragmatic trials in MEDLINE was highly specific but lacked sensitivity.
        J Clin Epidemiol. 2020; 124: 75-84
        • Oxman AD
        • Lombard C
        • Treweek S
        • Gagnier JJ
        • Maclure M
        • Zwarenstein M.
        A pragmatic resolution.
        J Clin Epidemiol. 2009; 62: 495-498
        • Dal-Ré R.
        The PRECIS-2 tool seems not to be useful to discriminate the degree of pragmatism of medicine masked trials from that of open-label trials.
        Eur J Clin Pharmacol. 2020;
        • Dal-Ré R
        • de Boer A
        • James SK.
        The design can limit PRECIS–2 retrospective assessment of the clinical trial explanatory/pragmatic features.
        J Clin Epidemiol. 2020;
        • Zwarenstein M
        • Thorpe K
        • Treweek S
        • Loudon K.
        PRECIS-2 for retrospective assessment of RCTs in systematic reviews: some thoughts on intention, dichotomization and applicability of RCTs.
        J Clin Epidemiol. 2020;
        • Devos F
        • Foissac F
        • Bouazza N
        • Ancel PY
        • Treluyer JM
        • Chappuy H.
        Study characteristics impacted the pragmatism of randomized controlled trial published in nursing: a meta-epidemiological study.
        J Clin Epidemiol. 2019; 116: 18-25
        • Aves T
        • Allan KS
        • Lawson D
        • Nieuwlaat R
        • Beyene J
        • Mbuagbaw L.
        The role of pragmatism in explaining heterogeneity in meta-analyses of randomised trials: a protocol for a cross-sectional methodological review.
        BMJ Open. 2017; 7e017887
        • Yoong SL
        • Wolfenden L
        • Clinton-McHarg T
        • Waters E
        • Pettman TL
        • Steele E
        • et al.
        Exploring the pragmatic and explanatory study design on outcomes of systematic reviews of public health interventions: a case study on obesity prevention trials.
        J Public Health (Oxf). 2014; 36: 170-176
        • Koppenaal T
        • Linmans J
        • Knottnerus JA
        • Spigt M.
        Pragmatic vs. explanatory: an adaptation of the PRECIS tool helps to judge the applicability of systematic reviews for daily practice.
        J Clin Epidemiol. 2011; 64: 1095-1101
        • Nicholls SG
        • Zwarenstein M
        • Hey SP
        • Giraudeau B
        • Campbell MK
        • Taljaard M.
        The importance of decision intent within descriptions of pragmatic trials.
        J Clin Epidemiol. 2020; 125: 30-37
        • The Pawson R.
        "pragmatic trial": An essentially contested concept?.
        J Eval Clin Pract. 2019;
        • Veritas Health Innovation
        Covidence systematic review software.
        Melbourne, Australia2019
      2. Clarivate Analytics: Research Areas (Categories /Classification) Available at: Accessed 24 March 2020.

        • Gopal AD
        • Wallach JD
        • Aminawung JA
        • Gonsalves G
        • Dal-Re R
        • Miller JE
        • et al.
        Adherence to the international committee of medical journal editors' (ICMJE) prospective registration policy and implications for outcome integrity: a cross-sectional analysis of trials published in high-impact specialty society journals.
        Trials. 2018; 19: 448
        • Viergever RF
        • Li K.
        Trends in global clinical trial registration: an analysis of numbers of registered clinical trials in different parts of the world from 2004 to 2013.
        BMJ Open. 2015; 5e008932
        • Inrig JK
        • Califf RM
        • Tasneem A
        • Vegunta RK
        • Molina C
        • Stanifer JW
        • et al.
        The landscape of clinical trials in nephrology: a systematic review of
        Am J Kidney Dis. 2014; 63: 771-780
        • Jones CW
        • Handler L
        • Crowell KE
        • Keil LG
        • Weaver MA
        • Platts-Mills TF.
        Non-publication of large randomized clinical trials: cross sectional analysis.
        BMJ. 2013; 347: f6104
        • Califf RM
        • Zarin DA
        • Kramer JM
        • Sherman RE
        • Aberle LH
        • Tasneem A.
        Characteristics of clinical trials registered in, 2007-2010.
        JAMA. 2012; 307: 1838-1847
        • Zarin DA
        • Tse T
        • Williams RJ
        • Rajakannan T.
        Update on Trial Registration 11 Years after the ICMJE Policy Was Established.
        N Engl J Med. 2017; 376: 383-391
      3. Richardson L. beautifulsoup4 4.8.2. 4.8.2 edn; 2019.

      4. Airtable. Available at: [] Accessed 19 April 2021.

        • Carlisle B
        • Kimmelman J
        • Ramsay T
        • MacKinnon N.
        Unsuccessful trial accrual and human subjects protections: an empirical analysis of recently closed trials.
        Clin Trials. 2015; 12: 77-83
        • van Lent M
        • IntHout J
        • Out HJ.
        Differences between information in registries and articles did not influence publication acceptance.
        J Clin Epidemiol. 2015; 68: 1059-1067.
        • Catalá-López F
        • Aleixandre-Benavent R
        • Caulley L
        • Hutton B
        • Tabarés-Seisdedos R
        • Moher D
        • et al.
        Global mapping of randomised trials related articles published in high-impact-factor medical journals: a cross-sectional analysis.
        Trials. 2020; 21: 34
        • Clifford TJ
        • Barrowman NJ
        • Moher D.
        Funding source, trial outcome and reporting quality: are they related? Results of a pilot study.
        BMC Health Serv Res. 2002; 2: 18
        • Hakoum MB
        • Jouni N
        • Abou-Jaoude EA
        • Hasbani DJ
        • Abou-Jaoude EA
        • Lopes LC
        • et al.
        Characteristics of funding of clinical trials: cross-sectional survey and proposed guidance.
        BMJ Open. 2017; 7e015997
        • Falk Delgado A
        • Falk Delgado A
        The association of funding source on effect size in randomized controlled trials: 2013-2015 - a cross-sectional survey and meta-analysis.
        Trials. 2017; 18: 125
        • Tang E
        • Ravaud P
        • Riveros C
        • Perrodeau E
        • Dechartres A.
        Comparison of serious adverse events posted at and published in corresponding journal articles.
        BMC Med. 2015; 13: 189
        • Khamis AM
        • Bou-Karroum L
        • Hakoum MB
        • Al-Gibbawi M
        • Habib JR
        • El-Jardali F
        • et al.
        The reporting of funding in health policy and systems research: a cross-sectional study.
        Health Res Policy Syst. 2018; 16: 83
        • Stensland K
        • Kaffenberger S
        • Canes D
        • Galsky M
        • Skolarus T
        • Moinzadeh A.
        Assessing genitourinary cancer clinical trial accrual sufficiency using archived trial data.
        JCO Clinical Cancer Informatics. 2020; 4: 614-622
        • Gresham G
        • Meinert JL
        • Gresham AG
        • Meinert CL.
        Assessment of trends in the design, accrual, and completion of trials registered in by sponsor type, 2000-2019.
        JAMA Netw Open. 2020; 3e2014682
        • Dal-Ré R.
        Articles provided insufficient information to conduct an appropriate retrospective assessment of the pragmatic/explanatory features of medicine trials with the PRECIS-2 tool.
        Eur J Clin Pharmacol. 2020;
        • Zarin DA
        • Tse T
        • Williams RJ
        • Califf RM
        • Ide NC.
        The results database — update and key issues.
        N Engl J Med. 2011; 364: 852-860
        • Pranic S
        • Marusic A.
        Changes to registration elements and results in a cohort of trials were not reflected in published articles.
        J Clin Epidemiol. 2016; 70: 26-37
        • Jones CW
        • Keil LG
        • Holland WC
        • Caughey MC
        • Platts-Mills TF.
        Comparison of registered and published outcomes in randomized controlled trials: a systematic review.
        BMC Medicine. 2016;
        • Talebi R
        • Redberg RF
        • Ross JS.
        Consistency of trial reporting between and corresponding publications: one decade after FDAAA.
        Trials. 2020; 21
        • Vallvé C.
        Revisión crítica del ensayo clínico pragmático.
        Med Clin (Barc). 2003; 121: 384-388
        • Kim SY
        • Miller FG.
        Ethical complexities in standard of care randomized trials: a case study of morning versus nighttime dosing of blood pressure drugs.
        Clin Trials. 2015; 12: 557-563
        • Kim SY
        • Miller FG.
        Varieties of standard-of-care treatment randomized trials: ethical implications.
        JAMA. 2015; 313: 895-896
        • Kalkman S
        • Kim SYH
        • van Thiel G
        • Grobbee DE
        • van Delden JJM.
        Ethics of Informed Consent for Pragmatic Trials with New Interventions.
        Value Health. 2017; 20: 902-908
        • Anderson ML
        • Califf RM
        • Sugarman J
        Ethical and regulatory issues of pragmatic cluster randomized trials in contemporary health systems.
        Clin Trials. 2015; 12: 276-286
        • Califf RM
        • Sugarman J.
        Exploring the ethical and regulatory issues in pragmatic clinical trials.
        Clin Trials. 2015; 12: 436-441
        • Sugarman J
        • Califf RM.
        Ethics and regulatory complexities for pragmatic clinical trials.
        JAMA. 2014; 311: 2381-2382
        • Luyckx VA
        • Biller-Andorno N
        • Saxena A
        • Tran NT.
        Health policy and systems research: towards a better understanding and review of ethical issues.
        BMJ Global Health. 2017; 2e000314
        • Dal-Ré R
        • Avendaño-Solà C
        • Bloechl-Daum B
        • de Boer A
        • Eriksson S
        • Fuhr U
        • et al.
        Low risk pragmatic trials do not always require participants' informed consent.
        BMJ. 2019; 364: l1092
        • Kim SYH
        • Miller FG.
        Informed Consent for Pragmatic Trials — The Integrated Consent Model.
        N Engl J Med. 2014; 370: 769-772
        • Oude Rengerink K
        • Kalkman S
        • Collier S
        • Ciaglia A
        • Worsley SD
        • Lightbourne A
        • et al.
        Participant eligibility, recruitment, and retention in pragmatic trials.
        J Clin Epidemiol. 2017; 89: 173-180