Advertisement

The GetReal Trial Tool: design, assess and discuss clinical drug trials in light of Real World Evidence generation

Open AccessPublished:December 17, 2021DOI:https://doi.org/10.1016/j.jclinepi.2021.12.019

      Abstract

      Methodologies incorporating Real World Elements into clinical trial design (also called pragmatic trials) offer an attractive opportunity to assess the effect of a treatment strategy in routine care and as such guide decision making in practice. Uptake of these methods is slow for several reasons, including uncertainty about acceptability of trial results, lack of experience with the methodology and operational challenges. We developed the “GetReal Trial Tool,” an easy-to-use online interface, which allows users to assess the impact of design choices on generalizability to routine clinical practice, while taking into account risk of bias, precision, acceptability and operational feasibility. The tool is grounded in the scientific literature combined with knowledge of experts from academia, pharmaceutical companies, HTA bodies, patient organizations, and regulators. The aim is to help researchers optimize trial design and facilitate translation of evidence from pragmatic trials to clinical practice. In this paper we describe the development, structure and application of the GetReal Trial Tool.

      Keywords

      What is new?
      • Progress towards the implementation of Real World Elements into trials is slow, for several reasons including uncertainty about how the data will be used in healthcare decision making, lack of experience with the methodology and operational challenges associated with such trials.
      • The GetReal Trial Tool allows users to assess the impact of design choices on generalisability of a trial to routine clinical practice, while taking into account other aspects of the trial, such as risk of bias, precision, acceptability and operational feasibility.
      • The GetReal Trial Tool is now available on www.getrealtrialtool.eu to help design and assess trials and facilitate translation to practice, to help overcome barriers to implementation.

      1. Introduction

      Real World Evidence (RWE), the evidence derived from the analysis and/or synthesis of real world data [
      • Zuidgeest MGP
      • Goetz I
      • Groenwold RHH
      • Irving E
      • van Thiel GJMW
      • Grobbee DE.
      Series: pragmatic trials and real world evidence: paper 1. Introduction.
      ] is increasingly important in health care decision-making. There is wide acknowledgement that evidence on the efficacy of treatments derived from strictly controlled conventional clinical trials, while the gold standard for initial market authorization, is often insufficient to guide patients, physicians and policy makers in making decisions on the use of treatments in routine care [
      • Zuidgeest MGP
      • Goetz I
      • Groenwold RHH
      • Irving E
      • van Thiel GJMW
      • Grobbee DE.
      Series: pragmatic trials and real world evidence: paper 1. Introduction.
      ,
      • Rothwell PM.
      External validity of randomised controlled trials: “to whom do the results of this trial apply?.
      ,
      • Sherman RE
      • Anderson SA
      • Dal Pan GJ
      • Gray GW
      • Groxx T
      • Hunter NL
      • et al.
      Real-world evidence – what is it and what can it tell us?.
      • Eichler HG
      • Bloechl-Daum B
      • Broich K
      • Kyrle PA
      • Oderkirk J
      • Rasi G
      • et al.
      Data rich, information poor: can we use electronic health records to create a learning healthcare system for pharmaceuticals?.
      ]. Conventional phase III trials are designed to determine efficacy of a drug (the drug effect under ideal circumstances), and observational studies have limitations in providing evidence on the (relative) effectiveness of treatment options in routine practice. Therefore, interest is growing in methodologies, which incorporate real world elements into clinical trial design, maintaining randomization. Data generated from these so called pragmatic trials offer an attractive opportunity to bridge the gap between conventional randomized clinical trial (RCT) derived efficacy and RWE from observational studies [
      • Zuidgeest MGP
      • Goetz I
      • Groenwold RHH
      • Irving E
      • van Thiel GJMW
      • Grobbee DE.
      Series: pragmatic trials and real world evidence: paper 1. Introduction.

      Food and Drug Administration. Framework for FDA's real-world evidence program. 2018. Available from https://www.fda.gov/media/120060/download [Accessed 13 July, 2021].

      ]. However, progress towards the implementation of real world elements into trials is slow [
      • Chalkidou K
      • Tunis S
      • Whicher D
      • Fowler R
      • Zwarenstein M.
      The role for pragmatic randomized controlled trials (pRCTs) in comparative effectiveness research.
      ,
      • Nicholls SG
      • Carrol K
      • Hey SP
      • Zwarenstein M
      • Zhang JZ
      • Nix HP
      • et al.
      A review of pragmatic trials found a high degree of diversity in design and scope, deficiencies in reporting and trial registry data, and poor indexing.
      ], for several reason including uncertainty about acceptability of the evidence, lack of experience with the methodology and operational challenges associated with such trials [
      • Kolitsopoulos FM
      • Strom BL
      • Faich G
      • Eng SM
      • Kane JM
      • Reynolds RF.
      Lessons learned in the conduct of a global, large simple trial of treatments indicated for schizophrenia.
      ,
      • New JP
      • Bakerly ND
      • Leather D
      • Woodcock A.
      Obtaining real-world evidence: the Salford Lung Study.
      ,
      • Laken MA
      • Dawson R
      • Engelman O
      • Lovelace O
      • Way C
      • Egan BM.
      Comparative effectiveness research in the ‘‘real’’ world: lessons learned in a study of treatment-resistant hypertension.
      ,
      • Califf RM
      • Sugarman J.
      Exploring the ethical and regulatory issues in pragmatic clinical trials.
      ,
      • van Staa TP
      • Dyson L
      • McCann G
      • Padmanabhan S
      • Belatri R
      • Goldacre B
      • et al.
      The opportunities and challenges of pragmatic point-of-care randomised trials using routinely collected electronic records: evaluations of two exemplar trials.
      ,
      • Kalkman S
      • van Thiel GJMW
      • Zuidgeest MGP
      • Goetz I
      • Pfeiffer BM
      • Grobbee DE
      • et al.
      Series: pragmatic trials and real world evidence: paper 4. Informed consent.
      ,
      • Worsley SD
      • Oude Rengerink K
      • Irving E
      • Lejeune S
      • Mol K
      • Collier S
      • et al.
      Series: pragmatic trials and real world evidence: paper 2. Setting, sites, and investigator selection.
      ,
      • Meinecke AK
      • Welsing P
      • Kafatos G
      • Burke D
      • Trelle S
      • Kubin M
      • et al.
      Series: pragmatic trials and real world evidence: paper 8. Data collection and management.
      ,
      • Welsing PM
      • Oude Rengerink K
      • Collier S
      • Eckert L
      • van Smeden M
      • Ciaglia A
      • et al.
      Series: pragmatic trials and real world evidence: paper 6. Outcome measures in the real world.
      ,
      • Zuidgeest MGP
      • Welsing PMJ
      • van Thiel GJMW
      • Ciaglia A
      • Alfonso-Cristancho R
      • Eckert L
      • et al.
      Series: pragmatic trials and real world evidence: paper 5. Usual care and real life comparators.
      ,
      • Oude Rengerink K
      • Kalkman S
      • Collier S
      • Ciaglia A
      • Worsley SD
      • Lightbourne A
      • et al.
      Series: pragmatic trials and real world evidence: paper 3. Patient selection challenges and consequences.
      • Irving E
      • van den Bor R
      • Welsing P
      • Walsh V
      • Alfonso-Cristancho R
      • Harvey C
      Series: pragmatic trials and real world evidence: paper 7. Safety, quality and monitoring.
      ]. In this paper we describe the development, structure and possible applications of a decision support tool for incorporating Real World Elements into clinical trials. This tool aims to facilitate the implementation of such elements intro trial design by guiding users through the possible methodological and operational implications of design choices towards RWE generation. For a more detailed description of the concept of pragmatic trial methodology please see our paper in this journal from 2017 called “Series: Pragmatic trials and real world evidence: Paper 1. Introduction” [
      • Zuidgeest MGP
      • Goetz I
      • Groenwold RHH
      • Irving E
      • van Thiel GJMW
      • Grobbee DE.
      Series: pragmatic trials and real world evidence: paper 1. Introduction.
      ].

      1.1 The need for evidence on relative effectiveness

      Relative effectiveness is the extent to which an intervention does more good than harm compared to one or more alternative interventions when provided under the usual circumstances of health care practice [
      • Zuidgeest MGP
      • Goetz I
      • Groenwold RHH
      • Irving E
      • van Thiel GJMW
      • Grobbee DE.
      Series: pragmatic trials and real world evidence: paper 1. Introduction.
      ]. Conventional phase III RCTs (also referred to as explanatory trials) are not designed to provide evidence on the (relative) effectiveness of treatment options [
      • Zuidgeest MGP
      • Goetz I
      • Groenwold RHH
      • Irving E
      • van Thiel GJMW
      • Grobbee DE.
      Series: pragmatic trials and real world evidence: paper 1. Introduction.
      ]. These trial are usually conducted in selected populations and in a highly controlled and optimized setting, since their aim is to show the efficacy and (short-term) safety of a given drug. A gap between efficacy and effectiveness arises when effect modifiers of treatment, sometimes also referred to as drivers of effectiveness, differ between the trial population/setting and the patients/setting in which the treatment is used after marketing authorization [
      • Rothwell PM.
      External validity of randomised controlled trials: “to whom do the results of this trial apply?.
      ,
      • Nordon C
      • Karcher H
      • Groenwold RHH
      • Ankarfeldt MZ
      • Pichler F
      • Chevrou-Severac H
      • et al.
      The “efficacy-effectiveness gap”: historical background and current conceptualization.
      ]. Such characteristics could include for example age, sex, severity of the disease, concomitant medications, and adherence to treatment, but also more practical elements of trial conduct which may influence clinical practice and with that the observed treatment effect, such as standardization of outcome measurements, trial monitoring and mode of (safety) data collection. To estimate the treatment effect in practice, and guide decision making in practice, a different estimand should be targeted, with a design and analysis that aligns that estimand [

      European Medicines Agency. ICH E9 (R1) addendum on estimands and sensitivity analysis in clinical trials to the guideline on statistical principles for clinical trials. Available from: https://www.ema.europa.eu/en/ich-e9-statistical-principles-clinical-trials [Accessed May 26, 2021].

      ].

      1.2 Real world elements in clinical trial design

      Historically, evidence on relative effectiveness has been generated after marketing-authorization through observational studies. However, there is often uncertainty around the use of this data in decision making because of known and unknown bias, especially regarding prognostic incomparability between patient groups, which cannot be ruled out for nonrandomized study designs [
      • Pocock SJ
      • Elbourne DR.
      Randomized trials or observational tribulations?.
      ,
      • Freemantle N
      • Strack T.
      Real-world effectiveness of new medicines should be evaluated by appropriately designed clinical trials.
      ]. Therefore, interest is growing in methodologies which introduce real world elements into clinical trial design, such as expanding the trial population to better reflect the target population and the choice for a comparison of treatment strategies as used in clinical practice (including flexibility regarding eg, dosing and concomitant interventions). If designed and executed well, such so called pragmatic trials generate evidence on relative effectiveness and allow generalizability (the extent to which the results of a study apply to the population of patients in a particular clinical setting) of trial results to the usual care setting that is reflected in the trial design, while maintaining the strength of randomization [
      • Zuidgeest MGP
      • Goetz I
      • Groenwold RHH
      • Irving E
      • van Thiel GJMW
      • Grobbee DE.
      Series: pragmatic trials and real world evidence: paper 1. Introduction.
      ,
      • Rothwell PM.
      External validity of randomised controlled trials: “to whom do the results of this trial apply?.
      ,

      Food and Drug Administration. Framework for FDA's real-world evidence program. 2018. Available from https://www.fda.gov/media/120060/download [Accessed 13 July, 2021].

      ,
      • Freemantle N
      • Strack T.
      Real-world effectiveness of new medicines should be evaluated by appropriately designed clinical trials.
      ]. In this way, pragmatic trials can provide the type of evidence needed for decision making in routine clinical practice and are an important addition to the available arsenal of study designs.
      Since the introduction of the concept of pragmatic trials by Schwartz and Lellouch in 1967 [
      • Schwartz D
      • Lellouch J.
      Explanatory and pragmatic attitudes in therapeutical trials.
      ] numerous publications have described the concept and methodology of pragmatic trials [
      • Zuidgeest MGP
      • Goetz I
      • Groenwold RHH
      • Irving E
      • van Thiel GJMW
      • Grobbee DE.
      Series: pragmatic trials and real world evidence: paper 1. Introduction.
      ,
      • Thorpe KE
      • Zwarenstein M
      • Oxman AD
      • Treweek S
      • Furberg CD
      • Altman DG
      • et al.
      A pragmatic-explanatory continuum indicator summary (PRECIS): a tool to help trial designers.
      ,
      • Ford I
      • Norrie J.
      Pragmatic trials.
      • Gamerman V
      • Cai T
      • Elsäßer A.
      Pragmatic randomized clinical trials: best practices and statistical guidance.
      ]; and tools, such as PRECIS-2, have been developed to guide trialists as to where on the pragmatic/explanatory continuum a planned trial lies [
      • Thorpe KE
      • Zwarenstein M
      • Oxman AD
      • Treweek S
      • Furberg CD
      • Altman DG
      • et al.
      A pragmatic-explanatory continuum indicator summary (PRECIS): a tool to help trial designers.
      ,
      • Loudon K
      • Treweek S
      • Sullivan F
      • Donnan P
      • Thorpe KE
      • Zwarenstein M.
      The PRECIS-2 tool: designing trials that are fit for purpose.
      ]. This continuum is of importance, because trials can incorporate real world elements in their design to a varying extent (ranging from “only” broadening the patient population to point-of-care-randomized trials where randomization and informed consent are the only deviation from routine clinical practice). The exact research question of the study should drive which real world elements should be considered for the trial design and which not and with that the position of the trial on the pragmatic/explanatory continuum. For example, to determine the true drug effect in a broader population than was included in phase III trials different design choices are required than to guide decision making regarding whether to treat patients with treatment A or B for an asthma exacerbation in an emergency setting.

      1.3 The importance of considering operational aspects

      There are many design choices that can be made to tailor clinical trial design to more closely reflect routine care. Each of these design choices may have consequences, not only on a methodological, but also on an operational level. Equally, operational limitations can impact design choices. Different operational challenges are associated with pragmatic trials than with explanatory trials [
      • Kolitsopoulos FM
      • Strom BL
      • Faich G
      • Eng SM
      • Kane JM
      • Reynolds RF.
      Lessons learned in the conduct of a global, large simple trial of treatments indicated for schizophrenia.
      • Irving E
      • van den Bor R
      • Welsing P
      • Walsh V
      • Alfonso-Cristancho R
      • Harvey C
      Series: pragmatic trials and real world evidence: paper 7. Safety, quality and monitoring.
      ]. Due to the limited experience with pragmatic trials, these challenges may be unanticipated and hamper the successful conduct of these trials. In comparison to conventional explanatory RCTs, in pragmatic trials particular aspects need to be addressed, such as understanding how routine care is delivered in the setting(s) of interest and understanding the potential limitations imposed by engaging sites not experienced in running clinical trials (the later because pragmatic trials seek to perform the study in the setting where patients would be treated in daily life, which in many cases are not the research experienced tertiary care centers).
      For example, methodologically simple-sounding concepts such as “comparing the new treatment strategy to usual care” can prove operationally challenging. How to deal with large differences in usual care between sites, regions or countries? How to deal with changes in usual care during the trial period? And what if usual care includes suboptimal care? Taking all these aspects into account during the design phase of the trial is essential, not only to ensure a smooth conduct of the trial, but also to prevent difficulties in interpreting the results as well as limitations to the generalizability and acceptability of the findings.
      Sometimes the methodologically favorable pragmatic design choice may not be advisable, because operational challenges that arise from this choice can work in the opposite direction. For example, appointing dedicated study staff to a site is often considered a less pragmatic choice [
      • Loudon K
      • Treweek S
      • Sullivan F
      • Donnan P
      • Thorpe KE
      • Zwarenstein M.
      The PRECIS-2 tool: designing trials that are fit for purpose.
      ] as it may change routine clinical practice. However, experience has shown that, for example in primary care, not offering this extra support, may lead to a low, selective sample of primary care practices willing to participate in the trial [
      • van Staa TP
      • Dyson L
      • McCann G
      • Padmanabhan S
      • Belatri R
      • Goldacre B
      • et al.
      The opportunities and challenges of pragmatic point-of-care randomised trials using routinely collected electronic records: evaluations of two exemplar trials.
      ]. The usual clinical care treatment in these practices may well differ from that in the nonparticipating practices. If the application of dedicated study staff leads to inclusion of a broader group of practices this not-so-very-pragmatic design choice may thus actually improve generalizability of the trial findings, instead of decreasing it; given that it is set up in a way to relieve the burden of informed consent procedures and other trial related activities, but with as less impact on routine clinical practice and the delivery of the interventions as possible.
      These examples show that operational challenges (and their solutions) can strongly impact the generalizability of trial findings and, depending on the research question, may compromise the original goal of the trial design. We propose that when designing or evaluating the findings of trials with a more pragmatic intent, attention should be paid to the effects of design choices not only on generalizability, but also on the other two key methodological principles of clinical trials, validity and precision of treatment effects, as well as operational challenges. This will help clinical project teams to better understand the consequences of their choices and evaluators of the results to better value the trial findings and understand whether they answer the research question the trial aimed to address.
      In order to help clinical project teams and evaluators to address the specific aspects of pragmatic trials, researchers of the IMI GetReal project, in cocreation with a broad stakeholder group, have developed the GetReal Trial Tool.

      1.4 What does the GetReal Trial Tool add?

      The GetReal Trial Tool builds on the aforementioned previous work on pragmatic trials by combining the methodological framework of pragmatic trials with three additional aspects, which need to be taken into account, to make balanced design choices towards Real World Evidence generation:
      • 1.
        A much more granular overview of the options for each “more pragmatic” design choice, to provide hands-on guidance to trialists beyond the methodological principle (for example, what are the different modes available to incorporate usual care as the comparator)
      • 2.
        Nine important possible implications of design choices, beyond generalizability,
        • a
          Regarding the other two key methodological principles of clinical trials: validity (described as “risk of bias”) and precision
        • b
          Regarding acceptability of the trial by various stakeholders, as well as cost and duration (which strongly influences whether a trial can be conducted in practice)
      • 3.
        Possible operational challenges that can arise from a certain design choice (being able to anticipate these up front can again strongly increase the chances of the trial being successfully executed)
      As such the GetReal Trial Tool is a comprehensive decision support tool for clinical trial design, taking methodology, acceptability and operational feasibility into account. The tool is not intended to score pragmatism but to aid in making more pragmatic design choices and/or evaluate the consequences of such choices.

      1.5 Development of the GetReal Trial Tool

      The development of the tool was initiated by a group of about 30 people from 16 different organizations, mostly (clinical) epidemiologists, statisticians, experts in clinical trial operations and/or real-world evidence, patient representatives and ethicists, from academia, industry, patient organizations and SMEs.

      2. The tool content

      As already highlighted in the introduction, the tool combines information on trial design elements, their possible implications and possible operational challenges related to these design choices. A description of these concepts, as used in the GetReal Trial Tool, can be found in Text Box 1 and an overview of the interplay between the concepts is visualized in Fig. 1.
      Fig. 1
      Fig. 1Visualization of the interplay between design choices, implications and operational challenges.
      Tabled 11 A description of the concepts used in the GetReal Trial Tool
      Trial design elements: An overview of choices that can be made when designing a trial, with a focus on those choices that differentiate between a more pragmatic and more explanatory trial. For example, whether to protocolize dosing in the trial or leave it at the discretion of the treating healthcare professional. These trial design choices have been captured as questions under the seven domains of the GetReal Trial Tool.
      Implications: The effect a design choice or operational challenge may have on interpretation of the results (generalizability, validity, precision), ethical and stakeholder acceptability (patient, prescriber, regulatory and HTA body) or the required resources or study duration.
      Methodological implications: The effect of design choices or operational challenges on the interpretation of the results (generalizability, validity, precision).
      Acceptability implications: Ethical and stakeholder acceptability (patient, prescriber, regulatory and HTA body)
      Feasibility implications: Implications for the required resources or study duration. These need to be distinguished from the operational challenges in the sense that these implications are a direct logical result of a design choice; whereas operational challenges are practical issues that may or may not occur depending on the exact specificities of where and how the trial is executed.
      Operational challenges: Practical issues that may arise during the execution of the trial, for example, low participation of sites due to required training and study burden, as a consequence of specific choices made in the trial design.
      The content of the tool is based on a combination of extensive literature review, in-depth stakeholder interviews, pragmatic trial study team conversations, and consortium member input. It has been validated by a team of trialists, clinical trial operational experts and epidemiologists. The methods and results of the literature review and stakeholder interviews have been published elsewhere [
      • Zuidgeest MGP
      • Goetz I
      • Groenwold RHH
      • Irving E
      • van Thiel GJMW
      • Grobbee DE.
      Series: pragmatic trials and real world evidence: paper 1. Introduction.
      ,
      • Kalkman S
      • van Thiel GJMW
      • Zuidgeest MGP
      • Goetz I
      • Pfeiffer BM
      • Grobbee DE
      • et al.
      Series: pragmatic trials and real world evidence: paper 4. Informed consent.
      ,
      • Worsley SD
      • Oude Rengerink K
      • Irving E
      • Lejeune S
      • Mol K
      • Collier S
      • et al.
      Series: pragmatic trials and real world evidence: paper 2. Setting, sites, and investigator selection.
      ,
      • Meinecke AK
      • Welsing P
      • Kafatos G
      • Burke D
      • Trelle S
      • Kubin M
      • et al.
      Series: pragmatic trials and real world evidence: paper 8. Data collection and management.
      ,
      • Welsing PM
      • Oude Rengerink K
      • Collier S
      • Eckert L
      • van Smeden M
      • Ciaglia A
      • et al.
      Series: pragmatic trials and real world evidence: paper 6. Outcome measures in the real world.
      ,
      • Zuidgeest MGP
      • Welsing PMJ
      • van Thiel GJMW
      • Ciaglia A
      • Alfonso-Cristancho R
      • Eckert L
      • et al.
      Series: pragmatic trials and real world evidence: paper 5. Usual care and real life comparators.
      ,
      • Oude Rengerink K
      • Kalkman S
      • Collier S
      • Ciaglia A
      • Worsley SD
      • Lightbourne A
      • et al.
      Series: pragmatic trials and real world evidence: paper 3. Patient selection challenges and consequences.
      • Irving E
      • van den Bor R
      • Welsing P
      • Walsh V
      • Alfonso-Cristancho R
      • Harvey C
      Series: pragmatic trials and real world evidence: paper 7. Safety, quality and monitoring.
      ]. A new literature search has been performed to include published challenges with pragmatic trials until March 2021 (publication expected shortly) and where needed the tool content regarding operational challenges has been updated.

      2.1 Development of the Trial Tool domains and design choices

      As a first step to content development a list of trial design choices that can be made as part of pragmatic trial design was created. The list was further refined and reviewed until agreement was reached. These choices were subsequently grouped into seven overarching domains for trial design, relating to participant, site, outcome, comparator, data, safety and quality aspects of trial design. See Table 1 for a more detailed description of the domains and the number of design choices identified per domain. Details of the domains, questions and design choices are presented in Appendix A. Details on the selected pragmatic trial domains and the most important design choices are also described in the aforementioned series of consecutive publications by the GetReal Consortium [
      • Zuidgeest MGP
      • Goetz I
      • Groenwold RHH
      • Irving E
      • van Thiel GJMW
      • Grobbee DE.
      Series: pragmatic trials and real world evidence: paper 1. Introduction.
      ,
      • Kalkman S
      • van Thiel GJMW
      • Zuidgeest MGP
      • Goetz I
      • Pfeiffer BM
      • Grobbee DE
      • et al.
      Series: pragmatic trials and real world evidence: paper 4. Informed consent.
      • Irving E
      • van den Bor R
      • Welsing P
      • Walsh V
      • Alfonso-Cristancho R
      • Harvey C
      Series: pragmatic trials and real world evidence: paper 7. Safety, quality and monitoring.
      ], and a more elaborate discussion is beyond the scope of this paper.
      Table 1The domains and number of design choices per domain in the GetReal Trial Tool
      DomainDescriptionNumber of design choices
      Participant selection, recruitment and attritionThis domain looks into the eligibility criteria for the trial participants, and recruitment and attrition strategies.5
      Site Selection and recruitmentThis domain evaluates the settings where the trial is performed, as well as selection, recruitment and set-up of the sites.6
      Outcome selection and measurementThis domain touches upon which outcomes are selected for the trial (eg, disease-specific survival/mortality, clinical outcomes, life impact) and how they are measured (alignment with routine practice, standardization).10
      Randomization, comparator choice and treatment strategiesThis domain looks into how the treatment strategies are implemented and compared in the trial (including strategies on randomization, blinding, switching, comparator choice, treatment supply and reimbursement).13
      Data collectionThis domain addresses the options for data collection (eg, use of existing data collection systems such as Electronic Health Records), validation and linkage.4
      Safety monitoringThis domain considers what safety data are collected and how they are handled.3
      Monitoring of trial conduct and data qualityThis domain looks into which aspects of the study conduct are monitored and how.2

      2.2 Identification of possible implications of design choices and their evaluation

      Three methodological implications, five acceptability implications and two feasibility implications were selected based on discussions and integrated into the tool. To be able to compare the possible implications of design choices a color coding system was developed. This was preferred to a numerical scoring system to avoid calculation of a “final” score for a trial since a single number cannot capture the complexities of trial design. The color coding shows, for each design option, whether there is opportunity for improvement in the design of the study for a specific implication. There are four possible colors, green stands for “optimal,” orange for “opportunity for improvement,” pink for “more opportunity for improvement” and grey for “no impact” expected. For example, both use of a placebo comparator and blinding of participants and/or health care provider for treatment allocation are coded as pink “more opportunity for improvement” for generalizability. This indicates that generalizability could be improved by adapting these design choices, for example by comparing to an active comparator in an open-label fashion. It is up to the trial designers or evaluators to determine whether such an “improvement of generalizability” is required, as this depends on the research question of the trial. Each color coding is accompanied by a written brief explanation of the implication, where available including a reference to relevant literature. The group developing the tool evaluated the possible implications of each design choice, assigned a color code and added the written explanation for the color given. See Fig. 2 for the list of implications, their definitions and the color coding system. When using the tool to design a trial it depends on the timing (pre- or post-registration) and setting of the trial whether all of these implications are of relevance, for example, costs may be less of an issue in industry-sponsored trials whereas for phase IV investigator-initiated trials the regulatory acceptability may be less relevant. When using the tool to assess a trial the cost and duration aspect do not need to be taken into account. Therefore, the color coding is only to be used to determine whether there is room for improvement of the specific design choice it relates to, not to compare colors across design choices. It is up to the trial team to determine whether for example, an increase in precision outweighs a possible decrease in generalizability.
      Fig. 2
      Fig. 2List of implications of design choices as presented in the tool and the color coding system.

      2.3 Collection and scoring of possible operational challenges

      Based on the findings from the extensive literature review, in-depth stakeholder interviews, pragmatic trial study team conversations, and consortium member input, a list of possible operational challenges of pragmatic design choices was constructed for each of the design choices, defined under 2.1. Again, for each operational challenge the possible implications of that challenge were evaluated and explained with a short text.

      2.4 Content validation: reaching consensus on the color coding, description of implications and operational challenges

      To validate and enrich the content of the GetReal Trial Tool, regarding implications and operational challenges, seven expert (group) sessions, involving 35 experts from academia and industry as well as clinicians, were organized. This consisted of two sessions on the methodological implications, and one session each on Health Technology Assessment body, prescriber, regulatory and ethical acceptability as well as operational challenges. The participants or experts of each session varied, depending on the content to be validated; the methodological implications sessions involved professors of epidemiology and public health and epidemiologists (mainly from academia) while the HTA acceptability session involved policy experts from Health Technology Agencies. Regulatory acceptability of the content was validated by statisticians, epidemiologists, as well as pharmacovigilance specialists working in the regulatory system. Prescriber acceptability was validated by a professor of general practice and a general practitioner and epidemiologist, whereas ethical acceptability was validated by an associate professor of medical ethics. The feasibility implications and operational challenges were validated by a group of operational experts from industry and academia as well as epidemiologists. Patient acceptability was already evaluated by the experts of a broad patient organization so no further validation sessions took place.
      During these sessions for each design choice, the color coding and wording regarding the respective implications was discussed (session 1–6). Discussion was continued until agreement was reached. During the session on operational challenges per design choice, the operational challenges were reviewed, as well as the described possible implications of each operational challenge (session 7).

      3. The tool functionalities and structure

      3.1 Development of the tool functionalities

      In order to develop a tool that is intuitive and easy to use in practice, an external commercial serious gaming company was involved to help build an attractive display for the complex interplay between design choices, operational challenges and implications. The principal focus points for the development of the functionalities of the tool were based on the requirements for the tool users to (1) not get lost in the vast amount of information provided in the tool, (2) keep track of progress through the different domains and design choices in the tool and (3) to be able to go through a large content base in an intuitive manner.
      The scrum framework (see scrum.org) was used to codevelop the tool functionalities. The intermediate products included a canvas (describing elements such as the aim of the tool, the target group, the envisioned core features, the project boundaries etc.), a paper prototype (a paper version showing the type of content and the flow through the tool), a Look & Feel version (of the interface and envisioned functionalities of the tool), a functional design prototype (with limited content but with the functionalities in place), a beta version and the final web-based version of the tool.

      3.2 Piloting of the tool functionalities

      The tool functionalities have been piloted extensively, first before being made available online in June 2017 and again after revisions made to the tool in 2019.
      In 2017 former GetReal partners and additional relevant stakeholder organizations were invited to participate in the pilot (N = 33). Responses were received (12 surveys, five F2F meetings) from pharmaceutical companies (five), patient organizations (two), Contract Research Organizations (one), nonprofit organizations (one), universities/university hospitals (six), HTA bodies (one) and regulatory agencies (one). The conclusions from the pilot were that the tool was fully functional and well received, albeit hosting was somewhat unstable. The “gamification”/visualization was seen as a positive feature of the tool. Changes that were made to the tool included increasing and optimizing the export functionalities, rephrasing text where unclear and optimizing the stability of the tool by changing the hosting environment.
      In 2019, the tool underwent a second round of extensive piloting, consisting of an internal interview round with eight consortium members, from industry, academia and patient organizations. This was followed by two focus sessions where the tool was used to review a trial protocol (13 people in total, six industries, two Contract Research Organizations and five academia). Subsequently, the tool was presented face-to-face to a group of 17 external RWE experts, who provided feedback on the tool in a round table discussion. Based on these pilots the name of the tool, the visuals and the wording around the tool has been adapted to better reflect the possibilities of the tool. The content of the tool did not require changes at this stage.
      Validation of the tool, assessing protocols of conducted trials, has been completed and is planned to be published shortly. In this validation both the ability of the tool to highlight the operational challenges that were actually experienced by trial teams conducting these trials and the interrater agreement on which design choice has been made for a trial, based on a trial protocol, have been assessed.

      3.3 Tool functionalities and structure

      The current tool provides a navigation wheel with the seven domains, a visual aid on progress through the design choices, a two-level approach with the design choices and their possible implications on the first level and the operational challenges with their possible implications on the second level, an overview possibility per domain and a note-taking function. See Fig. 3 for a snapshot of the main screen of the tool. The tool provides both a descriptive (Excel) and a visual export function (Fingerprint, see Appendix B).
      Fig. 3
      Fig. 3A snapshot of the main screen of the GetReal Trial Tool.

      4. Discussion

      The GetReal Trial Tool is developed to offer step-by-step guidance to evaluate the options and implications of introducing RWE in clinical trial design and help to optimize trial design and maximize the impact of the results. It allows users to assess the impact of design choices on generalizability of a clinical trial to the routine clinical practice the trial aims to reflect, while taking into account other aspects of the trial, such as risk of bias, precision, acceptability and operational feasibility. The tool enables trial teams and other stakeholders to access information from the scientific literature, combined with the knowledge of over 100 professional clinical trial experts from academia, pharmaceutical companies, HTA agencies, patient organizations, and regulators.
      During trial design, the tool offers an easy to use interface, which supports users to navigate easily through different aspects of their trial design and reach a balanced decision on a design that is expected to be not only fit for purpose in theory but that also has the best chance of being successful in practice. The tool helps ensure all key aspects of the trial design and associated operational challenges are considered as a team builds and optimizes their trial design in light of a specific research question.
      The tool will not make decisions for the user but rather highlight possible consequences of design choices, which need to be interpreted by the user to determine to what extent these consequences might apply to the specific trial the user is designing or evaluating, including the specific therapeutic area, intervention and health care setting. As such, the tool can best be used by a team with combined knowledge of clinical trial design, the disease area, health care setting and usual care options in scope for the specific research question.
      The tool can also be of use at a later stage, when evaluating and communicating trial findings with key decision makers and the scientific community, by giving transparency into which design choices of the trial might have influenced generalizability and other aspects of the trial. The tool is also being regularly and successfully used in online educational courses on RWE and trial design.
      It is explicitly not the aim of the tool to provide a “final score” regarding “level of pragmatism” for a trial, as the authors believe that each trial can only be evaluated taking into account the specific research question and context of the setting the trial aims to provide results for. In addition, similar color coding might have different implications for each trial, where for a sponsor with a limited budget higher costs might be a showstopper, while for another sponsor including more sites might be a more serious challenge.
      Limitations are related to the fact that experiences with operational aspects of clinical trial conduct often remain unpublished. The tool only shows those operational challenges that have been published or were brought in by the stakeholders involved in the development of the tool. Even though this information is periodically updated, as such, the tool most likely does not capture all operational challenges experienced in this field. In addition, to be able to effectively use the tool, some general knowledge on clinical trial design and execution is required.
      In conclusion, the GetReal Trial Tool offers an accessible and solid knowledge base to assess, design or discuss clinical trials in light of RWE generation. The tool is open access and can be used without entering any confidential trial information. The tool can be found on www.getrealtrialtool.eu. The GetReal team welcomes any feedback on the functionalities and content of the tool, so that they can continue to optimize the tool.

      Acknowledgement

      The GetReal Trial Tool is the product of the collective contribution of a much broader multidisciplinary team, as part of the IMI GetReal work package 3 and IMI GetReal Initiative Taskforce A projects and has received input from many external stakeholders. The following people provided substantial contributions to the development of the tool: Matthias Egger, Sven Trelle, Robin Vernooij, Thomas Kumke, Pieter Stolk, Antonio Ciaglia, George Kapatos, Rolf Groenwold, Evert Hoogendoorn, Laura de Ruijter, Simon Groenwolt, Vera Koot, Theo van Wijk

      REFERENCES

        • Zuidgeest MGP
        • Goetz I
        • Groenwold RHH
        • Irving E
        • van Thiel GJMW
        • Grobbee DE.
        Series: pragmatic trials and real world evidence: paper 1. Introduction.
        J Clin Epidemiol. 2017; 88: 7-13
        • Rothwell PM.
        External validity of randomised controlled trials: “to whom do the results of this trial apply?.
        Lancet. 2005; 365: 82-93
        • Sherman RE
        • Anderson SA
        • Dal Pan GJ
        • Gray GW
        • Groxx T
        • Hunter NL
        • et al.
        Real-world evidence – what is it and what can it tell us?.
        N Engl J Med. 2016; 375: 2293-2297
      1. Food and Drug Administration. Framework for FDA's real-world evidence program. 2018. Available from https://www.fda.gov/media/120060/download [Accessed 13 July, 2021].

        • Eichler HG
        • Bloechl-Daum B
        • Broich K
        • Kyrle PA
        • Oderkirk J
        • Rasi G
        • et al.
        Data rich, information poor: can we use electronic health records to create a learning healthcare system for pharmaceuticals?.
        Clin Pharmacol Ther. 2018; 105: 912-922
        • Chalkidou K
        • Tunis S
        • Whicher D
        • Fowler R
        • Zwarenstein M.
        The role for pragmatic randomized controlled trials (pRCTs) in comparative effectiveness research.
        Clin Trials. 2012; 9: 436-446
        • Nicholls SG
        • Carrol K
        • Hey SP
        • Zwarenstein M
        • Zhang JZ
        • Nix HP
        • et al.
        A review of pragmatic trials found a high degree of diversity in design and scope, deficiencies in reporting and trial registry data, and poor indexing.
        J Clin Epidemiol. 2021; 137: 45-57
        • Kolitsopoulos FM
        • Strom BL
        • Faich G
        • Eng SM
        • Kane JM
        • Reynolds RF.
        Lessons learned in the conduct of a global, large simple trial of treatments indicated for schizophrenia.
        Contemp Clin Trials. 2013; 34: 239-247
        • New JP
        • Bakerly ND
        • Leather D
        • Woodcock A.
        Obtaining real-world evidence: the Salford Lung Study.
        Thorax. 2014; 69: 1152-1154
        • Laken MA
        • Dawson R
        • Engelman O
        • Lovelace O
        • Way C
        • Egan BM.
        Comparative effectiveness research in the ‘‘real’’ world: lessons learned in a study of treatment-resistant hypertension.
        J Am Soc Hypertens. 2013; 7: 95-101
        • Califf RM
        • Sugarman J.
        Exploring the ethical and regulatory issues in pragmatic clinical trials.
        Clin Trials. 2015; 12: 436-441
        • van Staa TP
        • Dyson L
        • McCann G
        • Padmanabhan S
        • Belatri R
        • Goldacre B
        • et al.
        The opportunities and challenges of pragmatic point-of-care randomised trials using routinely collected electronic records: evaluations of two exemplar trials.
        Health Technol Assess. 2014; 18: 1-146
        • Kalkman S
        • van Thiel GJMW
        • Zuidgeest MGP
        • Goetz I
        • Pfeiffer BM
        • Grobbee DE
        • et al.
        Series: pragmatic trials and real world evidence: paper 4. Informed consent.
        J Clin Epidemiol. 2017; 89: 181-187
        • Worsley SD
        • Oude Rengerink K
        • Irving E
        • Lejeune S
        • Mol K
        • Collier S
        • et al.
        Series: pragmatic trials and real world evidence: paper 2. Setting, sites, and investigator selection.
        J Clin Epidemiol. 2017; 88: 14-20
        • Meinecke AK
        • Welsing P
        • Kafatos G
        • Burke D
        • Trelle S
        • Kubin M
        • et al.
        Series: pragmatic trials and real world evidence: paper 8. Data collection and management.
        J Clin Epidemiol. 2017; 91: 13-22
        • Welsing PM
        • Oude Rengerink K
        • Collier S
        • Eckert L
        • van Smeden M
        • Ciaglia A
        • et al.
        Series: pragmatic trials and real world evidence: paper 6. Outcome measures in the real world.
        J Clin Epidemiol. 2017; 90: 99-107
        • Zuidgeest MGP
        • Welsing PMJ
        • van Thiel GJMW
        • Ciaglia A
        • Alfonso-Cristancho R
        • Eckert L
        • et al.
        Series: pragmatic trials and real world evidence: paper 5. Usual care and real life comparators.
        J Clin Epidemiol. 2017; 90: 92-98
        • Oude Rengerink K
        • Kalkman S
        • Collier S
        • Ciaglia A
        • Worsley SD
        • Lightbourne A
        • et al.
        Series: pragmatic trials and real world evidence: paper 3. Patient selection challenges and consequences.
        J Clin Epidemiol. 2017; 89: 173-180
        • Irving E
        • van den Bor R
        • Welsing P
        • Walsh V
        • Alfonso-Cristancho R
        • Harvey C
        Series: pragmatic trials and real world evidence: paper 7. Safety, quality and monitoring.
        J Clin Epidemiol. 2017; 91: 6-12
        • Nordon C
        • Karcher H
        • Groenwold RHH
        • Ankarfeldt MZ
        • Pichler F
        • Chevrou-Severac H
        • et al.
        The “efficacy-effectiveness gap”: historical background and current conceptualization.
        Value Health. 2016; 19: 75-81
      2. European Medicines Agency. ICH E9 (R1) addendum on estimands and sensitivity analysis in clinical trials to the guideline on statistical principles for clinical trials. Available from: https://www.ema.europa.eu/en/ich-e9-statistical-principles-clinical-trials [Accessed May 26, 2021].

        • Pocock SJ
        • Elbourne DR.
        Randomized trials or observational tribulations?.
        N Engl J Med. 2000; 342: 1907-1909
        • Freemantle N
        • Strack T.
        Real-world effectiveness of new medicines should be evaluated by appropriately designed clinical trials.
        J Clin Epidemiol. 2010; 63: 1053-1058
        • Schwartz D
        • Lellouch J.
        Explanatory and pragmatic attitudes in therapeutical trials.
        J Clin Epidemiol. 2009; 62: 499-505
        • Thorpe KE
        • Zwarenstein M
        • Oxman AD
        • Treweek S
        • Furberg CD
        • Altman DG
        • et al.
        A pragmatic-explanatory continuum indicator summary (PRECIS): a tool to help trial designers.
        J Clin Epidemiol. 2009; 62: 464-475
        • Ford I
        • Norrie J.
        Pragmatic trials.
        N Engl J Med. 2016; 375: 454-463
        • Gamerman V
        • Cai T
        • Elsäßer A.
        Pragmatic randomized clinical trials: best practices and statistical guidance.
        Health Serv Outcomes Res Method. 2019; 19: 23-35
        • Loudon K
        • Treweek S
        • Sullivan F
        • Donnan P
        • Thorpe KE
        • Zwarenstein M.
        The PRECIS-2 tool: designing trials that are fit for purpose.
        BMJ. 2015; 350: h2147