If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
This is the introductory paper in a series of eight papers. In this series, we integrate the theoretical design options with the practice of conducting pragmatic trials. For most new market-approved treatments, the clinical evidence is insufficient to fully guide physicians and policy makers in choosing the optimal treatment for their patients. Pragmatic trials can fill this gap, by providing evidence on the relative effectiveness of a treatment strategy in routine clinical practice, already in an early phase of development, while maintaining the strength of randomized controlled trials. Selecting the setting, study population, mode of intervention, comparator, and outcome are crucial in designing pragmatic trials. In combination with monitoring and data collection that does not change routine care, this will enable appropriate generalization to the target patient group in clinical practice. To benefit from the full potential of pragmatic trials, there is a need for guidance and tools in designing these studies while ensuring operational feasibility. This paper introduces the concept of pragmatic trial design. The complex interplay between pragmatic design options, feasibility, stakeholder acceptability, validity, precision, and generalizability will be clarified. In this way, balanced design choices can be made in pragmatic trials with an optimal chance of success in practice.
Pragmatic trials offer the opportunity to obtain real-word data on the relative effectiveness of a treatment in an early phase of development, thus addressing the need for real-world evidence.
•
Opting for pragmatic trial characteristics may lead to different and unanticipated operational challenges compared to explanatory trials
What this adds to what was known?
•
In this introductory paper in a series of 8 papers on pragmatic trials we explain and discuss the main characteristics of pragmatic trial design, and the complex interplay with the operational practicality of implementation.
•
Each consecutive paper in this series will focus on a domain for which specific design choices need to be made in a pragmatic trial: the setting; the study population; operationalization of the intervention and choice of comparator; the outcome measure as well as data management and monitoring.
•
For each domain the papers will integrate the theoretical design options for pragmatic trials with the practice of pragmatic clinical trial conduct and raise awareness of the impact of design choices. Emphasis will be on operational implications, ethical considerations, stakeholder preferences, generalizability, validity and precision.
What is the implication and what should change now?
•
To gain the benefit of the full potential of pragmatic trials, there is a need for guidance and tools in designing these trials while ensuring operational feasibility.
1. Introduction
Evidence on the benefits and risks of treatments in health care can be obtained through several types of research, roughly grouped into either randomized controlled trials (RCTs) or observational studies. Research aimed at synthesizing evidence combines the results of different trials (through either direct or indirect comparison) [
]. It has been widely acknowledged that for most new treatments, the evidence at the moment of market approval is insufficient to fully guide decisions by physicians and policy makers to select the best treatment for patients in routine clinical practice [
Real-world evidence is the evidence derived from the analysis and/or synthesis of real-world data. It is an umbrella term for data regarding the effects of health interventions (e.g., safety, effectiveness, resource use, etc.) that are not collected in the context of highly controlled RCTs [
Updated glossary of definitions of common terms.2015; (Available at https://www.imi-getreal.eu/Portals/1/Documents/Publications/D1.3%20GetReal%20Glossary%20-%20Update.pdf. Accessed August 9, 2016)
] and is assumed to provide data that are applicable to the real-life use and users of drug treatments, including data on relative effectiveness. Relative effectiveness is the extent to which an intervention does more good than harm compared to one or more alternative interventions when provided under the usual circumstances of health care practice.
Both the traditional phase III RCTs and postlaunch observational studies have limitations in providing real-world evidence on the (relative) effectiveness of treatment options [
]. The first because these trials are usually conducted in selected populations, in a highly controlled setting, optimized to show the effect of the drug. The second because bias, especially prognostic incomparability between patient groups in observational research, cannot be ruled out. Pragmatic trials are a valid option to provide evidence to address the issues that patients, clinicians, and policy makers face in real life [
]. In this paper, we discuss the main characteristics of pragmatic trials as well as the operational challenges of their conduct. In addition, we discuss the opportunities that pragmatic trials provide to generate real-world evidence.
]. They intend to explore the effectiveness of a new drug or treatment in day-to-day clinical practice without altering the normal patient and physician behavior. Yet, whereas such observational data are generalizable to routine clinical practice, they are also more likely to be confounded and therefore impact validity (see Box1).
Suppose a study aims to test whether a new drug is more effective in reducing blood pressure compared to currently existing treatment options, an observational study would typically compare the blood pressure records from a group of patients who uses the new drug to a second patient group using the current medication. The observed mean difference in systolic blood pressure between the groups may, however, not validly be attributed to a difference in effectiveness of medication. Other differences between the patient groups could explain the observed differences in blood pressure. For example, the new drug could have been prescribed to more difficult to treat or treatment-resistant patients. Differences in drug tolerance or expectations of superior efficacy of the new drug could also affect the results. Although many advanced analytical approaches have been developed to reduce such confounding effects in comparative observations, they cannot exclude prognostic incomparability of patient groups. A Cochrane review from 2014 concluded that there is little evidence for significant effect estimate differences between observational studies and RCTs [
]. However, a more recent meta-epidemiological survey has shown that studies of routinely collected health data could give different answers from subsequent RCTs on the same clinical questions and may substantially overestimate treatment effects in spite of sophisticated methods to remove confounding [
The understanding of the main sources of confounding in observational research has been an important driver of the central role for randomized trials in drug research [
]. Randomization aims to remove differences in both unknown and known factors that lead to prognostic incomparability between patient groups, for example, resulting from confounding by indication [
]. If group sizes are large enough, the random allocation of patients to different treatments assures that any effect in clinical outcome can be attributed to the differences between interventions.
3. Explanatory and pragmatic trials
Randomized trials are often considered to be the gold standard for clinical development [
]. They are regarded as the most robust approach in clarifying the efficacy/risk profile and are typically performed in a highly selected patient group in a highly controlled environment. All other features of such trials are secondary to randomization and a matter of choice rather than of principle [
]. For example, blinding aims to equalize the treatment effects that are unrelated to efficacy per se. This includes both direct (“true”) placebo effects and the effects from behavioral changes that are evoked by the patient's or physician's knowledge and expectation of the treatment status. These are collectively known as “extraneous effects.” In some situations, however, incorporating such extraneous effects will yield results that are more relevant than the isolated effect of a medicine. This particularly applies when the goal is to obtain a broader effectiveness estimate that reflects the real-world situation [
], aiming at both estimating efficacy and understanding the biological underpinnings of a difference between two treatments. These trials tend to include highly selected patients and follow a strict treatment protocol to address the question whether a new treatment is efficacious and safe [
], offer the opportunity to combine the real-world nature of an observational study with the scientific rigor of a randomized trial and thereby give better answers to questions that are relevant to day-to-day clinical practice. Evidence from these trials is specifically relevant when treatment options do already exist for the disease under study and when the real-life situation, including extraneous factors, is expected to influence the treatment effect. Pragmatic and explanatory trials represent ends of a continuum rather than distinct entities [
] (next to discussing analytical techniques and ethical considerations) defined three areas within the field of trial design in which a choice can be made between a predominant explanatory or a pragmatic approach:
1.
Definition of the treatment: extraneous effects are either equalized in the comparison between the groups to study the “true” effects of the treatment (explanatory) or included in the full set of determinants of an overall treatment response (pragmatic).
2.
Assessment of the results: biologically meaningful (explanatory) vs. meaningful for decision-making in routine clinical practice (pragmatic).
3.
Choice of subjects: highly selected patients with the maximal probability to reveal a treatment effect (explanatory) vs. target group patients that are encountered in clinical practice (pragmatic).
The definition of the treatment addresses the nature of the comparison that is studied. To illustrate their point, Schwartz and Lellouch [
] describe the design options of a randomized trial addressing whether the pretreatment of cancer patients with a sensitizing drug enhanced the effect of radiotherapy. Two options were available for the comparator arm of the trial. The first was to treat the patients according to routine clinical practice, that is, to proceed directly with radiotherapy, without initial placebo administration. Placebo and other extraneous effects cannot be ruled out in this design. The second option was to start radiotherapy after the administration of a placebo drug. Although this approach would enable the removal of extraneous effects from the comparison, it would not reflect usual care as the start of radiotherapy would be delayed compared to routine clinical practice. The first option complies with a pragmatic approach (choosing between two treatments), and the second reflects an explanatory approach (aiming at understanding the difference in pharmacological action between two treatments) [
In short, an important decision to make while designing a trial is the choice of the comparator group and whether placebo and other extraneous effects should be ruled out. The treatment response in a pragmatic trial is the total difference between two treatment strategies, including the associated placebo and other extraneous effects (see Fig. 1). Compared to an explanatory trial, this could result in a more realistic reflection of the likely response in patients treated in routine clinical practice [
Assessing the comparative effectiveness of long-acting injectable vs. oral antipsychotic medications in the prevention of relapse provides a case study in comparative effectiveness research in psychiatry.
Assessment of the results addresses the question which outcome will truly help decision makers choose between treatment options. For example, a significant drop in blood pressure does not necessarily tell a physician which treatment option has the best chance of preventing myocardial infarction in his patients.
The third distinction between pragmatic and explanatory trials, the choice of subjects, has a strong bearing on the extent of the generalizability of the results to the patients who would receive the treatment in the real world. Explanatory trials often investigate a highly selected patient group, excluding patients with common comorbidities, risk factors, and other potential modifiers of drug response. Although this approach maximizes the possibility to reveal a treatment effect, it may also limit the generalizability of the results to the patient population that will receive the treatment in routine clinical practice, after market authorization is granted (see Fig. 1).
In this context, the choice of setting also needs to be introduced as an important design choice. To truly capture the extraneous effects that are part of the treatment strategy and the patients who would receive the treatment in the real world (the pragmatic approach), the setting in which the trial is conducted should be the one in which the patients are treated in real life. On top of that routine clinical practice should not be changed by the trial conduct, which has implications for the monitoring and data collection plan of pragmatic trials. The reason for considering the setting is the probability that a particular setting may impact treatment responses.
] brought the principles of pragmatism and generalizability together in the PRECIS tool, recently updated to the PRECIS-2 tool, which has been introduced to support making design choices that are consistent with the intended purpose of the trial [
The Pragmatic-Explanatory Continuum Indicator Summary (PRECIS) instrument was useful for refining a randomized trial design: experiences from an investigative team.
How pragmatic or explanatory is the randomized, controlled trial? The application and enhancement of the PRECIS tool to the evaluation of a smoking cessation trial.
]. With PRECIS-2, a multidisciplinary team can score a proposed trial design on nine items, such as eligibility, setting, follow-up and primary outcome, to determine to which extend it is in alignment with routine clinical practice [
]. PRECIS-2 is a valuable tool in assessing the theoretical match between more pragmatic trial decisions and their influence on generalizability of the trial results to the intended setting.
Of note, a trial performed in a very specific context (e.g., the use of acute migraine therapy in the emergency department) can be perfectly pragmatic in the sense that it aims to inform clinical decision-making [
] even though it yields a limited generalizability to other clinical settings.
Any characteristic of a patient or setting that impacts the benefit or risk of a treatment will affect generalizability if that characteristic differs between the trial population/setting and the patient group or setting where the results are applied to. Such characteristics could include age, sex, severity of the disease, concomitant medications, and adherence to treatment. The technical term for this impact on treatment effects is treatment effect modification. Generalizability may be impaired when the treatment outcomes of patients obtained in a highly selective explanatory trial are applied to patients who are encountered in routine clinical practice in the real world where responses are different [
Generalizability is not compromised by patient features that are unlikely to affect the effects of the treatment, such as eye color. Assessing the degree of generalizability of trial results therefore requires knowledge about modifiers of treatment effect. However, these are not always measured or known (e.g., genetic factors). The application of the results to the patients who will be treated in routine clinical practice is assured by enrolling participants in the trial that are similar to the target group regarding the characteristics that modify drug response and treating them as would be done in routine clinical practice.
The selection of the study population and setting is crucial if a trial is directly aimed at treatment decision-making in routine clinical practice. This will be discussed in more detail in the second and third paper in this series.
6. To recap
The ideal pragmatic trial aims at validly capturing the full effect of a treatment strategy in the real world. This entails the comparison of randomized groups of patients that are similar to the target group regarding the characteristics that modify drug response, in the setting where they would be treated in real life and the use of comparators and outcome measures that are relevant for treating patients in routine clinical practice. The choices to be made in pragmatic trial design pertain to four domains: the study population; the setting of the trial; operationalization of the intervention and chosen comparator; and the outcome measure. General issues of data management and monitoring need to be taken into account because of possible additional influences on routine clinical practice.
Because the term “pragmatic trial” is commonly used for trials that not only asses the difference between treatment strategies, including extraneous factors, but also aim to maximize generalizability to a broader setting or patient population (Fig. 1) [
], in the rest of this paper and the following series, the term “pragmatic trial” will be used as such.
7. From the drawing board into the practice of conducting the trial
Pragmatic, in our view, is not synonym to “easy to conduct” or “sloppy.” Depending on several factors, including the stage of drug development and the type of treatment, pragmatic trails can be designed to be point-of-care or large simple trials and thus relatively easy to conduct [
The opportunities and challenges of pragmatic point-of-care randomised trials using routinely collected electronic records: evaluations of two exemplar trials.
The opportunities and challenges of pragmatic point-of-care randomised trials using routinely collected electronic records: evaluations of two exemplar trials.
The opportunities and challenges of pragmatic point-of-care randomised trials using routinely collected electronic records: evaluations of two exemplar trials.
]. To increase generalizability, for example, the real-world prescribers of a drug could be involved in a pragmatic trial. The management of the trial may then be moved from specialized trial centers, experienced in explanatory trials, to the medical department or primary care setting. Yet, pragmatic trials have shown to be prone to selective and low participation rates in a primary care setting [
The opportunities and challenges of pragmatic point-of-care randomised trials using routinely collected electronic records: evaluations of two exemplar trials.
] as these sites may not be fully equipped or dedicated to support a clinical trial.
“Usual care” is the preferred comparator in pragmatic trials. The operationalization, however, can be difficult because the usual care might not reflect the standard of care. Usual care may consist of a range of treatments, which may also vary substantially across medical centers.
Monitoring and reporting of safety events as demanded by ethical guidelines and regulatory bodies may interfere with routine clinical practice [
]. These challenges may hamper the feasibility of the trial and generalizability of the trial results.
On the bright side, taking advantage of the ubiquitous availability of electronic health record systems and linking routine health care systems directly might facilitate “real-world” research including pragmatic trials. To manage technical hurdles, infrastructure for this is being developed for instance within the TransForm project [
For most new market-approved treatments, the clinical evidence is not sufficient to guide physicians and policy makers in choosing the optimal treatment for their patients. If designed and executed well, pragmatic trials provide real-world evidence on the value of a treatment strategy in routine clinical practice while maintaining the strength of RCTs. Pragmatic trials may provide the opportunity to generate evidence in an earlier stage than in the current development paradigm where real-world evidence generally is only collected postlaunch. However, careful consideration must be given to the design of pragmatic trials such as the study setting and population, mode of intervention, comparator regimes, and outcome measures to warrant appropriate generalizations to the target patient group in clinical practice while ensuring operational feasibility.
To gain the benefit of the full potential of a pragmatic trial, there is a need for guidance and tools in designing the trial while ensuring operational feasibility [
]. The consequences of design choices should be clarified, including the possible impact on feasibility, ethical acceptability, stakeholder preferences, validity, precision, and generalizability (see Box 1). In this way, balanced design choices can be made with an optimal chance of success in practice.
This is the first paper, introducing the pragmatic trial, in a series of eight papers in this journal (see Box 2). The following papers will address the four domains in which choices need to be made in pragmatic trial design, as well as the general issues of data management, monitoring, and informed consent. Design choices, their specific implications and operational challenges, stakeholder preferences, as well as generalizability, validity, and precision of trial results will be discussed. The findings in this series are based on extensive literature review, enhanced with in-depth stakeholder interviews and the expertise and experience of work package 3 of the IMI GetReal consortium (see Box 3)
Validity: If the result of a comparison is true and not systematically (nonrandom) overestimates or underestimates the effects of the treatment, such a result is valid [
]. Research results that are not valid are not useable whatever the other qualities of the research are. Therefore, assurance of validity of the result of a study, through the absence of bias, in drug research is first priority. Randomization in trials provides an important means to assure that a measurement of benefit or risk between two or multiple treatment groups is not confounded by incomparability of prognosis at baseline due to differential prescribing.
Precision: The precision of an estimate of a treatment effect from a study is reflected in the confidence interval (CI) of the effect estimates, which denotes the probabilistic boundaries for the true effect of a treatment. That is, if a study was repeated again and again, the 95% CI would contain the true effect in 95% of the repetitions. The smaller the CI, the higher the precision [
]. Precision is predominantly determined by the magnitude of random error in the effect estimates and the sample size of the study.
Generalizability: The process of applying findings in a particular study to a population of patients in a particular clinical setting is called generalization and the extent to which the results of a study apply to that population is called generalizability [
]. Sometimes the term external validity is used for generalizability, but this should be discouraged because generalizability is not about the truth (validity). Validity and generalizability need separate consideration. Findings may be perfectly valid but not applicable to another group of patients and thus not generalizable. First, validity needs to be assured; next, the trial findings should be generalizable to the population of interest.
Pragmatic trials aim to generate real-world evidence on the (relative) effects of treatments, generalizable to routine practice. In this series, we will discuss options and choices for pragmatic trial design, operational consequences, and the interpretation of results.
1. Introduction: Pragmatic trials and real-world evidence
2. Selection and inclusion of usual care sites
3. Participant eligibility, recruitment, and retention
4. Challenges of informed consent
5. Questions, comparators, and treatment strategies
Launched in October 2013, GetReal is a 3-year project of the Innovative Medicines Initiative (IMI), a EU public-private consortium consisting of pharmaceutical companies, academia, HTA agencies and regulators (e.g., NICE, HAS, EMA, and ZIN), patient organizations, and SMEs.
GetReal aims to show how robust new methods of RWE collection and synthesis could be adopted earlier in pharmaceutical R&D and the healthcare decision-making process.
The research leading to these results was conducted as part of the GetReal consortium and included literature review and interviews with stakeholders from academia, research institutions, contract research organizations, pharmaceutical industry, regulatory authorities, health care insurers, health technology assessment (HTA) agencies, general practitioners, and patient organizations. For further information, please refer to http://www.imi-getreal.eu/.
References
Naci H.
O'Connor A.B.
Assessing comparative effectiveness of new drugs before approval using prospective network meta-analyses.
Updated glossary of definitions of common terms.2015; (Available at https://www.imi-getreal.eu/Portals/1/Documents/Publications/D1.3%20GetReal%20Glossary%20-%20Update.pdf. Accessed August 9, 2016)
Assessing the comparative effectiveness of long-acting injectable vs. oral antipsychotic medications in the prevention of relapse provides a case study in comparative effectiveness research in psychiatry.
The Pragmatic-Explanatory Continuum Indicator Summary (PRECIS) instrument was useful for refining a randomized trial design: experiences from an investigative team.
How pragmatic or explanatory is the randomized, controlled trial? The application and enhancement of the PRECIS tool to the evaluation of a smoking cessation trial.
The opportunities and challenges of pragmatic point-of-care randomised trials using routinely collected electronic records: evaluations of two exemplar trials.
The prospect of early pharmaceutical pragmatic clinical trials: a background paper.
Center for Medical Technology Policy,
2013 (Available at: http://www.cmtpnet.org/docs/resources/Segal-PCT-Background-Paper.pdf. Accessed August 9, 2016)
Funding: The work leading to these results has received support from the Innovative Medicines Initiative Joint Undertaking under grant agreement no (115546), resources of which are composed of financial contribution from the European Union's Seventh Framework Programme (FP7/2007/2013) and EFPIA companies' in kind contribution.
30540-1&id=gr1.jpg){kind=link}