A new preference-based analysis for randomized trials can estimate treatment acceptability and effect in compliant patients

Article Outline

• Abstract
• 1. Introduction
• 2. Methods
  • 2.1. Estimation of patient preference distribution
  • 2.2. Estimation of outcome rates by patient preference groups
    • 2.2.1. Constraints I: constant relative risk (RR)
    • 2.2.2. Constraints II: independent preference effects
  • 2.3. Example 2
  • 2.4. Example 3: a lung cancer trial
  • 2.5. Preference-based analysis with only two levels of treatment
  • 2.6. Example 4: a clinical trial of treatment for carotid stenosis
• 3. Discussion
  • 3.1. Other preference patterns
  • 3.2. Other study designs
  • 3.3. Interpretation of preference-based analysis results
• Acknowledgments
• References
• Copyright

Abstract 

Backbround and Objectives

Development of a new method of analysis to evaluate the acceptability of (or preferences for) the treatments in a randomized trial, and the benefit of treatment among compliers.

Materials and Methods

We characterize trial participants through the groups who would: accept either treatment if offered (compliers); refuse one treatment but accept the other if it is offered to them (two groups of preferers); or prefer one treatment and insist on it if it is not offered to them initially (two groups of insisters).

Results

We show that in our framework, one can always estimate the proportions of patients in these five preference groups. However, constraints are required to estimate the corresponding outcome rates, and thus estimate the treatment effect in the compliers. We propose two possible sets of constraints and illustrate them by numerical examples.

Conclusions

The traditional intention-to-treat analysis avoids biases associated with the alternative per-protocol or as-treated approaches, but it provides imperfect information about the expected treatment effect among patients who are committed to taking the treatment. Many physicians and patients want to know the expected benefit if they adhere to the therapy. Our preference-based analysis provides an estimate of treatment benefit among such patients.

Keywords: Compliance, Efficacy, Randomized trials, Treatment preference

Back to Article Outline

1. Introduction 

There are several traditional approaches to the estimation of treatment effects in a clinical trial. The intention-to-treat (ITT) analysis retains all patients who have been randomized, and leaves them in the treatment group to which they were randomly assigned, regardless of whether they actually complied with their assigned treatment, with the alternative treatment being evaluated in the trial, or indeed with any other available treatment (Fig. 1). Because of its explicit recognition of the randomization, the ITT analysis provides an unbiased estimate of treatment effect in the entire study population. In particular, it is not subject to potential biasing effects of noncompliance, treatment crossovers, or potential confounders. The ITT approach is appropriate if a “pragmatic” analysis is intended, for the evaluation of treatment policies [1], but it likely underestimates the true treatment effect that compliant patients might expect.

• 
  • View Large Image
  • Download to PowerPoint

• Fig. 1. 

  Possible analyses of an randomized control trial: intention to treat (ITT).

The alternative per-protocol analysis includes only those patients who comply with their assigned treatment. It excludes protocol violators; that is, patients who cross over to the alternative treatment, or who take neither study treatment (Fig. 2). This analysis does not respect the randomization, and is therefore subject to possible selection bias, through potential prognostic differences between patients who do or do not comply with their randomized assignment [2]. Indeed, per-protocol analyses may be viewed as an “improper subgroup” analysis [3], which should be interpreted cautiously in most contexts.

• 
  • View Large Image
  • Download to PowerPoint

• Fig. 2. 

  Possible analyses of an randomized control trial: per protocol.

Another alternative is the on-treatment or as-treated analysis (Fig. 3). Here, patients are included in the group corresponding to the treatment they actually received, regardless of their randomized assignment. This approach is also potentially subject to selection bias [4]. In general, the bias in the estimated treatment effect can be in either direction. Overestimates of treatment effect result, for instance, if patients with good prognosis align with the more effective therapies. On the other hand, if patients with poor prognosis tend to override assignment to an inferior control arm and elect instead to take a promising experimental treatment, conservative estimates of treatment effect result.

• 
  • View Large Image
  • Download to PowerPoint

• Fig. 3. 

  Possible analyses of an randomized control trial: as treated.

Clinicians treat individual patients, many of whom are committed to, and destined to take treatment as prescribed. Such patients are interested in the benefit expected for people like themselves (who will comply), rather than the benefit in a mixed group of individuals, some of whom comply and some of whom do not. The magnitude of benefit that the compliers can expect may be substantially different from the magnitude reported from following the ITT principle—and the greater the number of noncompliers, the greater the potential difference [2], [4]. The data from research studies often suggest that treatment benefit effect varies (between studies, or over patient subgroups) [5], and such variation might well be due to different levels of compliance. A recent meta-analysis of empirical research showed an overall 26% difference in outcome rates between high and low compliance patients [6]. An estimate of treatment benefit that is uncontaminated by compliance issues would help physicians and patients interpret study results and predict what they might expect to experience if they adhere to the therapy.

The focus in this article will be on studies comparing two treatments (A and B). Typically, A and B will both be active interventions, possibly including “usual care.” We will primarily discuss situations where A and B are identifiable to the participants, so that they may express a preference for one or the other treatment being compared. Our framework will also incorporate the possibility that it is the clinicians who may express a preference for A or B, for example, in surgical trials in which the surgeon decides to override the randomized assignment because of clinical considerations encountered during the surgery. Because the treatments are identifiable, the study is therefore unblinded in this respect, but blinding of the data collection, outcome event adjudication, and data analysis remains possible.

Once patients have been randomized, we envisage three main scenarios for their subsequent treatment experience. First, they may accept the treatment that was assigned at random. Second, they may refuse the assigned treatment, and cross over to the other study treatment, either by insisting on it, or through some other route (e.g., if a patient decides not to accept an experimental treatment, and hence defaults to the other “usual care” treatment). Finally, they may refuse both of the treatments A and B in the study, and obtain alternative treatment elsewhere, or remain untreated. We will refer to the three possible treatments as A, B, or 0. We will think of the 0 group as receiving “nothing,” although in general they may receive some active treatment other than A or B. We will later discuss the simpler situation of only two treatment levels, typically A vs. B, or A vs. 0.

We propose a new “preference-based analysis” based on characterizing patients in terms of their preference for either of the treatments being compared in the study. Its first objective is to estimate the proportions of patients in various preference groups. So, for instance, we will estimate the proportion of patients who would comply by accepting either treatment, and the proportion who might prefer to receive treatment A and would refuse B. The second objective is to estimate outcome rates in the various preference groups, with particular interest in the treatment benefit among compliant patients. To the extent that patients are interested in benefits and downsides of interventions that they intend to use, one may consider these estimates as most relevant in the context of clinical decision making. We will address these goals in the context of unblinded studies, as outlined above, but we will also consider possible extensions to other types of study. Two hypothetical and two real-life examples will illustrate the concepts of preference-based analysis.

Back to Article Outline

2. Methods 

Our framework involves the characterization of preference-based groups in the patient population, according to their patterns of potential acceptance or refusal to each of the two treatments being compared in the study. The five preference groups we consider are shown in Table 1. We have particular interest in group 1, whom we designate as compliers, and who would accept either treatment if offered. The term compliance has been widely used to describe this group, although alternative terms such as cooperation, adherence, or concordance have been used by other authors [5], [6], [7], [8]. We also incorporate two groups of preferers (groups 2 and 3), who would accept their preferred treatment if it is offered, but decline treatment altogether (and hence join treatment group 0) if offered the other treatment. So, for example, we designate the patients who prefer A, and who would decline B if offered, to be A-preferers. Finally there are two groups of insisters (groups 4 and 5) who, like the preferers, accept their preferred treatment if it is offered, but if it is not, they insist on their preference, or are able to receive it through other means, as described earlier. (We might also think of the insisters as changers or crossovers, to reflect the notion that they may not actually insist on their preference, but might receive it for some other reason, such as by default in the case of usual care, or because of a clinical decision.)

Table 1. Framework of five preference groups of patients in a randomized trial comparing treatments A and B

The existence of the preferer and insister groups is an unfortunate reality of conducting clinical trials. Although a good study design might include procedures to reduce the numbers of such patients (i.e., with a detailed informed consent process), there will inevitably be some patients who either change their mind about accepting the whole idea of being randomized, or who enter the trial with a secret preference for one treatment, and hope to be “lucky” in their random assignment—but if they are not “lucky” they will not accept their assigned treatment. The greater the extent to which the number of these individuals can be limited, the closer the treatment benefit from an analysis using the ITT principle will be to the benefit in compliant patients.

Our framework allows for asymmetry in the preference patterns (e.g., some patients may prefer A and decline B if offered, while others may insist on B even if offered A). Not all of the five preference groups shown in Table 1 need actually occur in a given study, and indeed, some may be precluded by the study design (e.g., one treatment may be accessible only to study participants, and no crossovers may be allowed). For completeness, we will mention the possibility of other preference patterns, such as refusers, individuals who would refuse whichever treatment is randomly assigned to them. We will return to the implications of recognizing these additional preference groups in the Discussion section.

In developing our framework, we will make a number of simplifying assumptions. First, we will assume that compliance with a given treatment can be summarized with a binary variable, that is, patients either satisfy some requirements to be classified as compliant, or they are classified as noncompliant. This assumption is, for example, very suitable in studies involving two types of surgery, two one-time drug therapies (e.g., thrombolytic drugs), or two alternative diagnostic strategies. We will not cover situations where participants drop out or crossover because of side effects or other early experience on the assigned therapy, and hence exhibit partial compliance; where participants temporarily suspend their assigned treatment; or where they cross over to the other treatment and later return to the assigned treatment. We will also assume that a binary outcome is observed in all patients (i.e., there are no losses to follow-up or other missing data). Without loss of generality, we will presume the outcome event to be deleterious (e.g., death), so that treatments with lower event rates are preferred. Finally, we will assume there are no covariates available to predict compliance—although generalizations of our framework could allow for this possibility.

Figure 4 shows how the five preference groups in Table 1 are distributed among the six observable outcome groups, according to their randomized treatment and their actual treatment received. For example, the set of patients who were assigned A and actually received A is made up of some mixture of compliers, A-preferers and A-insisters, and there is a complementary set of patients who were assigned and received treatment B. Similarly, the patients who were assigned A but actually received B are B-insisters, while the patients who were assigned A but took neither A nor B are A-preferers. Note that some of the preference groups cannot be observed directly, because we do not necessarily know what would have happened in the counterfactual situation where the opposite treatment would have been offered.

• 
  • View Large Image
  • Download to PowerPoint

• Fig. 4. 

  Possible analyses of an randomized control trial: by patient preference groups.

The six observable outcome groups shown in Figure 4 can be described in terms of the numbers of patients they contain, and the number of outcome events they experience (and hence, their outcome rates can be derived). We will use these observations first to estimate the distribution of patients over preference categories, and second, to estimate the corresponding outcome rates. Because of the possibility of selection bias associated with patient preferences, we must in general allow for potentially different outcome rates in all five preference groups.

2.1. Estimation of patient preference distribution 

Table 2a shows data from hypothetical Example 1, in which 500 study patients have been randomly assigned to each of treatments A and B. Within each of the study arms, there is a set of three observed patient frequencies, according to whether they accepted the assigned treatment, took the opposite treatment, or took neither. Conditioning on their total sample sizes, there are two independent proportions that can be estimated from each study arm, implying that there are 4 df (degrees of freedom) altogether. Hence, we can estimate up to four independent proportions in the preference distribution. This is sufficient to support our framework, which has five preference groups, and hence four independent proportions that characterize the preference distribution.

Table 2. Example 1

Subscripts refer to preference groups shown in Table 1.

We denote α_A, α_B, and α₀ to be the observed proportions receiving A, B, or 0 in study arm A, and β_A, β_B, and β₀ to be the observed proportions receiving A, B, or 0 in study arm B. Note that α_A + α_B + α₀ = β_A + β_B + β₀ = 1. We also denote the proportions of patients in the preference groups shown in Table 1 by p₁, p₂, …p₅. By equating the observed proportions with their mixture expectations as implied by Figure 4, we can estimate the ps as follows:

p₂ = β₀ (A-preferers),p₃ = α₀ (B-preferers),

p₄ = β_A (A-insisters),p₅ = α_B (B-insisters)

and hence,

(1)

The numerical solution to Example 1 is shown in Table 2b. We find that 430/500 (86%) of patients accepted their assigned treatment in the A-arm, while 400/500 (80%) accepted their assigned treatment in the B-arm. From equation (1), we estimate that p₁ = 66% of patients would have accepted either treatment, these being the compliers. Treatment A was preferred a little more often than B (there were an estimated p₂ = 10% of A-preferers vs. p₃ = 8% B-preferers, and p₄ = 10% of A-insisters vs. p₅ = 6% B-insisters).

From equation (1) we can deduce that the estimated proportion of compliers (p₁) will be positive as long as α_A + β_B > 1, which implies that (overall) at least 50% of patients accept their assigned treatment. (By extension, a similar condition can be shown to pertain to the situation where unequal numbers of patients have been randomized to A or B.) Now we expect 50% acceptance in preferers and insisters, because the randomization dictates that an expected 50% of these groups will indeed be offered their preferred choice. (Again, a generalization of our method can cover the possibility of randomizing other than 50:50.) Therefore, we expect at least 50% acceptance in the entire study (including the compliers), because, by definition, the compliers always accept their assigned treatment. However, the empirical estimate of p₁ could be negative by chance in small studies, if it happens that α_A + β_B < 1.

2.2. Estimation of outcome rates by patient preference groups 

Having estimated the proportions of patients in the various preference groups, we now turn to the second goal, of estimating the corresponding outcome rates for those groups. The parameters of interest are shown in Table 3. Here, r denotes an outcome rate among patients actually receiving A, s denotes a rate among patients actually receiving B, and t denotes a rate among patients actually receiving 0. The subscripts refer to the preference groups defined previously.

Table 3. Notation for outcome rates by preference group

r denotes a rate among patients actually receiving A.

s denotes a rate among patients actually receiving B.

t denotes a rate among patients actually receiving 0.

Subscripts refer to preference group shown in Table 1.

As noted earlier, one cannot directly observe all of the outcome rate parameters above. However, outcome rates are observable in three groups per study arm, according to the actual treatment received. Our notation for the observable outcome rates is shown in Table 4, where θ denotes a rate among patients assigned to A, ϕ denotes a rate among patients assigned to B, and the subscript refers to the actual treatment received.

Table 4. Notation for outcome rates by assigned and actual treatments

θ denotes a rate among patients assigned to A.

ϕ denotes a rate among patients assigned to B.

Subscript denotes actual treatment.

In summary, Table 3, Table 4 indicate that there are eight underlying rate parameters, and observable outcome rates in six groups. Hence, in general, the full set of eight parameters cannot all be estimated simultaneously, and a minimum of two constraints is required. There are many possible constraints that will render the outcome rate parameters estimable, but we now propose two alternative pairs of constraints that we feel may be reasonable to adopt in some practical situations.

Under this first pair of constraints, we assume that the RR is the same in compliers, preferers and insisters. Specifically we assume that

(2)

These assumptions imply the same benefit (in RR terms) among compliers (who accept either treatment, and for whom we compare outcomes in those actually getting A or B), preferers (for whom we compare outcomes in those who actually accept A but would have declined B vs. those who actually accept B but would have declined A) and insisters (for whom we compare outcomes on their actual treatments A or B). Equating the RR in these three preference groups implies two independent constraints on the outcome rate parameters.

To estimate the rate parameters (r₁, r₂ , r₄ , s₁ , s₃ , s₅ , t₂, and t₃), we equate the observed rates (θ_A , θ_B , θ₀ , ϕ_A , ϕ_B , and ϕ₀) with the corresponding appropriately weighted mixtures of rate parameters, with the weights being obtained from the previously estimated preference proportions p₁, p₂, … p₅ , and applying constraints (2). Four of the observed outcome groups (those with patients not receiving their assigned treatment) consist of a single preference group (see Fig. 4), and so their rate parameters can be estimated directly as

(3)

The other two observed outcome groups (those receiving their assigned treatment) are made up of mixtures of three preference groups. After solving, we may obtain

(4)

and

(5)

and finally, r₁ and r₂ can be obtained from the previous parameters and the constraints. Specifically, r₁ = RR s₁ and r₂ = RR s₃ , where RR can be estimated, for example, from r₄/s₅.

We now illustrate this method by extending Example 1 to include outcome data, as shown in Table 5a. The treatment effect using the ITT analysis is RR = [265/500]/[405/500] = 0.654. Using the per-protocol analysis it is RR = [215/430]/[340/400] = 0.588, while using the as-treated approach it is RR = [235/480]/[360/430] = 0.585. The corresponding risk difference (RD) results are 0.280, 0.350, and 0.348 (see Table 5b).

Table 5. Example 1 (continues example begun in Table 2)

Abbreviations: RD, risk difference; RR, relative risk.

Solving equations (3), (4), (5) yields the estimated outcome rates by preference group as follows:

t₂ = 0.90 (A-preferers offered B);t₃ = 0.75 (B-preferers offered A);

r₄ = 0.40 (A-insisters);s₅ = 0.67 (B-insisters);

r₂ = 0.30 (A preferers offered A);s₃ = 0.50 (B-preferers offered B);

andr₁ = 0.55 (compliers offered A);s₁ = 0.91 (compliers offered B)

The last two results imply an RR among the compliers of 0.55/0.91 = 0.600, or an RD of 0.364. (An Excel spreadsheet for the calculation of all results in the preference-based analysis, as well as the traditional analyses, is available on request.)

In addition to the estimate of treatment effect among compliers, the preference-based analysis permits inferences about the other patient preference groups. For instance, in this example we see that the preferers who are not offered their treatment of choice (and who are therefore in the untreated group 0) have poorer outcomes than preferers who do receive their treatment of choice. Also, insisters have better outcomes than compliers, on either treatment.

We now adopt an alternative pair of constraints, by assuming that among those who are compliant or prefer A, the outcome rates for patients actually on A are independent of their preference for B, and similarly that among those who are compliant or prefer B, outcome rates for patients actually on B are independent of their preference for A. We refer to this as an assumption of independent preference effects, reflecting that, for instance, expected outcomes are the same for compliers and A-preferers who actually receive A, and in particular do not depend on their counterfactual behavior if they had been offered B. In terms of the rate parameters, these constraints imply that

(6)

Adopting a similar approach to previously, by equating the observed outcome rates with appropriately weighted mixtures of the constrained rate parameters, on this occasion we obtain

and

with the remaining parameters being implied by constraint (6). Applying these equations to the Example 1 data gives t₂ = 0.90, t₃ = 0.75, r₄ = 0.40, s₅ = 0.67, r₂ = 0.51, s₃ = 0.86, r₁ = 0.51, and s₁ = 0.86. The last two results correspond to RR = 0.593 and RD = 0.352 among the compliers.

In this example, there was a similar estimated treatment effect found in compliers with either of our two sets of assumed constraints (see Table 5b). The preference-based analysis showed a slightly stronger RR treatment effect for the compliers than in the ITT analysis, but a slightly weaker effect than in the per-protocol or the as-treated analyses. RD was essentially constant in all analyses except the ITT.

2.3. Example 2 

The results in Example 1 were encouraging, in the sense that we obtained qualitatively similar results with either set of constraints I or II, and adopting either a relative or absolute effect measure (RR or RD). To illustrate that this is not always the case, we now consider Example 2, whose data are shown in Table 6a and analyses in Table 6b and 6c.

Table 6. Example 2

Here 400/500 (80%) accepted their assigned treatment in the A-arm of the study, and 200/500 (40%) accepted their assigned treatment in the B-arm, but only an estimated p₁ = 20% would have accepted either treatment. Treatment A was preferred much more often than B (p₂ = 40% vs. p₃ = 14% preferers, and p₄ = 20% vs. p₅ = 6% insisters).

Estimating the outcome rate parameters assuming constraints I (constant RR) gives: t₂ = 0.88, t₃ = 0.71, r₄ = 0.50, s₅ = 0.67, r₂ = 0.72, s₃ = 0.96, r₁ = 0.36, and s₁ = 0.48, with RR = 0.750 and RD = 0.119 among the compliers. If we instead adopt constraints II (independent preference effects), we obtain r₂ = 0.60 (A-preferers offered A), s₃ = 0.68 (B-preferers offered B), the same outcome rates r₁ = 0.60 and s₁ = 0.68 as before among compliers, and the other parameters the same as under constraints I. Constraints II lead to RR = 0.887 and RD = 0.076 among the compliers.

In contrast to Example 1, here there are qualitatively different estimated treatment effects in compliers, depending on which set of assumptions is adopted. Also, the preference-based analysis showed either a stronger or a weaker treatment effect than the ITT, per-protocol or as-treated analyses (see results in Table 6c), depending on the assumptions made.

2.4. Example 3: a lung cancer trial 

To further illustrate these ideas with real data, we now consider a randomized trial of two methods of investigating the possibility of mediastinal disease among patients with apparently operable lung cancer [9]. A total of 685 patients were assigned to computed tomography (CT) or mediastinoscopy as the initial diagnostic procedure. The study objective was to determine which procedure had lower rates of thoracotomy in patients who were not subsequently cured of the disease, or in patients with benign disease. This outcome is referred to as the total number of "unnecessary” thoracotomies.

Some patients did not receive their assigned intervention, for reasons related to decisions made by the patients or their clinicians, or through protocol violations. Of the 343 patients who were randomized to CT, 338 actually were scanned. Among the five patients who did not receive a CT scan were: two patients shown to have benign disease, one who refused investigation, and two who had mediastinoscopy because of protocol violations.

Of the 342 patients randomized to mediastinoscopy, 315 actually underwent the procedure. Among the 27 others, in 11 patients the lesion disappeared or proved benign; in 7, other investigations revealed metastatic disease; 4 had medical contraindications; 4 went straight to thoracotomy because of protocol violations; and 1 patient died.

Outcome data for this study are shown in Table 7a, together with a summary of the preference-based and traditional analyses in Table 7b and 7c. Overall adherence to the randomized assignment was 338/343 (98.5%) for the CT group, and 315/342 (92.1%) for the mediastinoscopy group. The preference-based analysis indicates that 90.6% were “compliant,” which in this context means that they would have adhered to either randomized assignment, taking all protocol violations, clinical decision making, mortality, and personal choice into account. In the same spirit, there were an estimated 5.8% of patients who preferred CT, and 2.0% of CT-insisters, but only very small percentages of preferers and insisters for mediastinoscopy. Although the data are limited, there is a suggestion that patients who deviated from their randomized interventions had somewhat better outcomes. This is partly due to the identification of benign disease in some patients after randomization had occurred, but before the assigned investigative procedures were carried out.

Table 7. Example 3: Thoracotomy rates in a randomized trial of investigation for mediastinal disease in lung cancer patients

Abbreviations: CT, computed tomography; Med, mediastinoscopy.

Table 7c shows the estimated treatment effect (CT vs. mediastinoscopy) by various methods of analysis. Because of low thoracotomy outcome rates in patient subgroups not receiving their assigned intervention, the preference-based analysis can be unstable, and indeed can either be degenerate or predict inadmissible outcome rate parameters (above 1 or below 0) under the constant RR assumption. The particular solutions shown in Table 7c are obtained after adding 0.5 to the numerators and denominators of the two subgroups where no thoracotomies occurred. The resulting preference-based estimates of RR and RD were close to those from the per-protocol and as-treated analyses, but slightly stronger than those from the ITT analysis. In this example, all the results are reasonably consistent, mainly because of the relatively small number of patients who did not receive their randomized intervention.

2.5. Preference-based analysis with only two levels of treatment 

An important special case of the methods in our framework pertains to the situation where preference groups 2 and 3 in Table 1 (patients who prefer one treatment, and will refuse the other if offered) are absent. We will thus consider the design where patients either accept their randomly assigned treatment, or insist on or cross to the opposite treatment, but the option of refusing treatment altogether is eliminated. For example, patients receive either surgery (intervention A) or medical care (intervention B), but never opt completely out of care. Another instance of this special case is for preventive interventions, such as cancer screening; here, participants may be randomized to be offered screening, but the control group is offered nothing. Noncompliance with the intervention (failure of invitees to come for screening) or with the control protocol (control participants may be screened anyway) can both occur, but there are still only two possible levels of treatment (being screened or not).

In this circumstance, the parameters associated with preference groups 2 and 3 are not required. The restricted set of three preference proportions to be estimated is (p₁, p₄, and p₅), and because they sum to 1, there are now only two independent parameters involved. There are 2 df available, one from each arm of the study, as in Table 2 after elimination of the third row (patients who receive no treatment). The four outcome rate parameters are now r₁, r₄, s₁, and s₅, which are estimable without further assumptions from the four observable groups defined by assigned and actual treatments (as in a reduced version of Table 4, eliminating the last row).

The solutions for the three preference proportions are the same as the corresponding components of equation (1). The solutions for r₄ and s₅ are the same as in equation (3), that is,

Last, after equating the observed rates and their corresponding weighted averages of the component rate parameters, r₁ and s₁ can be obtained as

(7)

and

(8)

2.6. Example 4: a clinical trial of treatment for carotid stenosis 

We will use data from the NASCET trial of high-grade carotid stenosis, in which patients were randomized to carotid endarterectomy or medical care [10]. The primary outcome was the occurrence of any fatal or nonfatal ipsilateral stroke. A summary of the relevant data is shown in Table 8a. It is not possible to uniquely identify the number of outcome events in the crossover patients from the NASCET report, because its analysis retains the correctly randomized patients in their assigned groups, and counts outcome events up to the time of crossover (if they occur). However, the reported ITT analysis also includes some additional eligible patients who were incorrectly randomized.

Table 8. Example 4: Stroke^a rates in the NASCET study^b of carotid stenosis, by assigned and actual treatments

In the group randomized to medical care, 15/331 (4.5%) actually had surgery for reasons such as the patient refusing the randomization, or the randomization being overridden by an attending clinician. There were six additional patients who had surgery following a stroke. Because nearly all strokes after randomization were ipsilateral, we will assume that these six strokes were of this type, and should therefore be attributed as events in the medical care group. In the group randomized to surgery, one patient out of 328 rejected the randomized assignment and had medical care instead. The data in Table 8a are given as a likely close approximation to the data required for a preference-based analysis.

Using the data from Table 8a gives estimates p₅ = 15/331 = 0.045 (the proportion of surgery-insisters, the patients who crossed over from medical care to surgery, because of personal preference or clinical reasons) and p₄ = 1/328 = 0.003 (the proportion of medical-insisters, who crossed over from surgery to medical care). These lead to p₁ = 1 − 0.045 − 0.003 = 0.952 as the estimated proportion of compliant patients who would accept either treatment, and who also would have no overriding clinical objection to the randomized assignment (Table 8b).

Denoting medical care by M and surgery by S, the observable outcome rates from Table 8a are θ_M = 19.3%, ϕ_S = 8.0%, θ_S = 6.7% and ϕ_M = 100%. Substituting these and the estimated preference proportions into equations (7), (8) gives preference-based estimates of the outcome rates in the compliers as r₁ = 19.0% and s₁ = 8.0%, with corresponding values of RR = 0.421 and RD = 0.110. These estimated treatment effects are equivalent to those from the model proposed by Cuzick et al. [11], although these authors do not discuss estimation of the preference distribution. The traditional ITT analysis gives outcome rates for the entire medical and surgery groups as 18.7% and 8.2%, respectively, with RR = 0.439 and RD = 0.105. The per-protocol analysis gives outcome rates 61/316 (19.3%) and 26/327 (8.0%), with RR = 0.412 and RD = 0.114. Finally, the as-treated analysis gives outcome rates 62/317 (19.6%) and 27/342 (7.9%), with RR = 0.404 and RD = 0.117 (see Table 8c for a summary of these results).

In this example, all four types of analysis gave qualitatively similar results. This is primarily because of the relatively low crossover rates (especially from surgery to medical care), and the lack of any major difference in the outcome rates among the crossovers (e.g., 6.7% for those crossing into the surgery group, vs. 8.0% for patients randomized to surgery and actually having surgery), at least to the extent that the limited numbers of crossovers can inform this comparison. However, the results are ordered in the direction we would often anticipate: the compliant patient RR is intermediate between the ITT and either the per protocol or as-treated.

Back to Article Outline

3. Discussion 

We now consider limitations and some possible extensions of our analytic framework.

3.1. Other preference patterns 

Our preference-based framework included compliers, preferers, and insisters. In fact, there are other theoretical possibilities, and a full set of preference patterns would also include people who would refuse both treatments A and B (refusers); people who always insist on the opposite treatment to that offered (defiers); and people who would decline A if offered, but insist on A if offered B, and vice versa (two groups of contrarians). Sometimes the protocol will prevent the occurrence of some of these groups—for instance, if crossover from usual care to an experimental intervention is not permitted. However, to the extent that these groups do occur in a given study, they will imply extra parameters to be estimated, both in the preference distribution and for the outcome rates. Because all the available degrees of freedom have already been accounted for in our framework with three levels of treatment (A, B, or 0), one would need to somehow estimate these additional parameters by other means.

Refusers could occur in studies where there is a time interval between randomization and the clinical intervention, and when some people change their minds and decide not to participate in the study. Appropriate management of the randomization process could limit the size of this group. If necessary, the numbers of such individuals might be ascertained by retrospectively questioning patients about their reasons for treatment refusal, and their outcome rates can be obtained by follow-up. In fact, all patients should be followed anyway, if the investigators plan a conventional ITT analysis. The defiers and contrarians are somewhat “illogical” in their behavior, and such patients will occur rarely in practice. Other authors have generally dismissed these groups, or not allowed them to exist in their analytic framework [11], [12].

Our estimates of treatment effect in studies with only two levels of treatment (A vs. B, or A vs. 0) are equivalent to a model proposed by Cuzick et al. [11], although these authors do not discuss the preference distribution. Cuzick et al. show that in this more restricted situation, if the baseline outcome rates are the same for patients who refuse the assigned experimental treatment and for patients in the control arm who adopt the treatment, the estimated treatment effect is necessarily larger than the ITT estimate, although its confidence interval is wider. Their model assumes a constant RR effect of treatment in the patients who were randomized to treatment or who were randomized to control but actually received the active treatment. The model can incorporate defiers, although this requires an additional assumption that the same treatment effect also pertains to them. Greenland et al. [7] adopt an instrumental variable approach for the comparison of a single treatment to control, an idea that has been applied to survival data, although this was achieved by ignoring the possibility of insisters or defiers of the experimental treatment [12], [13].

3.2. Other study designs 

In our development of preference-based analysis, we have assumed that there are two active and identifiable treatments. We also allowed for possible crossovers to the opposite treatment, and for complete refusal of either treatment, in both study arms. Other scenarios might include elimination by protocol of one or both of the possibilities for treatment crossover, or blinding of treatment assignment. Each of these would affect the set of parameters of interest, and the feasibility of their estimation, as we now discuss.

The protocol may preclude one or both types of crossover. For instance, if treatment A is an expensive experimental treatment, access to it may be limited to study participants, and crossover from the control treatment B to A might be proscribed. In our framework, this restriction would eliminate the group of A-insisters, and their associated parameters p₄ and r₄. We would then be left with four preference proportions and seven outcome rate parameters to be estimated from the five remaining observable outcome groups. Similarly, if both types of crossover are prevented by the protocol, both insister groups are eliminated, leaving three preference proportions and six outcome rates to be estimated from four observable outcome groups. Thus, in both situations the preference distribution remains estimable, but we still require two assumed constraints on the outcome rates.

In the case of blinded studies, we can observe only four outcome groups, namely the acceptors and refusers of each arm of the study. Patients can no longer (rationally) express their treatment preference, or insist on one or other treatment, because of the fact of their blindedness. Accordingly, one is limited here to the traditional analyses (ITT, per-protocol, or as-treated), and a formal preference-based analysis is not possible. The per-protocol analysis supplies the closest analogy to the preference-based analysis, but unfortunately one cannot distinguish the various component groups among treatment acceptors. Thus, for instance, the subgroup of patients who accept their assigned treatment A is made up of some unknown mix of compliers, potential A- and B-preferers, and potential A- and B-insisters, with the now-blinded preferers and insisters not being able to identify themselves or express their preferences.

In a blinded study, one can regard the estimated treatment effect from the per-protocol analysis as the same as one would obtain in the preference-based analysis, but only under an assumption that all of the component groups among the two acceptor groups would receive the same treatment benefit. This is the so-called exclusion restriction referred to by McIntosh [12], where outcomes depend only on the treatment taken, and not on the assigned treatment, and the similar nonconfounding compliance assumption defined by Scheiner [14], in which preference groups are prognostically identical. In most practical situations, these assumptions are not likely to be valid. Sommer and Zeger [15] also discuss the situation of one active treatment vs. nothing, with some emphasis on allowing for a placebo effect. It might be possible to extend the preference-based analysis framework to cover blinded studies, for example, by using patient subgroups defined by covariates (such as age) that might be related to preference patterns, and then making additional assumptions about the relationship of outcomes to the covariates [2], [7], [16]. This, however, is beyond the scope of the present article.

Other authors have considered related issues, including the possibility of compliance not being observable with certainty [17], [18], and missing values in conjunction with noncompliance [19]. Newcombe [20] considered randomized consent designs, in which patients are randomized to one group that receives a standard treatment or a second group that receives an experimental treatment if they are willing, and standard treatment otherwise. He compared the estimated pragmatic treatment effect from an ITT analysis for a continuous outcome, with an explanatory estimate based on the compliers. The pragmatic estimate reflects the difference in treatment effects incorporating patient acceptance or refusal of treatment as an intrinsic part of the evaluation, and hence, is pertinent to a policy interpretation of the value of treatment in an entire population. These concepts are analogous to the ITT and preference-based analyses used in this paper, although the randomized consent design is inherently different from the standard trial design that we have considered.

In a recent study, King et al. [21] systematically reviewed articles describing two study designs in which patient or physician preferences were explicitly recognized. In the comprehensive cohort design, patients having strong preferences are offered their treatment of choice, while others are randomized in the usual way; all patients are followed up. In two-stage randomized designs, patients are randomized to either receive their preferred treatment or be randomized to an assigned treatment. King's series of 27 comprehensive cohort studies and five two-stage trials demonstrated that treatment preferences caused a substantial proportion of people to refuse randomization. Estimates of treatment benefit often differed little between randomization and preference groups of patients, suggesting that selection bias associated with treatment preferences had only a minor effect on study validity. However, there were seven studies in which there were statistically significant differences between the randomization and preference groups, with examples occurring of either group having the more favorable outcomes. Unfortunately, King et al. do not describe what the experience had been in their reviewed studies of patients who opted out and therefore received neither of the randomized trial treatments.

3.3. Interpretation of preference-based analysis results 

In situations with three levels of treatment (A, B, or 0), it is important to note that even in case of unblinded studies, we cannot solve the problem of estimating outcome rates in generality, because there are too many potentially different outcome rate parameters for the information provided by the smaller number of observable outcome groups. To make progress, we have to make additional assumptions, such as embodied in our proposed pairs of constraints I and II. Furthermore, these assumptions are unverifiable, at least within the confines of the given study data.

In Example 1, we found that the estimated treatment benefit among compliers was similar under either of our two proposed sets of assumptions. In contrast, the preference-based analysis in Example 2 showed either greater or smaller treatment benefit than the other analyses, depending on the assumptions made. This contrast underlines the importance of independent verification of the assumptions. For instance, one might draw on other literature to support the notion of a constant RR. Empirical explorations of the issue suggest that relative risk is similar across a variety of prognostic groups in most, but not all situations [22], [23]. The systematic review by King et al. [21] also provides some limited support for this position. The alternative assumption of independent preference effects, while having some face validity, would also need some external validation, for instance, through suitable survey data from actual or potential study participants, or from similar studies elsewhere.

In addition to estimating the treatment benefit for compliers and other preference groups, our approach also permits estimation of the preference distribution, and the proportion of compliers in particular. This information is itself of interest in projecting potential uptake of a new treatment. Thus, in Example 1 we found a fairly high percentage of compliers, and only a slight preference in favor of one treatment over the other. Such findings would be particularly helpful if the investigators hope to demonstrate equivalence between the two treatments under investigation. Similar preference patterns, in conjunction with similar outcome rates, would reinforce the likely acceptability of the new treatment compared to the old one. In other situations, it might be useful to estimate the proportion of patients likely to prefer one treatment and potentially reject the other if it is offered. For instance, in Example 2, there was a low percentage of compliers, and a substantial majority of patients preferred or insisted on one treatment, relative to the other. In the clinical trial data of Examples 3 and 4, we were able to identify differential preferences for the two interventions.

It is possible to statistically test a hypothesis of equal compliance rates to both treatments [11], but we prefer to focus on estimates of the actual compliance rates. Further work is needed to establish the statistical properties of the preference-based estimates of treatment effect and the preference distribution. It is also important to recognize that compliance and preference patterns in a randomized trial, in which patients undergo a consent process, may well be different from what might be expected in routine clinical practice in the future. Clinicians involved in clinical trials are relatively constrained, and are less likely to deviate from the randomized treatment, compared to clinicians in future clinical practice. One might therefore expect that the proportion of ambivalent (or compliant) patients or clinicians would decrease once the clinical trial is over.

When there are only two levels of treatment (A vs. B, or A vs. 0), the situation is somewhat simpler. We demonstrated that one can then estimate the treatment effect and the preference distribution without further constraints, as long as defiers can be ignored. Cuzick et al. [11] have discussed the statistical properties of the treatment effect estimate in this situation and compared it to the ITT analysis. In studies with randomization between an active intervention and nothing (such as in cancer screening), one can be confident about adopting a two-level approach to the analysis.

In other situations, however, it may be less clear if two or three levels of treatment are required. Limited reporting can be a barrier to implementing a preference-based analysis of previously published studies. When the randomization is between two active interventions (A vs. B), authors often fail to report the possibility or existence of patients who refused either form of treatment, and may simply perform a standard ITT analysis without explicit recognition of whether patients received either or neither treatment. Some authors do report the frequencies of crossovers, but they may not provide enough detail to estimate the outcome rates in those groups. For example, in several studies comparing coronary artery bypass grafting with angioplasty [24], [25], [26], investigators reported the number of patients who received the assigned or alternative study treatment, or who had neither, together with the reasons for noncompliance (such as randomization errors, physician override, or a change in patient condition). However, the outcome rates in the noncompliers were not described.

Similarly, studies have long been reported using only one of the traditional analyses, even though several types of analysis may have been done. For instance, in a 1977 study of treatments for angina [27], the authors reported carrying out three types of analysis: the ITT, as-treated, and per-protocol analyses, with the last analysis in two variants (deleting noncompliers, or deleting noncompliers after the point of crossover). Nevertheless, only the second variant of the per-protocol analysis was reported in detail. These several analyses were done because of the authors' concerns that no single analysis “satisfied both the requirement for common sense and unbiasedness” [27].

Our preference-based analysis may alleviate some of these perceived deficiencies with the traditional approaches, and it is one that can easily be done by investigators with access to individual level data. However, because of limitations in many published reports, assembling the data from previous studies in an appropriate form (patient frequencies and outcome rates according to the randomized treatment assignment and the actual treatment received) may not be possible without further effort, such as getting individual level data from the investigators.

In summary, the framework presented in this article provides a mechanism to assess the acceptability of two alternative treatments, and to estimate the relative treatment benefit among patients who would accept either randomized assignment, as well as in the other preference groups. Patients committed to adhering to an offered treatment, and their clinicians, will welcome a suitable estimate of the magnitude of treatment benefit they can expect.

Back to Article Outline

Acknowledgments 

The authors thank Drs. Brian Haynes and P.J. Devereaux for early discussions on this work, Lauren Griffith for assistance with the CT-Med example (all from Dept. of Clinical Epidemiology and Biostatistics, McMaster University), and Kevin Thorpe (Knowledge Translation Program and Dept. of Public Health Sciences, University of Toronto) for assistance with the NASCET data example.

Back to Article Outline

References 

1. White IR, Babiker AG, Walker S, Darbyshire JH. Randomization methods for correcting for treatment changes. Stat Med. 1999;18:2617–2634
   • View In Article
   • MEDLINE
   • CrossRef
2. Goetghebeur E, Loeys T. Beyond intention to treat. Epidemiol Rev. 2002;24:85–90
   • View In Article
   • MEDLINE
   • CrossRef
3. Yusuf S, Wittes J, Probstfield J, Tyroller HA. Analysis and interpretation of treatment effects in subgroups of patients in randomized clinical trials. JAMA. 1991;266:93–98
   • View In Article
   • MEDLINE
4. Lee J, Ellenberg J, Hirtz D, et al. Analysis of clinical trials by treatment actually received: is it really an option?. Stat Med. 1991;10:1595–1605
   • View In Article
   • MEDLINE
   • CrossRef
5. White IR, Dunn G. Adjustment for non-compliance in randomized controlled trials. In:  Everitt BS,  Palmer CR editor. Encyclopedic companion to medical statistics. London: Hodder; 2004;
   • View In Article
6. DiMatteo MR, Giordani PJ, Lepper HS, Croghan TW. Patient adherence and medical treatment outcomes. Med Care. 2002;40:794–811
   • View In Article
   • MEDLINE
   • CrossRef
7. Greenland S. An introduction for instrumental variables for epidemiologists. Int J Epidemiol. 2000;29:722–729
   • View In Article
   • MEDLINE
   • CrossRef
8. White IR. Uses and limitations of randomization-based efficacy estimators. Stat Methods Med Res. 2005;14:327–347
   • View In Article
   • MEDLINE
   • CrossRef
9. Canadian Lung Oncology Group . Investigation for mediastinal disease in patients with apparently operable lung cancer. Ann Thoracic Surg. 1995;60:1382–1389
   • View In Article
10. Barnett HJM NASCET collaborators. Beneficial effect of carotid endarterectomy in symptomatic patients with high-grade carotid stenosis. N Engl J Med. 1991;325:445–453
    • View In Article
    • MEDLINE
    • CrossRef
11. Cuzick J, Edwards R, Segnan N. Adjusting for non-compliance and contamination in randomized clinical trials. Stat Med. 1997;16:1017–1029
    • View In Article
    • MEDLINE
    • CrossRef
12. McIntosh MW. Instrumental variables when evaluating screening trials. Stat Med. 1999;18:2775–2794
    • View In Article
    • MEDLINE
    • CrossRef
13. Etzioni RD, Connor RJ, Prorock PC, Self SG. Design and analysis of cancer screening trials. Stat Methods Med Res. 1995;4:3–17
    • View In Article
    • MEDLINE
    • CrossRef
14. Scheiner LB. Is intent-to-treat analysis always (ever) enough?. Br J Clin Pharmacol. 2002;54:203–211
    • View In Article
    • MEDLINE
    • CrossRef
15. Sommer A, Zeger SL. On estimating efficacy from clinical trials. Stat Med. 1991;10:45–52
    • View In Article
    • MEDLINE
    • CrossRef
16. Robins JM. Correction for non-compliance in equivalence trials. Stat Med. 1998;17:269–302
    • View In Article
    • MEDLINE
    • CrossRef
17. Kenna LA, Scheiner LB. Estimating treatment effect in the presence of non-compliance measured with error. Stat Med. 2004;23:3561–3580
    • View In Article
    • MEDLINE
    • CrossRef
18. Dunn G. The problem of measurement error in modelling the effect of compliance in a randomized clinical trial. Stat Med. 1999;18:2863–2877
    • View In Article
    • MEDLINE
19. Mealli F, Imbens GW, Ferro S, Biggeri A. Analysing a randomized trial on breast self-examination with noncompliance and missing outcomes. Biostatistics. 2004;5:207–222
    • View In Article
    • MEDLINE
    • CrossRef
20. Newcombe RG. Explanatory and pragmatic estimates of the treatment effect when deviations from allocated treatment occur. Stat Med. 1988;7:1179–1186
    • View In Article
    • MEDLINE
    • CrossRef
21. King M, Nazareth I, Lampe F, Bower P, et al. Impact of participant and physician intervention preferences on randomized trials. JAMA. 2005;293:1089–1099
    • View In Article
    • CrossRef
22. Schmid CH, Lau J, McIntosh MW, Cappelleri JC. An empirical study of the effect of the control rate as a predictor of treatment efficacy in meta-analysis of clinical trials. Stat Med. 1998;17:1923–1942
    • View In Article
    • MEDLINE
    • CrossRef
23. Furukawa TA, Guyatt GH, Griffith LE. Can we individualize the Number Needed to Treat (NNT)? An empirical study of summary effect measures in meta-analyses. Int J Epidemiol. 2002;31:72–76
    • View In Article
    • MEDLINE
    • CrossRef
24. Hampton JR, Henderson RA, Julian DG, Parker J, et al. Coronary angioplasty versus coronary artery bypass surgery: the Randomized Intervention Treatment of Angina (RITA) trial. Lancet. 1993;341:573–580
    • View In Article
    • Abstract
    • CrossRef
25. CABRI Trial Participants . First-year results of CABRI (Coronary Angioplasty versus Bypass Revascularisation Investigation). Lancet. 1995;346:1179–1184
    • View In Article
    • Abstract
    • CrossRef
26. BARI Investigators . Comparison of coronary bypass surgery with angioplasty in patients with multivessel disease. N Engl J Med. 1996;335:217–225
    • View In Article
    • MEDLINE
    • CrossRef
27. Murphy ML, Hultgren HN, Detre K, Thomsen J, et al. Treatment of chronic stable angina. N Engl J Med. 1977;297:621–627
    • View In Article
    • MEDLINE
    • CrossRef