The effect of study sponsorship on a systematically evaluated body of evidence of head-to-head trials was modest: secondary analysis of a systematic review

1. Introduction 

Multiple studies have documented an association between industry funding and the reporting of positive results [1], [2], [3], [4], [5], [6], [7], [8], [9], [10], [11], [12], [13], [14], [15], [16], [17], [18], [19], [20], [21], [22], [23], [24], [25]. The exact mechanism of this phenomenon often called “industry bias” remains unclear. Conceivable is a concurrence of omission of outcomes or entire studies and an undue focus on positive results [26]. Other proposed mechanisms include biased methods [3], [19], [27] and analyses [5], [28].

Two recent publications on placebo-controlled trials of second-generation antidepressants, submitted to regulatory agencies, highlighted a disturbing picture of publication bias in this drug class [29], [30]. One study based on trials submitted to the Food and Drug Administration (FDA) reported that only 39% (14 of 36) of trials deemed “negative” by the FDA have been published in journals [29]. Similarly, only 29% of placebo-controlled studies not showing statistically significant differences, which were submitted to the Swedish regulatory agency, were published as stand-alone publications [30]. Because several groups (sponsors, editors, authors) are involved in the publication of research, identifying who or what is exactly responsible for publication bias has not been possible.

These studies indicate that for systemic reviews of placebo-controlled trials, publication bias can have dramatic effects on the validity of results and highlights the need for extensive efforts to uncover unpublished studies.

The situation, however, is even more difficult for systematic reviews that examine head-to-head trials to determine the comparative efficacy and safety of two or more interventions. Such reviews, often called comparative effectiveness reviews, have gained increasing importance for policy makers in recent years [31]. Particularly, the interest in the comparative efficacy and safety of drugs has increased over the past years, based on initiatives, such as the Drug Effectiveness Review Project [32] the Agency for Healthcare Research and Quality's Effective Healthcare Program under the Medicare Modernization Act of 2003 [33].

Although placebo-controlled trials are based on regulatory requirements, head-to-head trials are usually conducted postapproval and thus do not have to be registered with regulatory agencies. Therefore, the detection of unpublished head-to-head evidence is substantially more difficult than for placebo-controlled trials. Protocols of comparative effectiveness reviews usually include requests to the industry for unpublished studies. In reality, however, responses to these requests are minimal, usually leaving researchers conducting comparative effectiveness reviews bound to published studies in their analyses of the evidence. As a consequence, uncertainty exists to what extent findings of comparative effectiveness reviews are distorted by underlying industry bias.

Despite their stated objectives, most head-to-head studies of drugs within the same class do not attempt to formally establish superiority or noninferiority of one drug compared with another. Most studies have mere marketing objectives, and publications of results that render similar efficacy between two drugs frequently emphasize other advantages of one drug over the other, for example, differences in adverse events.

It is unclear to date how industry biases from multiple funding sources affect an entire body of evidence in a systematic review of head-to-head studies. It is conceivable that, in a best-case scenario, biases from multiple funding sources act in opposed directions and eliminate themselves in a systematically located body of evidence. In a worst-case scenario, industry bias could be introduced by a single funding source leading to a unidirectional bias that favors the intervention associated with the sponsor. Such a situation might arise, for example, if a new drug is tested against an established medication.

Because most comparative effectiveness reviews will not be able to detect unpublished evidence, we strove to explore the magnitude of industry bias that could be introduced by drawing conclusions based primarily on published evidence. Using data from a recently conducted comparative effectiveness review as a real-world example, our study set forth to simulate the effect of unidirectional industry bias (the worst-case scenario) on a systematically located and evaluated body of evidence. Our hypothesis was that a particular drug would perform better in trials that were funded by its manufacturer than in studies where it was used as a control drug by a competitor.

2. Methods 

The current study is based on a systematic review of the comparative efficacy and safety of second-generation antidepressants for the treatment of major depressive disorder (MDD) [34]. We chose this drug class because well-conducted systematic reviews and meta-analyses indicate that no substantial differences in efficacy exist among selective serotonin reuptake inhibitors (SSRIs), which are a subgroup of second-generation antidepressants [34], [35], [36]. However, results of individual head-to-head trials of SSRIs present considerable variations of effects. For example, point estimates of the comparative efficacy of achieving response range from a 0% relative benefit of one drug over the other in some studies, to up to a 68% higher relative benefit in others. Whether these differences are simply caused by chance fluctuations or whether underlying industry bias is favoring sponsored drugs over control drugs in some studies is unclear.

A detailed description of the methods of the underlying systematic review has been published elsewhere [34]. Briefly, to identify relevant articles, we searched MEDLINE, EMBASE, The Cochrane Library, and the International Pharmaceutical Abstracts database. We combined “major depressive disorder” with a list of 12 specific second-generation antidepressants (bupropion, citalopram, duloxetine, escitalopram, fluoxetine, fluvoxamine, mirtazapine, nefazodone, paroxetine, sertraline, trazodone, and venlafaxine) and their specific trade names; we used either Medical Subject Headings as search terms when available or key words when appropriate. We limited our searches to “human” and “English language.” Sources were searched from 1980 to 2006 (February). In addition, we manually searched reference lists of pertinent review articles and letters to the editor. Furthermore, we explored the Center for Drug Evaluation and Research database to identify unpublished research submitted to the US FDA.

Two persons independently reviewed abstracts and full-text articles and rated the internal validity of each study based on systems developed by the US Preventive Services Task Force [37] and the National Health Service Centre for Reviews and Dissemination [38]. Records were excluded if they did not meet preestablished eligibility criteria. We included all double-blinded randomized controlled trials (RCTs) lasting at least 6 weeks.

In addition to internal validity, we assessed the comparability of dosages. Because we could not find any clear definitions about equivalence of dosages among SSRIs in the published literature, we developed a roster of low, medium, and high dosages for each drug, which is outlined in Table 1. This classification, based on the interquartile dosing range, does not indicate dosing equivalence. We used this roster to detect gross inequalities in dosing that could affect the comparative efficacy. For the current study, we limited our analysis to published head-to-head RCTs of SSRIs (citalopram, escitalopram, fluoxetine, fluvoxamine, paroxetine, and sertraline).

Table 1. SSRIs with dosing classification based on lower and upper dosing range quartiles

	Generic name; US trade name	Dosage range (mg/d)	Low	Medium	High	Manufacturer
	Citalopram; Celexa	20–60	<30	30–50	>50	Forest Laboratories, Inc. and H. Lundbeck A/S
	Escitalopram; Lexapro	10–30	<15	15–25	>25	Forest Laboratories, Inc. and H. Lundbeck A/S
	Fluoxetine; Prozac	20–60	<30	30–50	>50	Eli Lilly and Company
	Fluvoxamine; Luvox	50–150	<75	75–125	>125	Solvay Pharmaceuticals
	Paroxetine; Paxil	20–60	<30	30–50	>50	GlaxoSmithKline
	Sertraline; Zoloft	50–150	<75	75–125	>125	Pfizer, Inc.

Abbreviation: SSRIs, selective serotonin reuptake inhibitors.

To determine the funding source or associations of authors with the funding source, two reviewers independently classified each study according to the following five criteria:

Conflicts in classifications were resolved by consensus. We contacted the lead authors of studies where the funding source was not reported. Table 1 summarizes the included drugs and their manufacturers.

Citalopram and escitalopram are both produced by the same manufacturer (H. Lundbeck A/S, Copenhagen, Denmark). Citalopram, however, is also available as a generic drug, whereas escitalopram is still patent protected. In studies comparing citalopram with escitalopram that were funded by Lundbeck A/S, we viewed citalopram as the control drug and escitalopram as the drug associated with the sponsor.

In a next step, we assigned drugs of studies termed to be clearly funded by the industry or probably funded by the industry to one of two groups:

Consequently, because of multiple funding sources, each drug was represented at least once in each group.

To simulate a situation where all studies have been funded by the same source causing unidirectional bias, we conducted relative-benefit meta-analyses comparing response rates of drugs termed as associated with the funding source with those of drugs termed as not associated with the funding source. The relative benefit reflects the ratio of benefits in one treatment group compared with that in another. Because we were not interested in the comparative response rates of specific drugs on specific rating scales, we included studies with different rating scales (e.g., Montgomery-Asberg Depression Rating Scale [MADRS], Hamilton Depression Rating Scale [HAM-D]). We used the term “response” as defined by the authors of the individual studies. Because we did not have enough data to compare the response rates of individual drugs, we examined all SSRIs as a class.

For each meta-analysis, we tested for heterogeneity of treatment effects using the I2 statistic. The I2 statistic describes the percentage of total variation across studies that is because of heterogeneity rather than chance [39]. Although no heterogeneity was present in any of our analyses (I2=0%), we report results of the more conservative random-effect models. Results of random-effect models, however, yielded almost identical findings as fixed-effect models.

To detect possible publication bias introduced by the tendency of published studies to be positive, we used funnel plots and the Egger regression approach. All statistical analyses were conducted using StatsDirect Statistical Software program, version 2.5.7 (StatsDirect Ltd, Cheshire, United Kingdom 2006), and STATA 9.1 (StataCorp, College Station, TX, USA 2005).

3. Results 

Our searches identified 2,099 citations. Twenty-nine studies met our eligibility criteria and compared one SSRI with another [40], [41], [42], [43], [44], [45], [46], [47], [48], [49], [50], [51], [52], [53], [54], [55], [56], [57], [58], [59], [60], [61], [62], [63], [64], [65], [66], [67], [68]. We did not detect any gross dosing inequalities in these studies. Twenty studies were clearly funded by the pharmaceutical industry [40], [41], [43], [44], [46], [47], [50], [51], [52], [54], [56], [57], [58], [59], [60], [61], [63], [65], [67], [68]. Six studies were probably funded by the industry [42], [45], [48], [55], [62], [66]. One study reported a mixed funding source of public and industry monies [49]. In two studies, we could not determine the funding source after contacting the corresponding authors multiple times [53], [64].

Most trials were funded by the following: H. Lundbeck A/S, the manufacturer of citalopram and escitalopram (seven studies, 2,691 patients) [43], [46], [54], [55], [58], [59], [62]; Pfizer, Inc., New York, NY, USA, the manufacturer of sertraline (six studies, 1,520 patients) [40], [41], [42], [61], [66], [68]; and Glaxo Smithkline, London, United Kingdom, the manufacturer of paroxetine (six studies, 959 patients) [44], [45], [48], [50], [65], [67]. Eli Lilly and Company, Indianapolis, IN, USA, the producer of fluoxetine [51], [52], [57], and Solvay Pharmaceuticals, Brussels, Belgium, the producer of fluvoxamine [47], [56], [60], [63], financially supported three and four RCTs each (1,169 and 439 patients, respectively). Table 2 summarizes the included studies.

Table 2. Study characteristics, status of funding, and quality ratings of included studies

	Study	Status of funding	Drug associated with funding source and dose (mg/d)	Control drug and dose (mg/d)	Sample size and duration (wk)	Response (%) and significance level	Quality rating
	Aberg-Wistedt et al., 2000 [40]	Clearly industry	Sertraline, 50–150	Paroxetine, 20–40	353; 24	69 vs. 72; P=NR (NS)	Fair
	Bennie et al., 1995 [41]	Clearly industry	Sertraline, 50–100	Fluoxetine, 20–40	286; 6	51 vs. 59; P=NR	Fair
	Boyer et al., 1998 [42]	Likely industry	Sertraline, 20–60	Fluoxetine, 50–150	242; 26	42.6 vs. 47.4; P=NR	Fair
	Burke et al., 2002 [43]	Clearly industry	Escitalopram, 20	Citalopram, 40	491; 8	45.6 vs. 51.2; P=NR (NS)	Fair
	Chouinard et al., 1999 [45]	Likely industry	Paroxetine, 20–50	Fluoxetine, 20–80	203; 12	88.4 vs. 85.7; P=NR (NS)	Fair
	Colonna et al., 2005 [46]	Clearly industry	Escitalopram, 10	Citalopram, 20	357; 24	78 vs. 80; P=NR (NS)	Fair
	Dalery and Honig, 2003 [47]	Clearly industry	Fluvoxamine, 100	Fluoxetine, 20	184; 6	60 vs. 60; P=NR (NS)	Fair
	Cassano et al., 2002 [44]	Clearly industry	Paroxetine, 20–40	Fluoxetine, 20–60	242; 52	NR	Fair
	De Wilde et al., 1993 [48]	Likely industry	Paroxetine, 20–40	Fluoxetine, 20–60	100; 6	62 vs. 67; P=NR	Fair
	Ekselius et al, 1997 [49]	Mixed public and industry	Sertraline, 50–150	Citalopram, 20–60	400; 24	81 vs. 75.5; P=NR (NS)	Good
	Fava et al., 2002 [52]	Clearly industry	Fluoxetine, 20–60	Paroxetine, 20–60; Sertraline, 50–200	284; 10–16	64.8 vs. 68.8 vs. 72.9; P=NR	Fair
	Fava et al., 2000 [51]	Clearly industry	Fluoxetine, 20–60	Sertraline, 50–200; Paroxetine, 20–60	284; 26–32	NR	Fair
	Fava et al., 1998 [50]	Clearly industry	Paroxetine, 20–50	Fluoxetine, 20–80	128; 12	NR	Fair
	Gagiano, 1993 [53]	NR	Paroxetine, 20–40	Fluoxetine, 20–60	90; 6	63 vs. 70; P=NR	Fair
	Haffmans et al., 1996 [54]	Clearly industry	Citalopram, 20–40	Fluvoxamine, 100–200	217; 6	30.5 vs. 28.4; P=NR	Fair
	Kasper et al., 2005 [55]	Likely industry	Escitalopram, 10	Fluoxetine, 20	518; 8	46 vs. 37; P=NR (NS)	Fair
	Kiev and Feiger, 1997 [56]	Clearly industry	Fluvoxamine, 50–150	Paroxetine, 20–50	60; 7	Data not reported; P=NR (NS)	Fair
	Kroenke et al. 2001 [57]	Clearly industry	Fluoxetine, 20	Paroxetine, 50; Sertraline, 20	601; 36	NR	Fair
	Lepola et al., 2003 [58]	Clearly industry	Escitalopram, 10–20	Citalopram, 20–40	471; 8	52.6 vs. 63.7; P=0.021	Fair
	Moore et al., 2005 [59]	Clearly industry	Escitalopram, 20	Citalopram, 40	280; 8	61.3 vs. 76.1; P=0.008	Fair
	Nemeroff et al., 1995 [60]	Clearly industry	Fluvoxamine, 50–150	Sertraline, 50–200	95; 7	Data not reported; P=NR (NS)	Fair
	Newhouse et al., 2000 [61]	Clearly industry	Sertraline, 50–100	Fluoxetine, 20–40	236; 12	71 vs. 73; P=NR	Fair
	Patris et al., 1996 [62]	Likely industry	Citalopram, 20	Fluoxetine, 20	357; 8	78 vs. 76; P=NR (NS)	Fair
	Rapaport et al., 1996 [63]	Clearly industry	Fluvoxamine, 100–150	Fluoxetine, 20–80	100; 7	NR	Fair
	Rossini et al., 2005 [64]	NR	Sertraline, 200	Fluvoxamine, 150	93; 7	Data not reported; P=NR (NS)	Fair
	Schöne and Ludwig, 1993 [65]	Clearly industry	Paroxetine, 20–40	Fluoxetine, 20–60	108; 6	Data not reported; P=0.03	Fair
	Sechter et al., 1999 [66]	Likely industry	Sertraline, 50–150	Fluoxetine, 20–60	234; 24	64 vs. 74; P=NR	Fair
	Tignol, 1993 [67]	Clearly industry	Paroxetine, 20	Fluoxetine, 20	178; 6	78 vs. 75; P=NR (NS)	Fair
	Van Moffaert et al., 1995 [68]	Clearly industry	Sertraline, 50	Fluoxetine, 20	165; 8	59 vs. 59; P=NR (NS)	Fair

Abbreviations: NR, not reported; NS, not significant.

All six SSRIs were associated with the funding source in at least one trial. Likewise, all drugs were used as control interventions in at least one trial. Table 3 outlines the number of comparisons of individual drugs and their association with the funding source. In the following paragraphs, we will refer to drugs not associated with the funding source as the “control drugs.”

Table 3. Number of studies comparing a drug associated with the funding source with individual control drugs

	Drug associated with the funding source	Control drug
		Citalopram	Escitalopram	Fluoxetine	Fluvoxamine	Paroxetine	Sertraline
	Citalopram	N/A	0	1	1	0	0
	Escitalopram	4	N/A	1	0	0	0
	Fluoxetine	0	0	N/A	0	2	1
	Fluvoxamine	0	0	2	N/A	1	1
	Paroxetine	0	0	6	0	N/A	0
	Sertraline	0	0	5	0	1	N/A

Abbreviation: N/A, not applicable.

Most studies received a fair quality rating. Only one study was rated good [49]. Limitations of internal validity included high attrition or unclear methods of randomization and allocation concealment. Some of these issues, however, are more likely to be attributed to insufficient reporting in the published manuscripts than to methodological flaws of the study designs.

All studies used either the HAM-D or the MADRS as instruments to assess the comparative efficacy of SSRIs. In all studies, response was defined as a 50% improvement of points on either of these scales. Populations were mostly younger than 60 years and suffered from acute-phase MDD. Twenty studies provided sufficient data for relative-benefit meta-analyses of response to treatment (Fig. 1) [40], [41], [42], [43], [45], [46], [47], [48], [50], [51], [52], [54], [55], [58], [59], [61], [62], [66], [67], [68]. Overall, these trials included data on 4,706 patients. No heterogeneity could be detected in any of our analyses (I2 statistic: 0%).

View Large Image Download to PowerPoint

Fig. 1 Relative-benefit meta-analysis (random effects) comparing response rates of selective serotonin reuptake inhibitors (SSRIs) when associated with the funding source with response rates of SSRIs when not associated with the funding source.

Thirteen out of 20 studies (65%) numerically favored drugs associated with the funding source over drugs used as controls. In two of these studies, differences reached statistical significance [58], [59]. None of the trials found a statistically significant difference in favor of a comparator drug.

The pooled relative benefit of achieving response at study endpoint for drugs clearly or probably funded by the pharmaceutical industry was 1.07 (95% confidence interval [CI]=1.02–1.11; Fig. 1) compared with the same drugs when not associated with the funding source. Although this result reached statistical significance, the difference is not likely to be clinically relevant.

Limiting the analysis to studies that explicitly reported industry support did not change the results (relative benefit=1.07; 95% CI=1.01–1.13). Including the study with mixed industry and public funding [49] in our analyses reduced the point estimate of the effect slightly (relative benefit=1.06; 95% CI=1.02–1.11). The funnel plot (presented in Fig. 2) and the Egger bias test do not suggest any publication bias (P=0.854).

View Large Image Download to PowerPoint

Fig. 2 Funnel plot of relative-benefit meta-analysis.

4. Discussion 

Our study found that the pooled response rate of SSRIs when associated with the funding source is statistically significantly greater than that of the same SSRIs when not associated with the sponsor. Pooled results indicated a relative benefit of 1.07 (95% CI=1.02–1.11) favoring drugs associated with the funding source. The magnitude of the effect of this estimate, however, is small and not likely to be clinically significant. Overall, 65% of studies favored drugs associated with the funding source. In two studies, differences reached statistical significance. None of the trials found a statistically significant difference in favor of the comparator drug.

Results of our study fit well into the mixed picture regarding the effects of industry bias. A comparison of numbers needed to treat (NNTs) derived from pooled estimates of industry- and nonindustry-funded studies of migraine drugs and acute pain relievers did not detect any statistically significant differences in NNTs [26]. Similarly, a meta-analysis of industry-sponsored statin trials [69] yielded the same risk reduction of coronary events as the large, independently funded Heart Protection Study [70].

Nevertheless, most studies have found that industry funding leads to more optimistic outcomes or interpretations of outcomes than public funding [1], [2], [3], [4], [5], [6], [7], [8], [9], [10], [11], [12], [13], [14], [15], [16], [17], [18], [19], [20], [21], [22], [23], [24]. Differences of findings might be attributable to different statistical approaches. The large majority of studies assessing industry bias determined statistical associations between funding sources and conclusions, rather than differences in effect sizes. Furthermore, the influence of industry bias might vary substantially among conditions and comparisons of interventions. Our study focused on head-to-head trials using a real-world example of a comparative effectiveness review on second-generation antidepressants.

One of the great strengths of systematic reviews is their ability to provide a comprehensive picture of the entire body of the available evidence. Individual articles often overemphasize positive results, differences in adverse events, or chance findings simply for marketing reasons. Drug representatives use such articles to aggressively promote their drug of interest to physicians. In such situations, when individual studies are taken as a basis for decision making, industry bias can lead to wrongful conclusions. Positive results can occur by chance, and overemphasizing such findings can be highly misleading. In contrast, comparative effectiveness reviews aggregate data from multiple studies with several different funding sources leading to an entirely different, more balanced perspective of the evidence.

The small effect of industry bias in our study cannot be extrapolated to other drug classes and certainly not to a body of placebo-controlled evidence. The effect of industry bias can be substantially greater in comparisons of drugs vs. placebo or first-generation drugs vs. second-generation drugs. For example, Montgomery et al. found that industry-sponsored trials consistently favored second-generation antipsychotics over first-generation antipsychotics [3]. The Clinical Antipsychotic Trials of Intervention Effectiveness investigation, a recently published, large, publicly funded RCT, however, detected no differences between first- and second-generation antipsychotics [71].

Our study has several limitations. First, the premise of our study was based on the assumption that no significant differences in efficacy exist among SSRIs. Although such a notion has been confirmed by three systematic reviews [34], [35], [36], an exact equivalence of treatment effects is unlikely and would be impossible to prove. Therefore, grouping interventions into two classes, one associated and the other not associated with the sponsor, could yield differences by chance alone. Such a random error, however, could introduce bias in either direction.

Second, our assumption was that industry and publication bias in head-to-head trials follow a similar pattern as was reported from placebo-controlled trials of this drug class. Because placebo-controlled trials are required for regulatory purposes and head-to-head trials are mainly conducted postapproval, this assumption might not be entirely correct. It is conceivable that the proportion of published head-to-head trials is greater than that of placebo-controlled trials. After all, “negative” head-to-head trials can still be used to promote a drug by switching the focus on other differences. The consequence would be that industry bias might actually be less prominent in head-to-head trials of the same drug class than in placebo-controlled trials, which could partly explain the small magnitude of our findings.

Third, as discussed earlier, our results are not generalizable to other drug classes or designs beyond head-to-head trials within this drug class. Our results should not imply that industry bias, in general, is modest. Moreover, our findings are limited to a systematically located and evaluated body of evidence. Finally, although two authors assessed data independently, they were not blinded to the funding status; therefore, author bias cannot be ruled out.

Taken as a whole, it appears that bias may be introduced by industry funding of research, but the evidence on the effect of such a bias is not conclusive. Whether or not industry bias causes erroneous conclusions may vary depending on the condition, intervention, and outcome. The registration of trials before patient enrollment, as required since 2005 by the International Committee of Medical Journal Editors [72], is an important step forward and will, to some extent, increase transparency and reduce industry bias.

Given the increasing occurrence of industry-funded studies in peer-reviewed medical literature, more attention should be paid to the magnitude of potential biases rather than the simple presence or absence of statistical associations between funding source and positive results.