Volume 60, Issue 11 , Pages 1123-1126, November 2007
Publication bias for CAM trials in the highest impact factor medicine journals is partly due to geographical bias
Article Outline
Abstract
Objective
To assess the presence of publication bias and its relation to geographical bias in clinical trials involving complementary and alternative medicine (CAM) published in the highest impact factor general medicine journals.
Study Design and Setting
All CAM clinical trials published in the four highest impact factor general medicine journals, Lancet and British Medical Journal (European), and New England Journal of Medicine and Journal of American Medical Association (U.S.), between 1965 and 2004 were abstracted using Medline. Three reviewers abstracted data from the individual studies. In a multivariate analysis, factors predictive of a positive study were assessed.
Results
A total of 259 studies met the inclusion criteria. CAM trials published in the European journals were significantly more likely to be positive compared to those published in the U.S. journals (76% vs. 50%, odds ratio [OR]
=3.15, P<0.0001). Studies originating outside of the United States were significantly more likely to be positive compared to the U.S. studies (75% vs. 49%, P<0.0001). Adjusting for location and other variables in a multivariate model, the OR for European vs. U.S. journals to publish a positive CAM trial was 1.95 (P=0.11).
Conclusion
Publication bias related to CAM trials among the highest impact factor general medicine journals is partly due to geographical bias.
Keywords: Publication bias, Complementary therapies, Treatment outcome, Clinical trials, Geographic factors, Alternative medicine
1. Introduction
Complementary and alternative medicine (CAM) journals are purported to be biased toward publishing only the positive clinical studies [1]. This bias has marginally improved in recent years [2], [3]. Clinical trials published from some countries, particularly from Asia, are significantly more likely to be positive compared to trials conducted in the western countries [4]. The impact of geographical bias (influence of the location of the study on the study outcome) on the publication bias, particularly in the highest impact factor general medicine journals, is not presently known.
The highest impact factor general medicine journals published in the United States (New England Journal of Medicine and Journal of American Medical Association) are observed to publish far more negative CAM clinical trials compared to the highest impact factor European general medicine journals (Lancet and British Medical Journal). In the last few years, negative studies with several high-profile biological products were published in the U.S. journals [5], [6], [7], [8], whereas studies with two of these products showed favorable results in the European journals [9], [10]. Assessment of the potential for publication bias in the highest impact factor journals is particularly important because these journals often publish the most newsworthy and defining articles in medicine.
In the present study, we investigate whether such a publication bias truly exists, and also explore the reasons for this bias.
2. Methods
2.1. Data sources
We selected the four highest impact factor general medicine journals, two from Europe (Lancet and British Medical Journal) and two from the United States (New England Journal of Medicine and Journal of American Medical Association), based on the ISI Web of Knowledge Journal Citation Reports (2004 edition). All clinical trials with CAM treatments published in these four journals were abstracted.
2.2. Search strategy
Two search strategies were used. The primary search was performed using the “Journals database” in PubMed. The search limits used were “Clinical Trial,” “1965 to 2004,” and “Complementary Medicine” subset. The “Complementary Medicine” subset adds a sensitive search term that allows the user to limit the search output to clinical trials using CAM treatments [11]. The second method used a comprehensive text words search developed with the help of an expert librarian.
2.3. Data extraction
Three board-certified practicing internal medicine physicians with experience in complementary medicine (A.S., B.A.B., and D.L.W.-R.) working in parallel reviewed abstracts of all the articles. Review articles, editorials, letters to the editor, and articles where the study intervention was not CAM as defined by the National Center for Complementary and Alternative Medicine (NCCAM) were then excluded [12]. NCCAM defines CAM as a group of diverse medical and health care systems, practices, and products that are not presently considered to be part of conventional medicine. Minor inconsistencies were resolved by consensus. A few disagreements about inclusion were resolved via a third party.
The selected articles were then reviewed to obtain the following variables: study outcome, blinding, randomization, sample size, duration of follow-up, and location of the study. A positive outcome was defined as P-value ≤0.05, or the confidence interval (CI) for the efficacy estimate (odds ratio [OR], relative risk) between the active and comparison groups excluding the possibility of no effect and favoring the study intervention for the primary outcome. The study was considered to be blinded if the participants, investigator, or both were kept masked to the study intervention. The study was considered randomized if this was noted in the methods section, and a concurrent control group was part of the study. We did not assess the quality or method of randomization for this analysis.
2.4. Statistical analysis
Univariate and multivariate logistic regression analyses were performed using outcome of the study (positive or negative) as the dependent variable, and country of journal, randomization, blinding, sample size, CAM intervention, duration of follow-up, and location of the study as the predictor variables. All statistical tests were performed using the SAS V 9.1.3 software.
3. Results
3.1. Search results
We identified a total of 721 clinical trials with the “Complementary Medicine” subset search (Table 1). Of these, 256 articles met all the inclusion criteria. An additional three articles fulfilling the inclusion criteria were obtained using the text words search. Articles wherein the study intervention could not be classified as a CAM treatment, or the study design was not a clinical trial, were excluded.
Table 1. Total number of clinical trials with CAM as the intervention published from 1965 to 2004 in the highest impact factor general medicine journals
| JAMA | NEJM | Lancet | BMJ | |
|---|---|---|---|---|
| Total | 110 | 148 | 277 | 186 |
| Not related to CAM | 31 | 89 | 126 | 80 |
| Not clinical trials | 20 | 16 | 66 | 27 |
| Other reasons (e.g., duplicate citation) | 0 | 0 | 1 | 8 |
| Articles included | 59 | 43 | 82 | 72 |
3.2. Study descriptions
Most studies (92%) were randomized, and 61% had the participants, investigators, or both blinded (Table 2). The median sample size was 107, with the median duration of follow-up being 4 weeks. Of all the studies, 170 (66%) had a positive outcome for the primary endpoint. The majority of studies originated in the United States or Europe. Studies originating outside of the United States were significantly more likely to be positive compared to the U.S. studies (75% vs. 49%, P<0.0001).
Table 2. Description of individual clinical trials with CAM as the intervention published from 1965 to 2004 in the highest impact factor general medicine journals
| Variable | n (%) | Median (SD, range) |
|---|---|---|
| Positive outcome | 171 (66) | |
| Randomized (yes) | 238 (92) | |
| Blinded (yes) | 157 (61) | |
| Location of the study | ||
| United States | 89 (35) | |
| Europe | 134 (52) | |
| Asia | 15 (6) | |
| Others | 19 (7) | |
| Sample size | 107 (5–29,133) | |
| Follow-up duration (wk) | 4 (0.01–312) | |
3.3. Comparison of European vs. U.S. journals
European journals were significantly more likely to publish positive CAM trials compared to U.S. journals (OR=3.15, 95% CI=1.85–5.38, P<0.0001). European journals were also significantly more likely to publish studies originating outside of the United States (93% vs. 24%, P<0.0001). No significant differences were noticed between the European and U.S. journals in randomization, blinding, sample size, CAM intervention, and duration of follow-up (Table 3). Adjusting for all these variables in a multivariable logistic regression model, the OR for European vs. U.S. journals to publish a positive CAM trial was 1.95 (95% CI=0.87–4.44, P=0.11).
Table 3. A univariate comparison of studies with CAM as the intervention published from 1965 to 2004 in the highest impact factor general medicine journals
| Variable | Category | U.S. journals, n (%) | European journals, n %) | P-value |
|---|---|---|---|---|
| Study outcome | Positive | 52 (50) | 119 (76) | <0.0001 |
| Negative | 51 (50) | 37 (24) | ||
| Randomization | Yes | 97 (94) | 141 (90) | 0.27 |
| No | 6 (6) | 15 (10) | ||
| Blinding | Yes | 59 (57) | 98 (63) | 0.37 |
| No | 44 (43) | 58 (37) | ||
| CAM intervention | Biologically based practices | 71 (69) | 85 (55) | 0.06 |
| Manipulative and body-based practices | 5 (5) | 12 (8) | ||
| Mind–body medicine | 16 (15) | 32 (21) | ||
| Whole medical systems | 0 (0) | 8 (5) | ||
| Energy medicine | 11 (11) | 18 (11) | ||
| Location of the study | U.S. study | 78 (76) | 11 (7) | <0.0001 |
| European study | 18 (17) | 116 (75) | ||
| Asian study | 1 (1) | 14 (9) | ||
| Others | 6 (6) | 15 (9) | ||
| Median (range) | Median (range) | |||
| Sample size | 132 (5, 29,133) | 96 (6, 20,536) | 0.19 | |
| Follow-up duration (wk) | 4 (0.1, 144) | 4 (0.01, 312) | 0.14 | |
4. Discussion
4.1. Summary of results
We find that among the highest impact factor general medicine journals, European journals are significantly more likely to publish positive CAM trials compared to U.S. journals (OR=3.15, 95% CI=1.85–5.38, P<0.0001). This is likely related to the geographic location of the study, with studies originating outside of the United States being significantly more likely to be positive compared to the U.S. studies (75% vs. 49%, P<0.0001). Adjusting for all reported factors related to the study such as the location of the study, the sample size, the duration of follow-up, and experimental design factors, the difference between the two groups of journals is not statistically significant.
4.2. Literature review
Several previous studies have reported the presence of publication bias in CAM studies. Vickers et al. systematically reviewed 252 CAM clinical trials published from 1966 to 1995 to understand the prevalence of geographical bias. Among the acupuncture studies, all the trials conducted in China, Japan, Hong Kong, and Taiwan were positive (100%), compared to 60% for the British and 53% for the U.S. studies [4]. This difference was also observed for studies with intervention other than acupuncture. Ernst et al. compared the number of negative trials published in three CAM journals in 1995 and 2000 [1], [2]. In 1995, only 1% (1/179) of the published trials reported a negative outcome, compared to 5% (10/207) in 2000. Pittler et al. evaluated 351 CAM clinical trials (1990–1997) and noted that CAM journals and non–impact factor mainstream journals were more likely (74% and 75%, respectively) to publish positive trials than high impact factor mainstream journals (50%) [13]. No distinction was however made between the different high impact factor journals.
4.3. Weaknesses
Our study has several weaknesses. First, CAM is a broad field with some of the interventions being difficult to classify as CAM or conventional. We used the standard definition of CAM as stated by the NCCAM [12]. Further, abstraction of the data by three board-certified practicing physicians who are actively engaged in CAM research minimized this limitation. Second, we did not use a defined quantitative estimate such as the Jadad score to assess study quality. This was partly related to the fact that in several trials, particularly some of the older studies, we did not find adequate description of all the variables for such calculation. Third, all the variables that might affect study outcome were not assessed, most importantly the attrition rate. Finally, our study does not answer the question why non-U.S. studies are more likely to be positive compared to U.S. studies.
4.4. Future directions
Recognition of the potential for publication bias is important because the four journals included in this study have enormous impact in the field of medicine. Further, with the recent publication of several negative studies with well-recognized natural products in the highest impact factor U.S. journals, it is important to elucidate if truly a bias exists in the editorial decision making [5], [6], [7], [8]. The findings from our study suggest that although systematic differences do exist, they are not likely related to the editorial decision making.
Acknowledgments
We thank Patricia J. Erwin, MLS, for her assistance in performing the text words search.
References
- . Alternative therapy bias. Nature. 1997;385:480
- . Bias in alternative medicine is still rife but is diminishing. BMJ. 2001;323:1071
- . A profile of journals of complementary and alternative medicine. Swiss Med Wkly. 2001;131:588–591
- . Do certain countries produce only positive results? A systematic review of controlled trials. Control Clin Trials. 1998;19:159–166
- Saw palmetto for benign prostatic hyperplasia. N Engl J Med. 2006;354:557–566
- . An evaluation of Echinacea angustifolia in experimental rhinovirus infections. N Engl J Med. 2005;353:341–348
- Fish oil supplementation and risk of ventricular tachycardia and ventricular fibrillation in patients with implantable defibrillators: a randomized controlled trial. JAMA. 2005;293:2884–2891
- Acupuncture for patients with migraine: a randomized controlled trial. JAMA. 2005;293:2118–2125
- . Acute treatment of moderate to severe depression with hypericum extract WS 5570 (St John's wort): randomised controlled double blind non-inferiority trial versus paroxetine. BMJ. 2005;330:503
- Long-term effects of glucosamine sulphate on osteoarthritis progression: a randomised, placebo-controlled clinical trial. Lancet. 2001;357:251–256
- . Search strategies for retrieving complementary and alternative medicine clinical trials in oncology. Integr Cancer Ther. 2006;5:202–205
- Available at http://nccam.nih.gov/health/whatiscam/2005Accessed Jan. 12, 2007
- . Location bias in controlled clinical trials of complementary/alternative therapies. J Clin Epidemiol. 2000;53:485–489
PII: S0895-4356(07)00032-7
doi:10.1016/j.jclinepi.2007.01.009
© 2007 Elsevier Inc. All rights reserved.
Volume 60, Issue 11 , Pages 1123-1126, November 2007
