Abstract
Objective
To explore the impact of methodological choices on the results of meta-analyses (MAs),
with acupuncture for smoking cessation as a case study.
Study Design and Setting
After performing an umbrella review (using MEDLINE, the COCHRANE Library, the Wan
Fang database, and the Chinese Journal Full-text Database/March 2018) of MAs exploring
the use of acupuncture for smoking cessation, we extracted all randomized controlled
trials. Numerous MAs were performed as per every possible combination of various methodological
choices (e.g., characteristics of the intervention and control procedures, outcome,
publication status, language) to assess their vibration of effects or more precisely
the existence of a Janus effect, that is, whether the 10th and 90th percentiles in
the distribution of effect sizes were in opposite directions.
Results
After including 7 MAs and 39 randomized controlled trials, we performed 496,528 MAs.
The effect size was negative at the 10th percentile (-0.1, favoring controls) and
positive at the 90th percentile (1.17, favoring acupuncture). In all, 104,491 MAs
showed a statistically significant difference in favor of acupuncture, whereas 392,037
failed to demonstrate the efficacy of acupuncture (including 96 that showed a statistically
significant difference in favor of the control).
Conclusion
The methodological choices made in performing pairwise MAs can result in substantial
vibration of effects, occasionally leading to opposite results.
Keywords
To read this article in full you will need to make a payment
Purchase one-time access:
Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online accessOne-time access price info
- For academic or personal research use, select 'Academic and Personal'
- For corporate R&D use, select 'Corporate R&D Professionals'
Subscribe:
Subscribe to Journal of Clinical EpidemiologyAlready a print subscriber? Claim online access
Already an online subscriber? Sign in
Register: Create an account
Institutional Access: Sign in to ScienceDirect
References
- ‘Acupuncture--a critical analysis’.J Intern Med. 2006; 259: 125-137
- ‘A literature review of de qi in clinical studies’.Acupunct Med. 2013; 31: 132-142
- ‘Acupuncture: theory, efficacy, and practice’.Ann Intern Med. 2002; 136: 374-383
- ‘Acupuncture and endorphins’.Neurosci Lett. 2004; 361: 258-261
- ‘Acupuncture prevents 6-hydroxydopamine-induced neuronal death in the nigrostriatal dopaminergic system in the rat Parkinson’s disease model’.Exp Neurol. 2003; 180: 93-98
- ‘Electro-acupuncture stimulation protects dopaminergic neurons from inflammation-mediated damage in medial forebrain bundle-transected rats’.Exp Neurol. 2004; 189: 189-196
- ‘Naloxone fails to reverse pain thresholds elevated by acupuncture: acupuncture analgesia reconsidered’.Pain. 1983; 16: 13-31
- ‘Electroacupuncture analgesia in rats: naltrexone antagonism is dependent on previous exposure’.Brain Res. 1991; 549: 47-51
- ‘Is there a BOLD response of the visual cortex on stimulation of the vision-related acupoint GB 37?’.J Magn Reson Imaging JMRI. 2002; 15: 227-232
- ‘A meta-analysis of the effectiveness of acupuncture in smoking cessation’.Drug Alcohol Rev. 1997; 16: 33-40
- ‘Is acupuncture beneficial in tobacco cessation, and is its effect specific? A meta-analysis’.Acupunct Moxibustion. 2002; 1: 76-85
- ‘Systematic review and meta-analysis of the effects of acupoint stimulation on smoking cessation’.Am J Chin Med. 2012; 40: 429-442
- ‘Alternative smoking cessation aids: a meta-analysis of randomized controlled trials’.Am J Med. 2012; 125: 576-584
- ‘A meta-analysis of studies into the effect of acupuncture on addiction’.Br J Gen Pract J R Coll Gen Pract. 1990; 40: 379-382
- ‘A meta-analysis of acupuncture techniques for smoking cessation’.Tob Control. 1999; 8: 393-397
- ‘The effects of auricular acupuncture on smoking cessation may not depend on the point chosen--an exploratory meta-analysis’.Acupunct Med J Br Med Acupunct Soc. 2006; 24: 149-156
- ‘Acupuncture and related interventions for smoking cessation’.Cochrane Database Syst Rev. 2014; 1: CD000009
- ‘A meta-analysis of ear-acupuncture, ear-acupressure and auriculotherapy for cigarette smoking cessation’.Drug Alcohol Depend. 2014; 142: 14-23
- ‘Evaluation of acupuncture as a treatment for smoking’.Am J Acupunct. 1988; 16: 135-142
- ‘Overlapping meta-analyses on the same topic: survey of published studies’.BMJ. 2013; 347: f4501
- ‘Assessment of vibration of effects due to model specification can demonstrate the instability of observational associations’.J Clin Epidemiol. 2015; 68: 1046-1058
- ‘Controlling for non-specific effects of acupuncture in clinical trials’.Clin Acupunct Orient Med. 2003; 2–3: 59-70
- ‘The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials’.BMJ. 2011; 343: d5928
- Introduction to meta-analysis.John Wiley & Sons, West Sussex, UK2021
- ‘[Condition and effectiveness evaluation of acupuncture for smoking cessation]’.Zhongguo Zhen Jiu Chin Acupunct Moxibustion. 2015; 35: 851-857
- ‘Aural electrical stimulation for smoking cessation - a double blind, placebo controlled study [Abstract].’.Eur Respir J. 2005; 26: 388-389
- ‘Efficacy of an alternative smoking cessation treatment’.J Addict Dis. 2011; 30: 368-381
- ‘Laser acupuncture for adolescent smokers—a randomized double-blind controlled trial’.Am J Chin Med. 2000; 25: 443-449
- ‘Helping people to stop smoking: randomised comparison of groups being treated with acupuncture and nicotine gum with control group’.Br Med J Clin Res Ed. 1985; 291: 1538-1539
- ‘A randomised 2 x 2 factorial design to evaluate different smoking cessation methods’.Rev Epidemiol Sante Publique. 1992; 40: 187-190
- ‘Smoking cessation with behaviour therapy of acupuncture--a controlled study’.Behav Res Ther. 1983; 21: 417-424
- ‘Laser and NRT Smoking Cessation Programmes in Areas of High Social Deprivation’.58. 2003: iii43
- ‘The efficacy of auriculotherapy for smoking cessation: a randomized, placebo-controlled trial’.J Am Board Fam Med JABFM. 2013; 26: 61-70
- ‘Electrical stimulation therapy in the treatment of cigarette smoking’.J Subst Abuse. 1998; 10: 265-274
- ‘Auricular acupuncture for smoking withdrawal’.Am J Acupunct. 1977; 5: 239-247
- ‘Acupuncture and group therapy in stopping smoking’.Pract. 1984; 228: 341-344
- ‘Effects of acupuncture on smoking cessation or reduction for motivated smokers’.Prev Med. 1997; 26: 208-214
- ‘Low level laser for the stimulation of acupoints for smoking cessation: a double blind, placebo controlled randomised trial and semi structured interviews’.J Chin Med. 2008; 86: 46-51
- ‘Acupuncture for smokers: lack of long-term therapeutic effect in a controlled study’.Can Med Assoc J. 1980; 122: 787-790
- ‘Smoking cessation by auricular acupuncture and behavioral therapy.’.Psychol Int J Psychol Orient. 1991; 34: 177-187
- ‘The efficacy of acupuncture as an aid to stopping smoking’.N Z Med J. 1981; 93: 421-423
- ‘The use of acupuncture for smoking withdrawal.’.Am J Acupunct. 1977; 5: 363-366
- ‘Acupuncture therapy for the treatment of tobacco smoking addiction’.Am J Chin Med. 1982; 10: 107-121
- ‘Treating smoking addiction with the ear point seed pressing method’.J Chin Med. 1996; 52
- ‘A single-blind, placebo-controlled trial of a simple acupuncture treatment in the cessation of smoking’.Br J Gen Pract J R Coll Gen Pract. 1998; 48: 1487-1490
- ‘Randomized trial of acupuncture for nicotine withdrawal symptoms’.Arch Intern Med. 1998; 158: 2251-2255
- ‘Acupressure for smoking cessation--a pilot study’.BMC Complement Altern Med. 2007; 7: 8-16
- ‘Auricular acupressure for smoking cessation: a pilot randomized controlled trial’.Med Acupunct. 2010; 22: 265-271
- ‘A randomized controlled clinical trial of auricular acupuncture in smoking cessation’.J Chin Med Assoc JCMA. 2007; 70: 331-338
- ‘A six-week acupoint stimulation intervention for quitting smoking’.Am J Chin Med. 2009; 37: 829-836
- ‘Ear acupressure for smoking cessation: a randomised controlled trial’.in: Evidence-Based Complementary and Alternative Medicine. Masters dissertation at Guangzhou University of Chinese Medicine. a University in Guangzhou, Guangdong Province, China2013: 2013 (637073)
- ‘Contributions of auriculotherapy in smoking cessation: a pilot study’.Rev Esc Enferm U P. 2014; 48: 883-890
- ‘The effects of the acupuncture treatment for smoking cessation in high school student smokers’.Yonsei Med J. 2005; 46: 206-212
- ‘Electroacupuncture therapy in nicotine dependence: a double blind, sham-controlled study’.Noro Psikiyatri Arsivi. 2016; 53: 28-32
- ‘The effects of auricular acupressure on smoking cessation for male college students’.West J Nurs Res. 2017; 39: 374-387
- ‘Clinical research of the seed of cowherb auricular pressing in treating 70 cases of quitting smoking’.Jilin J Tradit Chin Med. 2009; 29: 505-506
- ‘The approach of ear acupuncture combining to psychological intervention with the health belief mode in community tobacco control.’.Zhejiang Prev Med. 2011; 23: 23-25
- ‘[Clinical study on auricular acupuncture for treatment of smoking addiction][Chinese]’.Shanghai J Acupunct Moxibustion. 1983; 2: 30-31
- ‘The Clinical Observation of Nicotine Dependence Treated with Smoke Three-Needles Combine Auricular Pressure Therapy’, Masters Diss.Masters dissertation, Guangzhou University of Chinese Medicin, 2012
- ‘[Tobacco withdrawal. Efficacy of acupuncture in a comparative trial]’.Ann Med Interne (Paris). 1977; 128: 405-408
- ‘[Antismoking acupuncture. Short-term results of a double-blind comparative study]’.Nouv Presse Med. Mar. 1980; 9: 966
- ‘Study of the specific contribution of acupuncture to tobacco detoxication [Etude de l’action spécifique de l’acupuncture dans la cure de sevrage tabagique]’.Sem Hôp Paris. 1985; 61: 2155-2160
- ‘Clinical trial of the role of acupuncture in the fight against tobacco addiction [Essai therapeutique sur le role de l’acupuncture dans la lutte contre le tabagisme].’.Acupuncture. 1977; 51: 13-20
- ‘First results of an anti-smoke outpatient unit: Comparison among three methods [Primi risultati di un ambulatori antifumo. Confronto fra tre metodice].’.Riv. Cardiol. Prev. E Riabil. 1985; 3: 147-151
- ‘Vibration of effects from diverse inclusion/exclusion criteria and analytical choices: 9216 different ways to perform an indirect comparison meta-analysis’.BMC Med. 2019; 17: 174-186
- ‘Positive results in randomized controlled trials on acupuncture published in Chinese journals: a systematic literature review’.J Altern Complement Med. 2014; 20: A129
- ‘Complementary medicines’.in: Wiley StatsRef: Statistics Reference Online. Wiley, West Sussex, UK2015: 1-13
- ‘Evaluation des médecines complémentaires: quels compléments aux essais contrôlés randomisés et aux méta-analyses?’.PhD thesis at Paris-Saclay Université Paris Saclay (COmUE), Paris, France2017
- ‘Out of the laboratory and into the clinic: acupuncture research in the real world’.Clin Acupunct Orient Med. 2000; 1: 97-100
- ‘How large are the nonspecific effects of acupuncture? A meta-analysis of randomized controlled trials’.BMC Med. 2010; 8: 75-88
- ‘Are minimal, superficial or sham acupuncture procedures acceptable as inert placebo controls?’.Acupunct Med J Br Med Acupunct Soc. 2006; 24: 13-15
- ‘Do medical devices have enhanced placebo effects?’.J Clin Epidemiol. 2000; 53: 786-792
- ‘Is acupuncture a placebo therapy?’.Complement Ther Med. 2014; 22: 724-730
- ‘Characteristic and incidental (placebo) effects in complex interventions such as acupuncture’.BMJ. 2005; 330: 1202-1205
- ‘Acupuncture is theatrical placebo’.Anesth Analg. 2013; 116: 1360-1363
- Quality of evidence of the efficacy of therapeutic interventions on patient-important outcomes in Cochrane’s systematic reviews’ abstracts: A survey.Therapies. 2021; 76: 403-408
- ‘GOSH - a graphical display of study heterogeneity’.Res Synth Methods. 2012; 3: 214-223
- ‘Retraction: comparison of acupuncture and other drugs for chronic constipation: a network meta-analysis’.PLoS One. 2018; 13: e0201274
- ‘Association between analytic strategy and estimates of treatment outcomes in meta-analyses’.JAMA. 2014; 312: 623-630
- ‘Single-center trials show larger treatment effects than multicenter trials: evidence from a meta-epidemiologic study’.Ann Intern Med. 2011; 155: 39-51
- ‘Influence of reported study design characteristics on intervention effect estimates from randomized, controlled trials’.Ann Intern Med. 2012; 157: 429-438
- ‘Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study’.BMJ. 2008; 336: 601-605
- ‘Systematic reviews in health care: assessing the quality of controlled clinical trials’.BMJ. 2001; 323: 42-46
- ‘Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials’.Int J Epidemiol. 2007; 36: 847-857
- ‘Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials’.JAMA. 1995; 273: 408-412
- ‘Increasing transparency through a multiverse analysis’.Perspect Psychol Sci J Assoc Psychol Sci. 2016; 11: 702-712
- ‘Cherry-picking by trialists and meta-analysts can drive conclusions about intervention efficacy’.J Clin Epidemiol. 2017; 91: 95-110
- ‘The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses’.Milbank Q. 2016; 94: 485-514
- ‘The metawars’.Science. 2018; 361: 1184-1188
Article info
Publication history
Published online: September 20, 2022
Accepted:
September 5,
2022
Footnotes
Study registration: https://osf.io/a7ncu/.
Identification
Copyright
© 2022 Elsevier Inc. All rights reserved.
