We are happy to learn that our findings about the limited evidentiary value of the large number of existing randomized controlled trials (RCTs) of nicotine replacement therapy (NRT) are consistent with broader epidemiological evidence and public health concerns [
[1]
]. But, we must caution readers that our meta-analysis of NRT RCTs does not “prove” that NRT has no clinical value. Scientific study can never “prove” the absence of some effect or phenomenon [, 3
], and we do not wish to imply otherwise. Nonetheless, we find clear evidence that those RCTs which have greater risks of bias or use smaller, and thereby less reliable, samples report larger positive effects from NRT [[4]
]. Conversely, studies with larger samples and low risks of bias tend to show smaller effects. Our findings merely cast doubt on the strength of the evidence of NRT's clinical efficacy as has been typically reported [5
, 6
]. Although we do not wish to claim that NRT has no effect, we are confident that the size of NRT effect is substantially less than the 50–70% increase in quitting claimed by recent Cochrane Reviews [5
, 6
].Permit us to address a criticism that others are likely to make, especially in response to this letter by Mac Kenzie and Rogers [
[1]
]. The failure to find convincing evidence for a positive clinical effect from NRT need not depend on any meta-regression model of selective reporting bias (aka, “publication” bias). Not all meta-analysts have embraced these meta-regression methods to accommodate selective reporting bias. Skepticism of NRT's strong effectiveness (i.e., 50–70% increase in smoking cessation) need not depend on any meta-regression model, method, or assumption about publication bias.Aside from clear evidence of publication bias ([
4
, 6
]; p.22), another important limitation of these 122 RCTs of NRT is that they are largely underpowered and thereby incapable of detecting the typical effect size that has been reported in the scientific literature. All but five of these 122 RCTs (or 96%) are underpowered [[4]
]. That is, if we use the fixed-effect weighted average of these 122 RCTs as the proxy for “true” effect (log RR = 0.445; representing a 56% increase in quitting) and the widely accepted convention of 80% as adequate statistical power [[7]
], then only five of these 122 (or 4%) have adequate power. This deficiency alone justifies skepticism about the strength of evidence that can be derived from this body of research on NRT's effectiveness. As all researchers know, statistical power is a key dimension to the validity of their research findings. “Unless (we) begin to incorporate methods for increasing the power of (our) studies, the published literature is likely to contain a mixture of apparent results buzzing with confusion. Not only do underpowered studies lead to a confusing literature but they also create a literature that contains biased estimates of effect sizes” [[[8]
], p.161].One sensible response to the power failure of the clinical investigation of NRT is to focus on only those studies which are adequately powered. The weighted least-squares weighted average of these five adequately powered studies is log RR = 0.366 (or a 44% increase in quitting) [
[9]
]. This weighted average of the adequately powered has recently been shown to be useful in reducing selective reporting bias among 159 areas of economics research [[10]
]. However, Stead et al. (2012) judged the blinding integrity of two of these five studies to be at high risk. If we calculate the weighted average of the remaining adequately powered RCTS, we get a smaller effect size, log RR = 0.247 (or a 28% increase in quitting), which is about half the size reported by systematic reviews. Furthermore, the 95% confidence interval for this weighted average (−0.01; 0.50) contains zero; thus, it does not provide clear statistical evidence of the clinical effect from NRT. The advantage of this approach is that no assumption or model of selective reporting bias (aka publication bias and small-sample bias) is used. Merely concentrating on those studies with adequate statistical power and are not identified to be at high risk of bias is sufficient to question the strength of the clinical evidence for NRT.Aside from the statistical issues that cast doubt on the quality of evidence contained among these 122 RCTs of NRT, it is important to put these numbers in context. Most smokers who use NRT do not successfully quit—84%, on average. Smokers in the control groups succeed in quitting about 10% of the time, whereas those receiving NRT have an average quit rate of approximately 16%. When power or selective reporting is considered along with identified risks of bias, this 6% effectiveness is reduced by half, or more. Thus, it seems clear that the vast majority of smokers will not be helped by NRT. What then should our health care systems do for them?
References
- New insights into smoking cessation and nicotine replacement therapy.J Clin Epidemiol. 2016; ([Epub ahead of print])
- Logic of scientific discovery.Hutchinson, London1959
- Conjectures and refutations: the growth of scientific knowledge.Basic Books, New York1963
- Evidence of nicotine replacement's effectiveness dissolves when meta-regression accommodates multiple sources of bias.J Clin Epidemiol. 2016; 79: 49-53
- Nicotine replacement therapy for smoking cessation.Cochrane Database Syst Rev. 2008; : CD000146
- Nicotine replacement therapy for smoking cessation.Cochrane Database Syst Rev. 2012; : CD000146
- Statistical power analysis in the behavioral sciences.2nd ed. Erlbaum, Hillsdale1988
- The persistence of underpowered studies in psychological research: causes, consequences, and remedies.Psychol Methods. 2004; 9: 147-163
- Neither fixed nor random: weighted least squares meta-analysis.Stat Med. 2015; 34: 2116-2127
- The power of bias in economics research.in: The Economic Journal, forthcoming, SWP, Economics Series. Deakin University, Melbourne, Australia2016 (Available at) (Accessed July 15, 2016)
Article info
Publication history
Published online: June 25, 2016
Footnotes
Funding: The author acknowledges support from the Czech Science Foundation (grant 15-02411S).
Conflict of interest: None.
Identification
Copyright
© 2016 Elsevier Inc. All rights reserved.
