Advertisement

Adequate methods should be used to compare the features of effect measures

  • Ralf Bender
    Correspondence
    Tel.: +49-221-35685-451; fax: +49-221-35685-891
    Affiliations
    Department of Medical Biometry, Institute for Quality and Efficiency in Health Care (IQWiG), Dillenburger Str. 27, D-51105 Cologne, Germany
    Search for articles by this author
      To the Editor:
      In a recent article on the considerable variation of the number needed to treat (NNT), Wisløff et al. [
      • Wisløff T.
      • Aalen O.O.
      • Kristiansen I.S.
      Considerable variation in NNT: a study based on Monte Carlo simulations.
      ] used Monte Carlo simulations to compare the distribution of NNT with that of the relative risk (RR). The methods used have the following limitations and shortcomings.
      At first, to reach the trivial conclusion that observed values of RR have a regular continuous distribution, whereas the distribution of NNT is “irregular,” no Monte Carlo simulations are required. This conclusion follows directly from the known fact that the asymptotic distribution of RR and NNT is given by the lognormal and the inverse normal distribution, respectively [
      • Thabane L.
      A closer look at the distribution of number needed to treat (NNT): a Bayesian approach.
      ]. The lognormal distribution has one mode, and the inverse normal distribution is bimodal [
      • Johnson N.L.
      • Kotz S.
      • Balakrishnan N.
      ]. The properties of NNT have repeatedly been investigated and described before demonstrating undesirable statistical properties [
      • Thabane L.
      A closer look at the distribution of number needed to treat (NNT): a Bayesian approach.
      ,
      • Lesaffre E.
      • Pledger G.
      A note on the number needed to treat.
      ,
      • Walter S.D.
      Number needed to treat (NNT): estimation of a measure of clinical benefit.
      ,
      • Duncan B.W.
      • Olkin I.
      Bias of estimates of the number needed to treat.
      ]. Therefore, NNT should not be used as primary effect measure for data analysis [
      • Bender R.
      Number needed to treat (NNT).
      ,
      • Stang A.
      • Poole C.
      • Bender R.
      Common problems related to the use of number needed to treat.
      ].
      Secondly, further explorations regarding the probability that the observed treatment effect is zero or negative when the true value is positive are discussed in detail for NNTs only. However, all situations that lead to a large probability that the observed treatment effect is zero or negative when the true value is positive in terms of NNTs lead to the same large probability for any other effect measure including RR. Thus, it is not a disadvantage of the NNT.
      Thirdly, for the comparison of the variation of NNT and RR, it seems that histograms simply have been visually inspected. Only data situations are considered in which the distribution of RR approximately is symmetrical. However, RR has a skew distribution whereby the skewness depends on the standard error of log(RR). On the other hand, the distribution of the risk difference (RD) is always approximately symmetrical. By using the same simple visual inspection of histograms in data situations in which RR has a clear skew distribution, the conclusion that RR varies more than RD could be made. Thus, the simple visual inspection of a few histograms is insufficient. An appropriate quantitative analysis is required to compare the variation of effect measures.
      A useful information given in the article by Wisløff et al. [
      • Wisløff T.
      • Aalen O.O.
      • Kristiansen I.S.
      Considerable variation in NNT: a study based on Monte Carlo simulations.
      ] is the conclusion that the true baseline risk is important to correctly interpret any measure of effectiveness, including RD and NNT. The explanation and discussion of this issue deserve more room, as this issue is not sufficiently explored in the existing NNT literature.
      In summary, previous research findings have been neglected leading to superfluous simulations. The general problem of insufficient power is described as negative feature of NNTs, although it affects every effect measure, and the method used to compare the variation of NNT and RR is insufficient. Nevertheless, I agree with the conclusions that NNTs should be used with caution and that the baseline risk is important for the correct interpretation of NNTs.

      References

        • Wisløff T.
        • Aalen O.O.
        • Kristiansen I.S.
        Considerable variation in NNT: a study based on Monte Carlo simulations.
        J Clin Epidemiol. 2011; 64: 444-450
        • Thabane L.
        A closer look at the distribution of number needed to treat (NNT): a Bayesian approach.
        Biostatistics. 2003; 4: 365-370
        • Johnson N.L.
        • Kotz S.
        • Balakrishnan N.
        2nd edition. Continuous Univariate Distributions. Vol 1. Wiley, New York, NY1994
        • Lesaffre E.
        • Pledger G.
        A note on the number needed to treat.
        Control Clin Trials. 1999; 20: 439-447
        • Walter S.D.
        Number needed to treat (NNT): estimation of a measure of clinical benefit.
        Stat Med. 2001; 20: 3947-3962
        • Duncan B.W.
        • Olkin I.
        Bias of estimates of the number needed to treat.
        Stat Med. 2005; 24: 1837-1848
        • Bender R.
        Number needed to treat (NNT).
        in: Armitage P. Colton T. 2nd edition. Encyclopedia of Biostatistics. Vol 6. Wiley, Chichester, UK2005: 3752-3761
        • Stang A.
        • Poole C.
        • Bender R.
        Common problems related to the use of number needed to treat.
        J Clin Epidemiol. 2010; 63: 820-825

      Linked Article

      • Regarding adequate methods for comparison of effect measures
        Journal of Clinical EpidemiologyVol. 65Issue 4
        • Preview
          We are pleased to observe that we agree with Prof Dr Bender that baseline risk should be considered when effect measurers are reported [1,2]. When it comes to his critique, we are clearly aware that one can look at the number needed to treat (NNT) from a mathematical point of view, and that a main reason for the difficulties with this measure is that it is defined by inverting a difference. This creates a fundamental instability in the NNT. Our aim, however, is to illustrate the practical implications of this for medical studies in a simple way, and we believe we do this rather clearly.
        • Full-Text
        • PDF