Advertisement

Large variation existed in standardized mean difference estimates using different calculation methods in clinical trials

  • Yan Luo
    Correspondence
    Corresponding author. Department of Health Promotion and Human Behavior, School of Public Health in the Graduate School of Medicine, Kyoto University, Yoshida Konoe-cho, Sakyo-ku, Kyoto 606-8501, Tel.: +81-75-753-9491; fax: +81-75-753-4641.
    Affiliations
    Department of Health Promotion and Human Behaviour, School of Public Health in the Graduate School of Medicine, Kyoto University, Yoshida Konoe-cho, Sakyo-ku, Kyoto 606-8501, Japan
    Search for articles by this author
  • Satoshi Funada
    Affiliations
    Department of Health Promotion and Human Behaviour, School of Public Health in the Graduate School of Medicine, Kyoto University, Yoshida Konoe-cho, Sakyo-ku, Kyoto 606-8501, Japan

    Department of Urology, Graduate School of Medicine, Kyoto University, 54 Shogoinkawahara-cho, Sakyo-ku, Kyoto 606-8507, Japan
    Search for articles by this author
  • Kazufumi Yoshida
    Affiliations
    Department of Health Promotion and Human Behaviour, School of Public Health in the Graduate School of Medicine, Kyoto University, Yoshida Konoe-cho, Sakyo-ku, Kyoto 606-8501, Japan
    Search for articles by this author
  • Hisashi Noma
    Affiliations
    Department of Data Science, The Institute of Statistical Mathematics, 10-3 Midori-cho, Tachikawa, Tokyo 190-8562, Japan
    Search for articles by this author
  • Ethan Sahker
    Affiliations
    Department of Health Promotion and Human Behaviour, School of Public Health in the Graduate School of Medicine, Kyoto University, Yoshida Konoe-cho, Sakyo-ku, Kyoto 606-8501, Japan

    Population Health and Policy Research Unit, Medical Education Centre, Graduate School of Medicine, Kyoto University, Yoshida Konoe-cho, Sakyo-ku, Kyoto 606-8501, Japan
    Search for articles by this author
  • Toshi A. Furukawa
    Affiliations
    Department of Health Promotion and Human Behaviour, School of Public Health in the Graduate School of Medicine, Kyoto University, Yoshida Konoe-cho, Sakyo-ku, Kyoto 606-8501, Japan
    Search for articles by this author

      Abstract

      Background and Objectives

      The standardized mean difference (SMD) can be calculated from different mean differences (MDs) and standard deviations (SDs). This study aims to investigate how clinical trials calculated, reported and interpreted the SMD, and to examine the variation between different SMDs.

      Methods

      We searched the PubMed for randomized controlled trials of general medicine and psychiatry that estimated SMDs. We explored how the SMD was computed and interpreted. We calculated SMDs based on different MDs and SDs, and the variation in these SMD estimates for each study.

      Results

      We included 161 articles. Various MDs and SDs were used to calculate SMDs, yet 69.0% studies failed to provide sufficient details. Variations in SMD estimates using different MDs and SDs in one study could be substantial (median of the absolute differences was 0.3, interquartile range IQR 0.17 to 0.53). However, 68.3% studies interpreted the SMD based on the same reference, Cohen's rule of thumb. The largest variations were observed in studies with small sample sizes and large reported effects.

      Conclusion

      SMDs using different MDs and SDs could vary considerably, but the report was often insufficient and the interpretation was oversimplified. To avoid selective reporting bias and misinterpretation, prespecifying and reporting the method and interpreting the result from multiple perspectives are desirable.

      Keywords

      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'

      Subscribe:

      Subscribe to Journal of Clinical Epidemiology
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect

      References

        • Wasserstein R.L.
        • Lazar N.A.
        The ASA statement on p-values: context, process, and purpose.
        Am Stat. 2016; 70: 129-133
        • Cumming G.
        Understanding the new statistics: effect sizes, confidence intervals, and meta-analysis.
        1st ed. Routledge/Taylor & Francis Group, New York, NY2012
        • Cusin C.
        • Yang H.
        • Yeung A.
        • Fava M.
        Rating scales for depression.
        in: Baer L. Blairs M.A. Handbook of clinical rating scales and assessment in psychiatry and mental health. 1st ed. Humana Press, Totowa, N.J.2010: 7-35
        • Cohen J.
        Statistical power analysis for the behavioral sciences.
        Academic Press, New York, N.Y.1969
        • Cohen J.
        Statistical power analysis for the behavioral sciences.
        2nd ed. Academic Press, New York, N.Y.1988
        • Morris S.B.
        Estimating effect sizes from pretest-posttest-control group designs.
        Organ Res Methods. 2008; 11: 364-386
        • Carlson K.D.
        • Schmidt F.L.
        Impact of experimental design on effect size: findings from the research literature on training.
        J Appl Psychol. 1999; 84: 851
        • Peduzzi P.
        • Henderson W.
        • Hartigan P.
        • Lavori P.
        Analysis of randomized controlled trials.
        Epidemiol Rev. 2002; 24: 26-38
        • Aloe A.M.
        • Thompson C.G.
        • Liu Z.
        • Lin L.
        Estimating partial standardized mean differences from regression models.
        J Exp Educ. 2021; : 1-18
        • What Works Clearinghouse
        What works clearinghouse procedures handbook, version 4.1.
        Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Washington, D.C.2020
        • Luo Y.
        • Funada S.
        • Noma H.
        • Furukawa T.A.
        How is standardized mean difference computed, reported and interpreted in randomized controlled trials: protocol for a meta-epidemiological study.
        OSF Preprint, 2020 (Available at)
        https://osf.io/g9rdh/
        Date accessed: November 1, 2021
        • Clifton L.
        • Clifton D.A.
        The correlation between baseline score and post-intervention score, and its implications for statistical analysis.
        Trials. 2019; 20: 43
        • Twisk J.
        • Bosman L.
        • Hoekstra T.
        • Rijnhart J.
        • Welten M.
        • Heymans M.
        Different ways to estimate treatment effects in randomised controlled trials.
        Contemp Clin Trials Commun. 2018; 10: 80-85
        • Medicines Agency European
        Guideline on adjustment for baseline covariates in clinical trials.
        European Medicines Agency (EMA), London, UK2015
        • Laird N.
        Further comparative analyses of pretest-posttest research designs.
        Am Stat. 1983; 37: 329-330https://doi.org/10.1080/00031305.1983.10483133
        • Glass G.V.
        • McGaw B.
        • Smith M.L.
        Meta-analysis in social research.
        SAGE, Beverly Hills, C.A.1981
        • Becker B.J.
        Synthesizing standardized mean-change measures.
        Br J Math Stat Psychol. 1988; 41: 257-278
        • Hedges L.V.
        • Olkin I.
        Statistical methods for meta-analysis.
        Academic Press, San Diego, C.A.1985
        • Higgins J.P.T.
        • Thomas J.
        • Chandler J.
        • Cumpston M.
        • Li T.
        • Page M.J.
        • et al.
        Cochrane handbook for systematic reviews of interventions. Version 6.2.
        Cochrane Collaboration, Hoboken, NJ2021
        • Olejnik S.
        • Algina J.
        Measures of effect size for comparative studies: applications, interpretations, and limitations.
        Contemp Educ Psychol. 2000; 25: 241-286
        • Coe R.
        It’s the effect size, stupid: what effect size is, and why it is important. Annual conference of the British educational research association.
        University of Exeter, England2002
        • Tendal B.
        • Higgins J.P.
        • Juni P.
        • Hrobjartsson A.
        • Trelle S.
        • Nuesch E.
        • et al.
        Disagreements in meta-analyses using outcomes measured on continuous or rating scales: observer agreement study.
        BMJ. 2009; 339: b3128
        • Cohen J.
        The statistical power of abnormal-social psychological research: a review.
        J Abnorm Soc Psychol. 1962; 65: 145-153
        • Schafer T.
        • Schwarz M.A.
        The meaningfulness of effect sizes in psychological research: differences between sub-disciplines and the impact of potential biases.
        Front Psychol. 2019; 10: 813
        • Valentine J.C.
        • Aloe A.M.
        • Wilson S.J.
        Interpreting effect sizes.
        in: Cooper H.M. Hedges L.V. Valentine J.C. Handbook of research synthesis and meta-analysis. 3rd ed. Russell Sage Foundation, New York, N.Y.2019: 433-452
        • Sterne J.A.
        • Gavaghan D.
        • Egger M.
        Publication and related bias in meta-analysis: power of statistical tests and prevalence in the literature.
        J Clin Epidemiol. 2000; 53: 1119-1129
        • Feingold A.
        A regression framework for effect size assessments in longitudinal modeling of group differences.
        Rev Gen Psychol. 2013; 17: 111-121
        • Balk E.M.
        • Earley A.
        • Patel K.
        • Trikalinos T.A.
        • Dahabreh I.J.
        Empirical assessment of within-arm correlation imputation in trials of continuous outcomes.
        Agency for Healthcare Research and Quality (AHRQ), Rockville, M.D.2012