Advertisement

Overinterpretation of findings in machine learning prediction model studies in oncology: a systematic review

  • Paula Dhiman
    Correspondence
    Corresponding author: Dr Paula Dhiman, Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, OX3 7LD, UK.
    Affiliations
    Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, OX3 7LD, UK

    NIHR Oxford Biomedical Research Centre, Oxford University Hospitals NHS Foundation Trust, Oxford, United Kingdom
    Search for articles by this author
  • Jie Ma
    Affiliations
    Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, OX3 7LD, UK
    Search for articles by this author
  • Constanza L. Andaur Navarro
    Affiliations
    Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
    Search for articles by this author
  • Benjamin Speich
    Affiliations
    Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, OX3 7LD, UK

    Meta-Research Centre, Department of Clinical Research, University Hospital Basel, University of Basel, Basel, Switzerland
    Search for articles by this author
  • Garrett Bullock
    Affiliations
    Nuffield Department of Orthopaedics, Rheumatology, and Musculoskeletal Sciences, University of Oxford, Oxford, UK
    Search for articles by this author
  • Johanna AA Damen
    Affiliations
    Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
    Search for articles by this author
  • Lotty Hooft
    Affiliations
    Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
    Search for articles by this author
  • Shona Kirtley
    Affiliations
    Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, OX3 7LD, UK
    Search for articles by this author
  • Richard D Riley
    Affiliations
    Centre for Prognosis Research, School of Medicine, Keele University, Staffordshire, UK. ST5 5BG
    Search for articles by this author
  • Ben Van Calster
    Affiliations
    Department of Development and Regeneration, KU Leuven, Leuven, Belgium

    Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, the Netherlands

    EPI-centre, KU Leuven, Leuven, Belgium
    Search for articles by this author
  • Karel GM Moons
    Affiliations
    Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
    Search for articles by this author
  • Gary S Collins
    Affiliations
    Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, OX3 7LD, UK

    NIHR Oxford Biomedical Research Centre, Oxford University Hospitals NHS Foundation Trust, Oxford, United Kingdom
    Search for articles by this author
Open AccessPublished:March 17, 2023DOI:https://doi.org/10.1016/j.jclinepi.2023.03.012
      This paper is only available as a PDF. To read, Please Download here.

      Abstract

      Background

      In biomedical research, spin is the overinterpretation of findings, and it is a growing concern. To date, the presence of spin has not been evaluated in prognostic model research in oncology, including studies developing and validating models for individualised risk prediction.

      Study design and setting

      We conducted a systematic review, searching MEDLINE and EMBASE for oncology-related studies that developed and validated a prognostic model using machine learning published between 01/01/2019 and 05/09/2019. We used existing spin frameworks and described areas of highly suggestive spin practices.

      Results

      We included 62 publications (including 152 developed models; 37 validated models). Reporting was inconsistent between methods and the results in 27% of studies due to additional analysis and selective reporting. Thirty-two studies (out of 36 applicable studies) reported comparisons between developed models in their discussion and predominantly used discrimination measures to support their claims (78%). Thirty-five studies (56%) used an overly strong or leading word in their title, abstract, results, discussion or conclusion.

      Conclusion

      The potential for spin needs to be considered when reading, interpreting, and using studies that developed and validated prognostic models in oncology. Researchers should carefully report their prognostic model research using words that reflect their actual results and strength of evidence.

      Keywords

      List of abbreviations:

      TRIPOD (Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis), PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses), STROBE (STrengthening the Reporting of OBservational studies in Epidemiology), SGB (Stochastic Gradient Boosting), ANN (Artificial Neural Network), AUC (Area Under the Curve)