Advertisement

Systematic review identifies the design and methodological conduct of studies on machine learning-based prediction models

  • Constanza L. Andaur Navarro
    Correspondence
    Correspondence to: Constanza L Andaur Navarro Julius Centre for Health Sciences and Primary Care, Universiteitsweg 100, P.O. Box 85500, 3508 GA, Utrecht, The Netherlands
    Affiliations
    Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands

    Cochrane Netherlands, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
    Search for articles by this author
  • Johanna AA. Damen
    Affiliations
    Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands

    Cochrane Netherlands, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
    Search for articles by this author
  • Maarten van Smeden
    Affiliations
    Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
    Search for articles by this author
  • Toshihiko Takada
    Affiliations
    Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
    Search for articles by this author
  • Steven WJ. Nijman
    Affiliations
    Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
    Search for articles by this author
  • Paula Dhiman
    Affiliations
    Center for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology & Musculoskeletal Sciences, University of Oxford, Oxford, United Kingdom

    NIHR Oxford Biomedical Research Centre, Oxford University Hospitals NHS Foundation Trust, Oxford, United Kingdom
    Search for articles by this author
  • Jie Ma
    Affiliations
    Center for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology & Musculoskeletal Sciences, University of Oxford, Oxford, United Kingdom
    Search for articles by this author
  • Gary S. Collins
    Affiliations
    Center for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology & Musculoskeletal Sciences, University of Oxford, Oxford, United Kingdom

    NIHR Oxford Biomedical Research Centre, Oxford University Hospitals NHS Foundation Trust, Oxford, United Kingdom
    Search for articles by this author
  • Ram Bajpai
    Affiliations
    Centre for Prognosis Research, School of Medicine, Keele University, Keele, United Kingdom
    Search for articles by this author
  • Richard D. Riley
    Affiliations
    Centre for Prognosis Research, School of Medicine, Keele University, Keele, United Kingdom
    Search for articles by this author
  • Karel GM. Moons
    Affiliations
    Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands

    Cochrane Netherlands, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
    Search for articles by this author
  • Lotty Hooft
    Affiliations
    Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands

    Cochrane Netherlands, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
    Search for articles by this author
Open AccessPublished:November 24, 2022DOI:https://doi.org/10.1016/j.jclinepi.2022.11.015
      This paper is only available as a PDF. To read, Please Download here.

      Abstract

      Objective

      We sought to summarize the study design, modelling strategies, and performance measures reported in studies on clinical prediction models developed using machine learning techniques.

      Study Design and Setting

      We search PubMed for articles published between 01/01/2018 and 31/12/2019, describing the development or the development with external validation of a multivariable prediction model using any supervised machine learning technique. No restrictions were made based on study design, data source, or predicted patient-related health outcomes.

      Results

      We included 152 studies, 58 (38.2% [95%CI 30.8-46.1]) were diagnostic and 94 (61.8% [95%CI 53.9-69.2]) prognostic studies. Most studies reported only the development of prediction models (n=133, 87.5% [95%CI 81.3-91.8]), focused on binary outcomes (n=131, 86.2% [95%CI 79.8-90.8), and did not report a sample size calculation (n=125, 82.2% [95%CI 75.4-87.5]). The most common algorithms used were support vector machine (n=86/522, 16.5% [95%CI 13.5-19.9]) and random forest (n=73/522, 14% [95%CI 11.3-17.2]). Values for area under the Receiver Operating Characteristic curve ranged from 0.45 to 1.00. Calibration metrics were often missed (n=494/522, 94.6% [95%CI 92.4-96.3]).

      Conclusions

      Our review revealed that focus is required on handling of missing values, methods for internal validation, and reporting of calibration to improve the methodological conduct of studies on machine learning-based prediction models.

      Systematic review registration

      PROSPERO, CRD42019161764.

      Keywords